Re: Model-level validation

2022-09-29 Thread Carl Meyer
On Thu, Sep 29, 2022 at 6:19 PM Curtis Maloney  wrote:

> On Thu, 29 Sep 2022, at 14:29, Aaron Smith wrote:
>
> Why doesn't Django validate Models on save()?
>
>
> The short answer is: backwards compatibility.
>
> Model level validation hasn't always been a thing, with Django initially
> depending primarily on Form validation.
>
> Since it hadn't _always_ been there, it wasn't possible to introduce it,
> enforce it, and not break most apps out there.
>
> There was so much code written that generally assumed it could call
> `save()` and not have to catch validation errors.
>
> For what it's worth, I'm all in favor of making it run on `save()` ...
> updating the documentation and preparing the community is going to be a
> mammoth task, however. A safe course through this will take some very
> careful planning.
>

Another factor that should be considered is that the Django ORM gives you
plenty of ways to update your database (eg bulk updates on a queryset) that
clearly cannot materialize and validate every object in the queryset. So is
it better to consistently say “the ORM doesn’t validate, forms do,” or
better to say “the ORM will validate on Model.save() but not in various
other cases, so you still can’t really rely on the ORM to enforce model
validation invariants consistently.”

Carl

>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAGLapGA7NCkNrBNe7LQmTOzbj%2Bpa3xm7_tEPnBa7eZbUsq1xGg%40mail.gmail.com.


Re: Simplify middlewares (again) and get rid of process_view

2018-05-30 Thread Carl Meyer
On 5/16/18 10:49 AM, Florian Apolloner wrote:
> Hi Carl,
> 
> On Wednesday, May 16, 2018 at 5:58:02 AM UTC+2, Carl Meyer wrote:
> 
> I'm not sure this part is feasible. It's an intentional part of
> middleware design AFAIK (and useful) that middleware can modify
> request.path and have this modification respected in view resolution.
> 
> 
> my proposed urlresolver_factory takes the request as argument and should
> therefore allow for anything that the middleware would allow too -- are
> there any specific usecases you have in mind that would require a
> middleware?

Composing two different middleware that both alter request.path for
different purposes? This will require the site owner to manually compose
in their urlconf_factory.

I think your core proposal that the main middleware call layering should
run after view resolution instead of before it is probably a better
design for middleware in principle, but it needs work in a couple areas:

1. Motivating the change in a way that clarifies how it will benefit
typical Django users, sufficiently to justify the churn. What new
possibilities are unlocked by the change, and has there been demand for
those possibilities?
2. Fleshing out how a reasonable transition path for existing middleware
(including third-party middleware) would look.

It's too bad we didn't think about this possibility during the original
DEP5, to avoid multiple middleware changes within a few Django versions.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/679c0352-d374-a979-90e3-732ea619ac63%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


Re: Simplify middlewares (again) and get rid of process_view

2018-05-15 Thread Carl Meyer
Hi Florian,

On 5/12/18 10:22 AM, Florian Apolloner wrote:

> After refactoring the middlewares to new-style middlewares as we have them 
> now, I am left with two pain points:
> 
>  * atomic requests are still special cased and not in a middleware
>  * process_view is as useless as always (it can neither alter nor convert 
> args/kwargs or the view)
> 
> To change this I am proposing the following changes:
> 
>  * Deprecate request.urlconf and provide a way to set the urlconf __before__ 
> the middleware chain is entered
>  * Resolve view before the middleware chain is entered

I'm not sure this part is feasible. It's an intentional part of
middleware design AFAIK (and useful) that middleware can modify
request.path and have this modification respected in view resolution.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0c8e1164-931e-72a9-438c-c099fd7447ad%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


Re: About migrations

2017-07-07 Thread Carl Meyer
On 07/07/2017 05:09 AM, Patryk Zawadzki wrote:
> Right, and one reason for generating those "no-op" migrations is that
> they aren't actually no-ops, if you value being able to write data
> migrations in Python using the ORM. They keep the historical Python
> models accurate.
> 
> I would argue that this is a fairly optimistic view of the current state :)
> 
> They are technically "historically accurate" but the point in history
> they represent is not necessary the one you had in mind unless you only
> have a single application and linear migrations (ie. no merge
> migrations). Our current dependency system only allows you to express
> "no sooner than X" but the graph solver can execute an arbitrary number
> of later migrations between the one you depend on and the one you wrote.
> 
> Imagine you have app A and migration M1 adds field F. You then create a
> migration M2 in another application B that needs to access F so you have
> it depend on (A, M1). Two months later field F is removed or renamed in
> migration M3. Django has two ways to linearize the graph: (A, M1), (B,
> M2), (A, M3) or (A, M1), (A, M3), (B, M2). Both are valid but the latter
> will result in a crash when migrating from an empty DB state. In
> practice we often have to add arbitrary dependencies to later migrations
> to force a Python migration to execute in the correct order.

Yeah, that's an issue I've certainly run into. It's not _that_
unreasonable or arbitrary to solve this by adding a dependency of (A,
M3) on (B, M2), but it would be better if we could express this as a
"must-run-before" dependency on the B side (since the dependency of app
B on app A may be one-way, and we shouldn't have to introduce knowledge
of B into A's migrations -- and A may even be third-party). IMO this
would be a reasonable feature addition.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/042cd6a3-1e7c-b5a5-7909-2ef90f91c284%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: About migrations

2017-07-04 Thread Carl Meyer
On 07/04/2017 07:04 AM, Patryk Zawadzki wrote:
> Have DB backends understand certain field types expressed as strings
> ("varchar", "text", "blob", "decimal" and so on).
> 
> Possibly some backends could implement a wider set than the others
> ("json", "xml", "rasterimage" etc.).
> 
> Have each Field class deconstruct to a field name and params, eg:
> "decimal", {"digits": 12, "decimals": 2}.
> 
> Then a model becomes essentially a list of tuples:
> 
> [
> ("title", "varchar", {"length": 100}),
> ("price", "decimal", {"digits": 12, "decimals": 2}),
> ...
> ]
> 
> This is not far from what "render model states" does currently except
> that it compares much richer model descriptions that leads to no-op
> migrations being generated each time you change a label or a
> user-visible part of choices.

Right, and one reason for generating those "no-op" migrations is that
they aren't actually no-ops, if you value being able to write data
migrations in Python using the ORM. They keep the historical Python
models accurate.

Of course, we do pay a cost in complexity for the "historical ORM"
feature, and it's reasonable to prefer a tradeoff that doesn't pay that
cost and requires all data migrations to be written in SQL. As Andrew
mentioned, there's nothing to prevent anyone from writing an alternative
migrations frontend that takes this approach. It should be able to reuse
the schema editor backend, which does the heavy lifting of cross-db
schema alteration.

It's worth remembering, though, that five or six years ago we _had_ a
range of different migrations solutions that chose different tradeoffs,
and South was the clear winner in user uptake. It's not due to arbitrary
whim that the Django migrations system is based on South and preserves
its popular features, like the historical ORM.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/43abcfed-edaa-6f05-04cc-f30a9b0e59e6%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Exceptions in model._meta._property_names with some 3rd-pty libraries

2017-06-02 Thread Carl Meyer
Hi Zack,

On 06/02/2017 02:02 PM, Zack Voase wrote:
> Hi all,
> 
> I'm encountering exceptions in a number of popular third-party Django
> libraries when upgrading from 1.8 to 1.11. The code path is typically
> `MyModel.objects.get_or_create(...)`, which causes
> `model._meta._property_names` to be checked (to make sure we're not
> querying/setting a property). The problem is, `_property_names` is
> implemented as follows:
> 
> |
> # in django/db/models/options.py:
> def_property_names(self):
> returnfrozenset({
>attr forattr in
>dir(self.model)ifisinstance(getattr(self.model,attr),property)
> })
> |
> 
> The problem is when a custom field installs a field descriptor on the
> model class (during `contribute_to_class()`), with a `__get__()` method
> like this:
> 
> |
> classSomeFieldDescriptor(object):
> # ...
> def__get__(self,instance,type=None):
> ifinstance isNone:
> raiseAttributeError("Can only be accessed via an instance.")
> # ...
> |

I can see two things here that could be done better, one on Django side
and one on third-party app side.

1) I've never seen a good reason for a descriptor to raise an
AttributeError like that. Typically `return self` is a much more useful
option for the "access on the class" scenario, if the descriptor can't
provide useful behavior in that case, as it allows introspecting the
class and finding the descriptor object.

2) On the Django side, I think we should switch to using
`inspect.getattr_static`, to avoid any possibility of triggering
side-effecty descriptor code of any kind. AttributeError is only the
most visible problem here; there could be much subtler problems (e.g.
performance issues) caused by e.g. accessing a descriptor that does a
database query or something.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fbd639ad-6a51-7657-f647-753cbc3b6b90%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Cython usage within Django

2017-05-21 Thread 'Carl Meyer' via Django developers (Contributions to Django itself)
On 05/21/2017 02:59 PM, Tom Forbes wrote:
> There are lots of considerations to take into account (like ensuring the
> Cython functions are in sync with the fallback ones)

It's possible to only have one version of the code, using only Python
syntax, and conditionally compile it with Cython if available. This
gives up some potential efficiency wins from type annotation, but avoids
the need to keep two copies in sync.

Regardless, though, I think CI would need to run the tests both with
Cython and with non-Cython fallback.

We've moved toward releasing wheels instead of sdist on PyPI for recent
versions; for this to be useful it would mean releasing multiple binary
wheels for different platforms.

There's no question this could make a big difference to Django CPU
usage; the question is whether it's worth the added CI and release
complexity when it would likely provide little value to the majority of
Django users.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ff2d35c1-e384-e010-b57f-d249df3340b2%40instagram.com.
For more options, visit https://groups.google.com/d/optout.


Re: Database "execute hooks" for instrumentation

2017-04-07 Thread Carl Meyer
Hi Shai,

On 04/07/2017 06:02 AM, Shai Berger wrote:
> This is an idea that came up during the djangocon-europe conference: Add the 
> ability to install general instrumentation hooks around the database 
> "execute" 
> and "executemany" calls.
> 
> Such hooks would allow all sorts of interesting features. For one, they could 
> replace the current special-case allowing assertNumQueries & friends to 
> record 
> queries out of debug mode (it's an ugly hack, really), but they could also 
> support my imagined use case -- a context-manager which could prevent 
> database 
> access during execution of some code (I'm thinking mostly of using it around 
> "render()" calls and serialization, to make sure all database access is being 
> done in the view).

Another use-case is for preventing database access during tests unless
specifically requested by the test (e.g. pytest-django does this,
currently via monkeypatching).

> My idea for implementation is to keep a thread-local stack of context-
> managers, and have them wrap each call of "execute". We could actually even 
> use one such context-manager instead of the existing CursorDebugWrapper.

Why a new thread-local instead of explicitly per-connection and stored
on the connection?

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5d5ee8f1-6b22-c48c-7a6b-11eac2928b64%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: A proposal for a new auto-reloader in Django

2017-04-02 Thread Carl Meyer
Hi David,

I wouldn't bother with the wsgiwatcher repo; it's proof-of-concept code
from one sprint's worth of hacking, not used by anyone. Look at hupper
instead, it is based on wsgiwatcher, but much changed, and actually used
in production.

Carl

On 04/02/2017 04:27 PM, qingnian...@gmail.com wrote:
> Hi Carl,
>   I don't quite understand why get_module_paths() in your wsgiwatcher
> project is returning a list of python module paths. I thought it would
> return the directory that needs to be monitored. Could you please tell
> me how this part works? Thanks.
> 
> David Ma
> 
> On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:
> 
> Anyone working on this project should at least be aware of
> https://github.com/Pylons/hupper <https://github.com/Pylons/hupper>
> (based on work David Glick and I
> originally did in https://github.com/carljm/wsgiwatcher
> <https://github.com/carljm/wsgiwatcher>), which aims to
> be a framework-agnostic solution to this problem for any Python web
> project. Docs at
> http://docs.pylonsproject.org/projects/hupper/en/latest/
> 
> <http://www.google.com/url?q=http%3A%2F%2Fdocs.pylonsproject.org%2Fprojects%2Fhupper%2Fen%2Flatest%2F&sa=D&sntz=1&usg=AFQjCNHhMlzZ6K6HLWsxUxOPanWzxLeMlg>
> 
> 
> Carl
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-developers+unsubscr...@googlegroups.com
> <mailto:django-developers+unsubscr...@googlegroups.com>.
> To post to this group, send email to django-developers@googlegroups.com
> <mailto:django-developers@googlegroups.com>.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/4fac9a8a-231d-41c5-a5fb-6eaeb2bd148f%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/4fac9a8a-231d-41c5-a5fb-6eaeb2bd148f%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/99fbed1c-3ac4-64b9-5030-3d5e2533cef4%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: A proposal for a new auto-reloader in Django

2017-03-30 Thread Carl Meyer
Anyone working on this project should at least be aware of
https://github.com/Pylons/hupper (based on work David Glick and I
originally did in https://github.com/carljm/wsgiwatcher), which aims to
be a framework-agnostic solution to this problem for any Python web
project. Docs at http://docs.pylonsproject.org/projects/hupper/en/latest/

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/af0d4ea6-e97e-522e-947a-83db87c6ebd0%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Adding a middleware to match cookies

2017-01-07 Thread Carl Meyer
On 01/07/2017 03:25 AM, Florian Apolloner wrote:
> On Saturday, January 7, 2017 at 3:50:56 AM UTC+1, Jeff Willette wrote:
> 
> What if there was an optional middleware early in the request
> processing that matched cookies based on a regex in settings and
> then modified the header to only include the matched cookies?
> 
> 
> I do not see how this would help -- you'd still have to set "Vary:
> Cookie" on the response as soon as you are accessing request.user. Or is
> the goal of this to allow Django's internal page caching stuff to ignore
> some cookies? That seems doable, but very very dangerous.

Right, the latter is how I understood it; you'd still use Vary: Cookie,
but strip some cookies before the request reaches the cache middleware.

I don't think it's too dangerous, if you're conservative about the
cookies you strip (e.g. only strip cookies that are known for sure to be
unused on the server, like Google Analytics cookies for instance.)

> 
> This issue reminds me of another issue I came up with (or as Carl puts
> it: "…presenting the hypothetical case that exposed this bug."), namely
> https://code.djangoproject.com/ticket/19649 -- Basically as soon as
> Django accesses __any__ cookie we should set "Vary: Cookie", with all
> the downsides this entails. I think we finally should fix that and put a
> fix for it into the BaseHandler.

+1

> What would be great would be an HTTP header which allowed for something
> ala "Cache: if-request-did-not-have-cookies" -- usually it is pointless
> to cache __anything__ with cookies anyways. That said, with all the
> analytics super cookies out there, there are not many pages without
> cookies anymore :(

+1. Basically analytics have already effectively broken HTTP caching as
it was designed to work.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/709febeb-8336-aafa-7faa-74d1e2b46802%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Adding a middleware to match cookies

2017-01-07 Thread Carl Meyer
On 01/06/2017 11:26 PM, Jeff Willette wrote:
> Wy would this not help the efficiency of the downstream caches? Is it
> because the request has already passed through them with the cookies
> intact? and when it comes back through the response they have no way to
> know they have been stripped?

That's correct. Stripping cookies from the request in Django is far too
late to have any effect on an external cache. If the request has reached
Django, then it's already passed through any external caching proxies,
with all cookies, and the cache has already decided not to serve a
cached response. (And if the cache holds on to the response, it'll
associate with the the request it saw, which still had all its cookies).

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d1383be4-749e-3183-4354-ac0047bd0517%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Adding a middleware to match cookies

2017-01-06 Thread Carl Meyer
Hi Jeff,

On 01/06/2017 06:21 PM, Jeff Willette wrote:
> I understand that calling is_authenticated on a user will require the
> session to be accessed and the vary by cookie header to be in the
> response, but if I understand how caching systems work then this will
> cause all cookies in the request to be taken into account, correct?

Yes. HTTP doesn't provide any way to say "vary only on this cookie, not
the others." Be nice if it did!

> What if there was an optional middleware early in the request
> processing that matched cookies based on a regex in settings and then
> modified the header to only include the matched cookies?
> 
> That way...the unauthed users request will vary by cookies, but we
> would have removed all inconsequential cookies so all unauthed users
> will have the same set of cookies (likely none), and authed users
> will have (sessionid) or whatever else you wish to match and everyone
> will be happily cached correctly.
> 
> Is there a hole in my thinking anywhere? Would this work as I
> expect?

I think it could work, yeah. It won't help the efficiency of any other
downstream HTTP caches, but they would still be safe (not serve anyone
the wrong response). And you should be able to help efficiency of
Django's own cache this way, if you strip cookies that Django / your
code doesn't care about before the request ever reaches the caching
middleware. Try it and experiment!

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/65281bd7-a2f6-d428-9743-683714c83057%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template handling of undefined variables

2017-01-04 Thread Carl Meyer
Hi Tim,

On 01/04/2017 03:39 PM, Tim Martin wrote:
> I've been looking at bug #24977 (Template variables with a value of
> None are considered to be == to non-existent properties).
...
> The suggestion on the ticket is to use an instance of a special
> `Undefined` class to indicate an undefined variable, which could never
> compare equal to anything else.
> 
> This sounds sensible, but in implementing it I got to thinking about
> whether this can generalise and simplify. It seems like it might be
> possible to combine both Engine.string_if_invalid and the
> ignore_failures parameter to FilterExpression.resolve:
> 
> * The Undefined object can have a __str__() that returns the
>   string_if_invalid value, so it acts like a string for the purposes
>   of rendering.
> 
> * Returning the Undefined value is also sufficient to provide the
>   behaviour we need in the ignore_failures case: the calling code can
>   check for Undefined in the same case where it currently checks for
>   None.
> 
> Overall the code is simpler, because a bunch of code paths are
> eliminated.

Sounds great!

> However, there are 2 issues:
>
> 1) There are test cases where we have templates that should treat "x
>is y" as True where x and y are both undefined.

Really? Is this an accidental side effect of some historical change, or
really the intent of the test? That behavior seems buggy to me; more
likely to be a bug-magnet than useful.

...
>I don't see any way to satisfy both these requirements. Is it
>reasonable to relax the requirement that "x is y" should be treated
>as True when both variables are undefined?

Seems reasonable to me.

> 2) There appears to be an inconsistency in the default_if_none
>modifier. If I have a template with
> 
>x|default_if_none:y = {{x|default_if_none:y}}
>{% if x|default_if_none:y %}
>x|default_if_none:y is apparently True
>{% endif %}
> 
>with y defined to True and x undefined, then it produces:
> 
>x|default_if_none:y =
>x|default_if_none:y is apparently True
>   
>IOW it appears that the default_if_none doesn't cause y's value to
>be used in the case where the variable is being printed, even
>though it causes the expression to have y's value when evaluated in
>an if. I think this is something to do with the fact that the two
>ways of handling failures (ignore_failures and string_if_invalid)
>do different things.
> 
>I don't see a way to make backward compatible code here.
> 
>I think this is just a bug, does anyone think otherwise?

I agree.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/14c76486-2649-c68f-a463-b5acf41b9556%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Changing {% include %} to strip a trailing newline

2017-01-04 Thread Carl Meyer
On 01/04/2017 02:53 PM, Tim Graham wrote:
> I'd like to know from Carl, Adam, and others, how much effort would be
> required to adapt the templates in your project for the new behavior. I
> imagine a script could be written to add newlines after all {% include
> %} in all plain text templates, for example.

Not much, I'd expect. Plain text templates aren't the common case, and
many of them don't use {% include %} anyway. And as you say, it is a
scriptable change if necessary.

That said, IMO the perceived cost of backwards-incompatible changes at
upgrade time scales with the number of changes required, even when
individually they don't require much work to fix. In theory, we promise
backwards-compatibility, and I don't think "call it a bugfix" can be a
universal escape hatch; it's clear that the current behavior has been
known and accepted for years. If we change this, it's a small but
incompatible change in the known and intended behavior, it's not fixing
a bug. But we've bitten the bullet and made far larger
backwards-incompatible changes in the past, so...

There is also the possibility that someone is relying on this behavior
in a more intentional way than I've ever done myself; e.g. has an
include file with multiple trailing newlines or other trailing
whitespace, specifically because it's desired in multiple different
places (in a preformatted text context). This (admittedly hypothetical)
case isn't quite as trivial to work around, if you propose to strip all
trailing whitespace. Or do you propose to strip only exactly one
trailing newline character?

In the end, I'm happy to abstain from this decision; I wouldn't really
be upset by any of the three options Aymeric listed.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d0dd9fb3-a009-bbe6-64fa-906d5cc44ff3%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-12-20 Thread Carl Meyer

On 12/20/2016 02:04 PM, Tim Graham wrote:
> I think it would be nice to be able to look at the name of the "project"
> renderer and understand that it's connected to settings.TEMPLATES. I'm
> not sure if the term "project" does that well. Maybe
> "TemplatesSettingEngines"?

Yeah... I guess I thought ProjectTemplates got reasonably close to that,
since settings.TEMPLATES is the template configuration for your project.
I guess "TemplatesSettingEngines" could be OK, it just fails to roll off
the tongue. Not sure why we'd tack on "Engines" when we don't to any of
the other names (even though they all use a template engine or engines);
maybe just "django.forms.renderers.TemplatesSetting"?

I still slightly prefer "ProjectTemplates", but you're painting the
bikeshed, feel free to choose the color :-)

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f7b131b4-910a-ec72-a045-badd4cba74ad%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-12-20 Thread Carl Meyer
On 12/20/2016 01:26 PM, Tim Graham wrote:
> Okay I removed 'POST_APP_DIRS' from the PR.
> 
> Next, we should discuss whether or not to make the ProjectTemplate
> renderer the default FORM_RENDERER in the startproject template. (For
> backwards compatibility, DjangoTemplateRenderer has to be the default in
> global_settings.py.)
> 
> Florian said:
> 
>   the first question in IRC I (and everyone else in IRC) will have to
> answer for weeks will be:
> I have a nice context_processor where I specify the theme (color) of
> my material design template, why is the variable not in the templates
> for widgets, after all it is in the same template directory.
> I am already using django-sniplates via libraries in the TEMPLATE
> config, as migration step I want to reuse that for the widgets, why are
> they not getting picked up.
> 
> Carl said:
> 
> "the best we can do to address the support concern that may arise once
> people start to use custom widgets and wonder why their TEMPLATES config
> doesn't apply, is to clearly document that if you're creating custom
> widgets that need custom template features, you should upgrade to
> ProjectTemplateRenderer. (And we should update startproject to use
> ProjectTemplateRenderer, so this support concern is a temporary thing
> during the upgrade timeframe only.)"
> 
> I feel these concerns may be overstated. I think custom template widget
> rendering won't be high on the list of things that Django beginners are
> looking to do and different values betwen "setting default" and
> "startproject default" causes confusion too. If we do make the addition,
> I guess 'django.forms' would be added  somewhere in INSTALLED_APPS also.

Yes, it would mean adding django.forms to INSTALLED_APPS as well.

Personally, I'm totally fine with deferring this decision until we see
how the default DTL renderer plays out in practice, and whether it does
cause confusion with people trying to build custom widgets.

I do think it'd be good to add to the DjangoTemplates renderer docs,
noting specifically that if you are using this renderer your custom
widget templates will use a "stock" DTL engine and won't have access to
any template engine customizations in your TEMPLATES setting, and
recommend ProjectTemplateRenderer if you need the latter.

> Finally, I'd like if the renderer names were less verbose. Proposal:
> django.forms.renderers.DjangoTemplates, Jinja2, TemplateEngines.

+1. The first two names are fine with me, not sure about
TemplateEngines. It seems kinda generic and unclear: all of the
renderers use "template engines" in one way or another. What about
"ProjectTemplates"? "ConfiguredTemplates"?

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5d2bed65-88cf-5ab8-4f75-cdc61ef2b7b6%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-12-20 Thread Carl Meyer
Sure, I guess Florian mentioned dropping Jinja2TemplateRenderer, but I
don't really see a strong argument against keeping and documenting it.
As you say, cost is low.

Carl

On 12/20/2016 11:55 AM, Tim Graham wrote:
> I thought maybe that use case could be an argument for keeping the
> Jinja2TemplateRenderer -- we need it for testing the Jinja2 templates
> anyway, so it's not adding any maintenance overhead, it's just a matter
> of whether it's public with docs or if it just lives in the tests.
> 
> About the fate of 'POST_APP_DIRS' -- it seems the consensus among you
> and Florian is to drop it. Fine with me.
> 
> On Tuesday, December 20, 2016 at 2:26:33 PM UTC-5, Carl Meyer wrote:
> 
> I think that a)  wanting render forms via Jinja and everything else via
> DTL, and b) caring about the perf impact of checking two engines is an
> edge case, and a great reason to have the full flexibility of the form
> renderers system. When we say "promote the usage of
> ProjectTemplateRenderer", I think we mean as the most flexible and
> least
> surprising option for most projects that just want Things To Work, not
> that it will always meet everyone's needs. If you have more specific
> needs, you can always write a form renderer that meets them.
> 
> Carl
> 
> On 12/20/2016 11:16 AM, Florian Apolloner wrote:
> > One option would be to introduce a new PREFIXES option in the
> template
> > engine settings which ignores template paths not starting with one of
> > those prefixes (if they are set). That said I didn't bother
> checking for
> > performance issues, I know that the cached loader in Django will also
> > cache if it doesn't find anything, ie it will not do extra I/O for
> > non-existing templates (after the initial round). As far as I
> understand
> > the Jinja2 code, non-existing file are extra I/O every time, so that
> > might be something to consider.
> >
> > Cheers,
> > Florian
> >
> > On Tuesday, December 20, 2016 at 7:34:21 PM UTC+1, Tim Graham wrote:
> >
> > Florian started a long discussion [0] on the pull request and
> > concluded in him saying, "If we are going to promote the usage of
> > |ProjectTemplateRenderer| (which I think we should), we probably
> > should bite the dust and get rid of |POST_APP_DIRS| and in the
> same
> > breath of the jinja renderer -- ie provide the Django renderer
> > really only as backwards compat shim."
> >
> > One concern I have is that if you want to
> > ues ProjectTemplateRenderer and render Jinja2 widget templates
> but
> > use Django templates elsewhere, then you have to put a Jinja2
> > backend first in TEMPLATES which means all your non-form widget
> > template lookups have an additional cost of checking if the
> Jinja2
> > template exists. Does anyone have a sense if that would be a
> > noticeable performance penalty? (and perhaps an argument against
> > ProjectTemplateRenderer for a use case like that)
> >
> > [0]
> >
> https://github.com/django/django/pull/6498#issuecomment-268120711
> <https://github.com/django/django/pull/6498#issuecomment-268120711>
> >
> 
> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F6498%23issuecomment-268120711&sa=D&sntz=1&usg=AFQjCNH6pYlfIAgKUN59fcVGOTnZ7FikCw
> 
> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F6498%23issuecomment-268120711&sa=D&sntz=1&usg=AFQjCNH6pYlfIAgKUN59fcVGOTnZ7FikCw>>
> 
> >
> > On Monday, December 19, 2016 at 5:53:23 PM UTC-5, Tim Graham
> wrote:
> >
> > It's ready for review:
> >
> > Add 'POST_APP_DIRS' TEMPLATES option.
> > https://github.com/django/django/pull/7693
> <https://github.com/django/django/pull/7693>
> > <https://github.com/django/django/pull/7693
> <https://github.com/django/django/pull/7693>>
> > Template based widget rendering:
> > https://github.com/django/django/pull/6498
> <https://github.com/django/django/pull/6498>
> > <https://github.com/django/django/pull/6498
> <https://github.com/django/django/pull/6498>>
> >
> > On Wednesday, December 14, 2016 at 6:5

Re: Template-based widget rendering

2016-12-20 Thread Carl Meyer
I think that a)  wanting render forms via Jinja and everything else via
DTL, and b) caring about the perf impact of checking two engines is an
edge case, and a great reason to have the full flexibility of the form
renderers system. When we say "promote the usage of
ProjectTemplateRenderer", I think we mean as the most flexible and least
surprising option for most projects that just want Things To Work, not
that it will always meet everyone's needs. If you have more specific
needs, you can always write a form renderer that meets them.

Carl

On 12/20/2016 11:16 AM, Florian Apolloner wrote:
> One option would be to introduce a new PREFIXES option in the template
> engine settings which ignores template paths not starting with one of
> those prefixes (if they are set). That said I didn't bother checking for
> performance issues, I know that the cached loader in Django will also
> cache if it doesn't find anything, ie it will not do extra I/O for
> non-existing templates (after the initial round). As far as I understand
> the Jinja2 code, non-existing file are extra I/O every time, so that
> might be something to consider.
> 
> Cheers,
> Florian
> 
> On Tuesday, December 20, 2016 at 7:34:21 PM UTC+1, Tim Graham wrote:
> 
> Florian started a long discussion [0] on the pull request and
> concluded in him saying, "If we are going to promote the usage of
> |ProjectTemplateRenderer| (which I think we should), we probably
> should bite the dust and get rid of |POST_APP_DIRS| and in the same
> breath of the jinja renderer -- ie provide the Django renderer
> really only as backwards compat shim."
> 
> One concern I have is that if you want to
> ues ProjectTemplateRenderer and render Jinja2 widget templates but
> use Django templates elsewhere, then you have to put a Jinja2
> backend first in TEMPLATES which means all your non-form widget
> template lookups have an additional cost of checking if the Jinja2
> template exists. Does anyone have a sense if that would be a
> noticeable performance penalty? (and perhaps an argument against
> ProjectTemplateRenderer for a use case like that)
> 
> [0]
> https://github.com/django/django/pull/6498#issuecomment-268120711
> 
> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F6498%23issuecomment-268120711&sa=D&sntz=1&usg=AFQjCNH6pYlfIAgKUN59fcVGOTnZ7FikCw>
> 
> On Monday, December 19, 2016 at 5:53:23 PM UTC-5, Tim Graham wrote:
> 
> It's ready for review:
> 
> Add 'POST_APP_DIRS' TEMPLATES option.
> https://github.com/django/django/pull/7693
> <https://github.com/django/django/pull/7693>
> Template based widget rendering:
> https://github.com/django/django/pull/6498
> <https://github.com/django/django/pull/6498>
> 
> On Wednesday, December 14, 2016 at 6:59:16 PM UTC-5, Tim Graham
> wrote:
> 
> I don't have any strong objections at this point, just
> wanted to think it through and offer possible alternatives.
> If there's no further input, I'll finish adding tests and
> docs for the TEMPLATES 'POST_APP_DIRS' option tomorrow,
> after which I think the widget rendering patch will be ready
> for final reviews.
> 
> On Wednesday, December 14, 2016 at 6:40:36 PM UTC-5, Carl
> Meyer wrote:
> 
> Hi Tim,
> 
> On 12/14/2016 12:50 PM, Tim Graham wrote:
> > My thinking is that there should typically be only one
> directory in each
> > project that overrides built-in templates, otherwise
> if multiple apps
> > provide overrides, they'll stomp on each other when
> using the
> > app_directories loader.Are your projects usually set
> up that way? Using
> > the app_directories loader eases setup but adds a
> cognitive overhead of
> > figuring out where the template overrides are coming
> from.
> 
> All of these seem like generic objections to APP_DIRS / the
> app_directories loader, which has been around in some
> form (and been the
> default behavior) effectively forever. For better or
> worse, we have a
> template system that layers multiple loaders in the same
> template
> namespace and allows them to 

Re: Template-based widget rendering

2016-12-14 Thread Carl Meyer
Hi Tim,

On 12/14/2016 12:50 PM, Tim Graham wrote:
> My thinking is that there should typically be only one directory in each
> project that overrides built-in templates, otherwise if multiple apps
> provide overrides, they'll stomp on each other when using the
> app_directories loader.Are your projects usually set up that way? Using
> the app_directories loader eases setup but adds a cognitive overhead of
> figuring out where the template overrides are coming from.

All of these seem like generic objections to APP_DIRS / the
app_directories loader, which has been around in some form (and been the
default behavior) effectively forever. For better or worse, we have a
template system that layers multiple loaders in the same template
namespace and allows them to stomp on each other, and leaves it up to
you to figure it out if they do. This does give people some rope to
create confusion, but it's also powerful and flexible, and so far we've
decided that tradeoff is worth it. How is any of this specific to form
templates?

> If you feel
> the ship has sailed and the pattern of putting the main "project" module
> in INSTALLED_APPS to get "project-level"  templates rendered should be
> endorsed, then I guess we'll do that.

I think some kind of "project" or "core" app is in practice pretty
common (e.g. it also becomes necessary if you want "project-level"
management commands, or DTL template tags, or...). I'm sure it's not
universal; I don't think there's anything wrong with it, or anything
wrong with not doing it. And I don't think that any decision we make
here needs to imply an endorsement of one approach or the other.

I understand that in my proposed default setup, if a project relies on
DIRS for their project-level templates, they won't be able to override
form templates along with the rest of their project-level templates;
they'll either need to switch to the ProjectTemplateRenderer or put
their form override templates in an app. That's not ideal, but I think
it's still a reasonable set of options (I'd probably go for
ProjectTemplateRenderer in that situation -- "if you want project
templates to override form templates, use ProjectTemplateRenderer" seems
reasonable). While I agree this is a wrinkle, I don't think this wrinkle
is a good rationale for making it impossible to override built-in form
templates from an app in the default setup.

If you disagree with that, though, I could live with with just saying to
everyone "if you want to override form templates, use
ProjectTemplateRenderer" -- it's not that hard to switch to
ProjectTemplateRenderer.

I would not be in favor of the FORM_TEMPLATE_DIRS setting. It adds a
brand-new concept and setting, which would typically be set to the same
value as your template DIRS in the common case, without really gaining
any flexibility or much ease-of-use compared to just switching to
ProjectTemplateRenderer.

Carl

> An alternative could be an additional setting to declare the
> "project-level" templates directory. That would allow overriding
> built-in and third-party app templates without create a "project" app.
> For example, the engine for the DjangoTemplateRenderer would look like:
> 
> def engine(self):
> dirs = [os.path.join(ROOT, 'templates')]
> if settings.FORM_TEMPLATES_DIR:
> dirs.insert(0, FORM_TEMPLATES_DIR)
> return DjangoTemplates({
> 'APP_DIRS': True,
> 'DIRS': dirs,
> 'NAME': 'djangoforms',
> 'OPTIONS': {},
> })
> 
> This loading order makes it impossible for apps to override the built-in
> templates unless that app is pointed to by FORM_TEMPLATES_DIR. You
> couldn't easily do further customizations of built-in templates without
> copying that app's templates into a new directory and pointing
> FORM_TEMPLATES_DIR at it, then editing those templates or adding new ones.


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cac98bf1-4f8c-be8a-8f7b-1ff95d627c24%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-12-14 Thread Carl Meyer
On 12/14/2016 11:08 AM, Carl Meyer wrote:
> As for whether a hypothetical third-party app that wants to override
> form widgets should do it by just providing the template overrides and
> clearly documenting that, or should do it by providing a custom form
> renderer, that's a separate question. I wouldn't have a big problem with
> either approach. There are already plenty of apps out there that
> override e.g. the default admin templates; this isn't much different.

Actually, on further thought, I do have a strong opinion here -- such a
hypothetical third-party app should just provide the template overrides
in the expected location; it should not provide a custom form renderer.
The former is easier to use for the simple case (just add to
INSTALLED_APPS) and also more flexible and easier to
integrate/control/override at the project level (e.g. by modifying order
of INSTALLED_APPS, or using DIRS and ProjectTemplateRenderer, or using a
project-specific custom renderer) for advanced uses.

Whereas a custom app-specific renderer doesn't compose well with an
existing custom form renderer that a project might use, making advanced
use more difficult.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/77412aeb-5c89-d62a-10e5-dbe1c5b604fc%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-12-14 Thread Carl Meyer
On 12/14/2016 10:27 AM, Tim Graham wrote:
> I'm not sure if having the default renders insert the built-in form
> templates directory after the app dirs (as opposed to before, as usual
> with 'DIRS') is the best approach. On the PR, we discussed adding a new
> TEMPLATES 'POST_APP_DIRS' option to accomplish this but it might be
> unneeded complexity. I mentioned, "I'm concerned that it gives the idea
> that third-party apps should be overriding templates to customize things
> when that's really not appropriate since only the template in the
> earliest installed app will be found." Carl said "Yeah, I agree that
> third-party apps in most cases shouldn't be overriding form widgets
> (though I suppose I could see some exceptions for e.g. something like a
> django-bootstrap-forms app, if it clearly advertises that this is what
> it does)." If overriding templates via app directories is the exception,
> should that behavior be eased by default? I think using a custom
> renderer (perhaps even one provided by the app itself) that uses DIRS
> would be a more explicit setup.

There's a big difference between "apps" and "third-party apps" -- not
all apps are third-party; in fact I think most aren't. I've never seen a
project that didn't have installed "apps" that were part of the project
itself. In fact I've seen a number of people who put their main
"project" module itself in INSTALLED_APPS and get their "project-level"
templates rendered that way, and don't use template DIRS at all.

These "local" apps is the primary use case I'm thinking of for installed
apps overriding form widget templates, and I think it's a strong and
valuable use case that will be very commonly used once templated widgets
are in core. I've been using templated widgets (via django-floppyforms)
for many years, and my experience is that I've wanted to override
built-in widget templates to tweak their markup on every single project
I've used them on. Frankly, overriding widget templates is the primary
motivation for having templated widgets in the first place! I think its
important that the default setup make overriding easy.

As for whether a hypothetical third-party app that wants to override
form widgets should do it by just providing the template overrides and
clearly documenting that, or should do it by providing a custom form
renderer, that's a separate question. I wouldn't have a big problem with
either approach. There are already plenty of apps out there that
override e.g. the default admin templates; this isn't much different.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f64544f7-ce0e-f58a-da5d-7ffe709d38d1%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-12-07 Thread Carl Meyer
Hi Tim,

On 12/07/2016 08:41 AM, Tim Graham wrote:
> This scheme seems to be working well so far.

Great, thanks for working on implementation. I did have intentions of
putting together the implementation once I'd gotten your feedback on the
design, but I won't complain if you've already done it :-)

> One thing you may not have thought of is that switching
> to JinjaTemplateRenderer is incompatible with the admin because jinja2
> templates aren't provided for those widgets. I think the reasoning was
> that they're complicated to convert due to the use of the i18n and
> static template tags and (under the old rendering scheme) a
> DjangoTemplates backend had to be available anyway for rendering the
> main admin templates. So I think JinjaTemplateRenderer may not be that
> useful in practice as it requires your project and all its third-party
> apps to provide Jinja2 templates for all widgets.
> 
> I used these steps to use Jinja2 and allow the admin to continue using
> DjangoTemplates:
> 1. Prepend this to the default settings.TEMPLATES:
>  {
> 'BACKEND': 'django.template.backends.jinja2.Jinja2',
> 'APP_DIRS': True,
> }
> 2. Add 'django.forms' to INSTALLED_APPS.
> 3. Add settings.FORM_RENDERER =
> 'django.forms.renderers.templates.ProjectTemplateRenderer'

Makes sense. I still think we may as well provide JinjaTemplateRenderer,
for completeness, and because not all projects use the admin. But the
docs should be clear about the limitations, and point to
ProjectTemplateRenderer for more flexible setups.

Let me know once you have a PR that you feel is ready for review, happy
to review it.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6768a102-edc7-ae69-ceec-32aab4beedbb%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-12-02 Thread Carl Meyer
Hi Tim,

On 11/16/2016 06:23 AM, Tim Graham wrote:
> I took a stab at backwards-compatibility with respect to allowing the
> admin to work without adding 'django.forms' to INSTALLED_APPS. It seems
> to work for at least the simple TEMPLATES = 'BACKEND':
> 'django.template.backends.django.DjangoTemplates', ...} case, although
> I'm not proud of the code and I'm nervous it's a bit fragile. What do
> you think of the approach?

I have similar concerns about fragility, as well as the behavior being
complex and hard to keep a clear mental model of.

I went back and re-read this thread and took a fresh approach to the
problem, trying to adhere to "keep the easy things easy, make the hard
things possible" and keep the behavior as explicit as possible. Here's
what I came up with, let me know what you think:

We provide three built-in template renderer classes:

1. The default FORM_RENDERER is DjangoTemplateRenderer. It does not use
settings.TEMPLATES/get_template(), but constructs its own DTL template
engine. This engine has APP_DIRS=True, and automatically inserts the
built-in form templates directory _after_ the app dirs. (This will
require a subclass of the normal DTL Engine class that overrides the
template_dirs property, because DIRS takes priority over APP_DIRS, so
the default Engine has no way to specify a directory that has lower
priority than APP_DIRS. But I don't think needing this subclass is a
problem, and I do think it's important that out of the box you can
override the default form templates, as well as adding your own; this
latter of course is needed for the admin.)

2. We also provide JinjaTemplateRenderer, which is just like
DjangoTemplateRenderer, except it uses Jinja. (I.e. we get rid of the
"automatically use Jinja2 if it's installed" behavior of the
StandaloneTemplateRenderer in the current PR, and instead split into two
different renderers for DTL vs Jinja. Magically changing behavior just
because something is installed is bad. The choice to use Jinja should be
explicitly configured, not automatic when you install it.)

3. Lastly, we provide ProjectTemplateRenderer (name up for
bikeshedding), for those who want form template rendering to use their
overall project templating settings, as defined in TEMPLATES. This
renderer just uses the stock get_template() function. If you choose to
switch to this renderer, you will need to either put 'django.forms' in
INSTALLED_APPS (and have at least one engine with APP_DIRS=True), or add
the forms directory in DIRS of one of our engines, or supply all the
form templates you need yourself, or whatever. You've chosen to take
full control, so it's your responsibility to make the form templates you
need are available.

I think this meets all the requirements:

The default renderer is backwards-compatible with no settings changes,
and does not require Jinja2, but it still allows overrides/additions of
widget templates. It's also standalone-compatible, in that it is
self-contained and has no dependency on TEMPLATES config.

Switching to Jinja2 is just a six-character change to your FORM_RENDERER
setting.

If you want complete control of how your widget templates are sourced,
switch to ProjectTemplateRenderer.

And of course you can always write your own renderer class if you need
even more control than that.

Anything I missed?

Carl

 I thought I'd try something to get the
> conversation going again. I've been reacting to some tickets based on
> the assumption that this will get into 1.11 so I'd like to make it happen.

That would be great.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7b4d51f5-6f48-acb5-a3bf-802cf7a8ea1f%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: automating Django releases

2016-11-26 Thread Carl Meyer
Hi Tim,

On 11/25/2016 11:49 AM, Tim Graham wrote:
> After doing releases about once a month for a while, I'm thinking it
> would be nice if releasing Django could be a bit more automated. As far
> as I know, the process hasn't changed too much in 10 years, while the
> state of Python packaging has improved.
> 
> Currently doing a release requires bumping the VERSION tuple in
> django/__init__.py [0], creating tarball/wheel and checksum files,
> asking someone in IRC to verify the checks look fine, and upload files
> to various locations [1]. My idea would be to use setuptools_scm [2]
> (that choice because this is the only tool I know about) to eliminate
> the need to bump/set a VERSION tuple and instead fetch the version
> information based on git tags. Something like (based on what I found in
> djanog-hosts [3]):
> 
> __version__ = pkg_resources.get_distribution('django').version

+1 for more automation, but I recommend simply automating the bumping of
the hardcoded VERSION in the source tree (should not be hard to do). I
recommend against relying on git for release versioning, and even more
strongly against depending on pkg_resources to get the right version.
It's just so much less robust and reliable than having a version
hardcoded into the source tree.

The above pkg_resources line only works if the current source tree is
also installed in the current Python environment. If you are working in
one source tree, but some other Django is installed, you will get the
wrong version. If you are working in a source tree you haven't
installed, you will get no version at all. We used to use a similar
pkg_resources-based scheme in pip, and it caused no end of trouble.

The git-based version fetching only works if you are working from a git
repo, not an exported source tree. An exported source tree, independent
of git, should still know at least what released version it represents.
I am sure if we break this, it will break somebody's workflow.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7fc5a22c-bd8c-5c4e-0d8d-dee495837d81%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Should we require pytz for timezone support in Django?

2016-10-10 Thread Carl Meyer
Hi Kevin,

On 10/09/2016 11:09 AM, Kevin Brown wrote:
> I agree with requiring pytz if you have timezone support enabled in
> Django, as that cuts out a set of edge cases and likely simplifies the
> code base a bit in the long run. And I even agree with forcing it to be
> installed with Django if we want to encourage people to use Django with
> timezone support by default.
> 
> But I don't see the value in requiring pytz to be installed whenever you
> install Django, even if you explicitly disable timezone support. This
> means that there is the potential to force pytz to be installed in cases
> where it may never be used, and at the moment isn't even required.
> 
> If there was interest in dropping support for using Django without
> timezones (so removing USE_TZ = False), then I would understand the push
> for making pytz required for everyone.

I made this same argument a while back. The main problem with it is that
the default startproject sets USE_TZ = True. That means that with your
proposal, the default experience for every new Django dev would be 1.
install Django, 2. run startproject, 3. run runserver and get an error
that pytz isn't installed. That's not acceptable. Having pytz installed
in some cases when it isn't strictly needed is a very small price to pay
for a smoother path in the common case.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f03ff8ad-a71e-ba56-df1e-42f1ef02a41c%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Logging config tries too hard

2016-09-09 Thread Carl Meyer
On 09/08/2016 11:31 PM, Ivan Sagalaev wrote:
> I'm not familiar with the current deprecation policy in Django. If you
> can point me to it, I could probably carve some time in the nearby
> future and prepare a pull request.

Here is the deprecation policy:
https://docs.djangoproject.com/en/dev/internals/release-process/#deprecation-policy

As it applies to this case, basically we need to ensure that people's
existing logging config continues to function as it does now in the next
release of Django. If they need to make updates to their settings in
order to preserve the same logging behavior, we need a deprecation
warning raised that will alert them to to the need for this change (and
they should be able to silence the deprecation warning by making the
needed change).

I think there's probably some wiggle room in the definition of
"continues to function as it does now." As you pointed out, it's
probably OK if a logger that someone has currently silenced with
disable_existing_loggers=True starts giving output in the next version -
at least that's a change that they are likely to notice and can deal
with as desired. (It should still be noted in the release notes). But
it's a big problem if someone were to silently stop getting logging
output that they currently get.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/492f2fa0-f91c-3730-0821-5bef2376d1f3%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Logging config tries too hard

2016-09-08 Thread Carl Meyer
Hi Ivan,

> On Sep 8, 2016, at 9:00 PM, Ivan Sagalaev  wrote:
> 
> Actually… I'm failing to imagine a scenario where such a change would lead to 
> any bad things:

You may be right, but it's hard to evaluate that without knowing precisely what 
change you are envisioning. Can you be more specific about exactly what change 
you propose?

It sounds like you are maybe proposing to decide whether to pre-initialize 
logging with the built-in default config or not based on whether the user's 
LOGGING setting has disable_existing_loggers set to True or False? I think that 
would have the backwards-compatibility properties you outline below, but it 
doesn't seem to me that it arrives at a very satisfying end result; the system 
becomes if anything more complex and harder to explain than it is now. 

I think the best end result would be one where LOGGING simply defines the full 
config and it is always applied (by Django) exactly once, and the defaults we 
want are set as the global default for LOGGING, and just documented so that 
people who want to set LOGGING themselves can easily copy them as a starting 
point. But we'll need some kind of deprecation path to get there, at least for 
people who currently have disable_existing_loggers=False, in order to prevent 
them from suddenly losing the default config. 

Carl

> - If `disable_existing_loggers` is True, the incompatibility would manifest 
> itself in Django loggers not being silenced, provided the user has defined a 
> root logger. Which is probably the intended behavior, and I'm having a hard 
> time imagining someone actually relying on it. Most probably users currently 
> work around it either by doing their own config from scratch, or by 
> redefining Django loggers, both of which will continue working after the 
> change.
> 
> - If `disable_existing_loggers` is explicitly set to False, then we could 
> initialize them as we do now.
> 
> How does that sound?
> 
>> On Tuesday, September 6, 2016 at 5:44:37 PM UTC-7, Carl Meyer wrote:
>> On 09/06/2016 04:57 PM, Ivan Sagalaev wrote: 
>> > I'm 
>> > talking about this function: 
>> > https://github.com/django/django/blob/master/django/utils/log.py#L66 
>> > 
>> > When `LOGGING_CONFIG` isn't explicitly disabled it first unconditionally 
>> > initializes logging with the hard-coded configuration and then applies a 
>> > user one, if present. And the latter can't really neutralize the effect 
>> > of the former. 
>> 
>> FWIW I agree that the current behavior here is broken and should be 
>> fixed. The design seems to be based in an assumption that 
>> `disable_existing_loggers=True` can "undo" the effects of the previous 
>> config, when it can't, and in fact it does something much nastier -- it 
>> silences all previously-created loggers. 
>> 
>> I clearly recall outlining this issue in a Trac ticket at some point 
>> quite a while ago, but I can't seem to find that ticket now. 
>> 
>> Fixing this in a way that doesn't change logging behavior for projects 
>> upgrading from a previous Django version may be tricky, unless we gate 
>> the effect of the change on (yet another) setting. 
>> 
>> Carl
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/78325f82-badf-4e75-9206-be244a8e9cfc%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/F00ABFF1-803D-429C-A0B5-0E5E4164D937%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


Re: Logging config tries too hard

2016-09-06 Thread Carl Meyer
On 09/06/2016 04:57 PM, Ivan Sagalaev wrote:
> I'm
> talking about this function:
> https://github.com/django/django/blob/master/django/utils/log.py#L66
> 
> When `LOGGING_CONFIG` isn't explicitly disabled it first unconditionally
> initializes logging with the hard-coded configuration and then applies a
> user one, if present. And the latter can't really neutralize the effect
> of the former.

FWIW I agree that the current behavior here is broken and should be
fixed. The design seems to be based in an assumption that
`disable_existing_loggers=True` can "undo" the effects of the previous
config, when it can't, and in fact it does something much nastier -- it
silences all previously-created loggers.

I clearly recall outlining this issue in a Trac ticket at some point
quite a while ago, but I can't seem to find that ticket now.

Fixing this in a way that doesn't change logging behavior for projects
upgrading from a previous Django version may be tricky, unless we gate
the effect of the change on (yet another) setting.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/21d52615-35ac-4091-92c9-4ef14fbd4e0f%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Adding support for importing models through the app registry while it is loading

2016-09-06 Thread Carl Meyer
On 09/06/2016 12:55 PM, Aymeric Augustin wrote:
> Hi Carl,
> 
> Thanks for the feedback!

Of course! Thanks for working on things :-)

> ...
> The change I’m proposing doesn’t introduce random bugs. If models are
> missing reverse relations, that will be deterministic.

+1

>> Is it possible
>> to make it so that even the model meta introspection APIs (and of course
>> also any attempt to query the db) fail quickly, loudly, and with a clear
>> error message if the app registry is not yet fully populated? If so,
>> then I think there's little risk to adding this API (and in fact I think
>> we could even make it the default behavior of `get_model`).
> 
> If not checking for models_ready was the default for get_model(), then I’d
> expect it to be the default for get_models() as well, but that doesn’t make
> sense. get_models() really needs all models to be imported.
> 
> My initial patch added a new method, `import_model`, because the operation is
> quite different from the app registry's perspective. Instead of just looking
> up the model, we try to import it. Even though the end result feels the same,
> the context isn't the same.

Yes, this makes sense. It suggests another possible name for the kwarg,
`force_import=True`?

...
> Without looking at the code base, I can’t say. It might help if you’re doing
> swappable-model-style dynamic imports. It won’t help if you’re importing
> models directly or indirectly from your applications’s __init__.py, which is
> a more common problem.

Yes, I think it's likely this wouldn't help much.

> I think #21719 would have led there as well but it turned out to be
> surprisingly hard -- in the realm of "not sure I'd get anywhere even if I
> spent two weeks full-time on it”.

I still think that's the right approach (and should be possible) in
theory, but I trust your estimation of complexity and am currently short
of two-week-plus blocks of free time :-)

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7d1c5071-b1c5-6fdb-e12c-7d7ebf3cb852%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Adding support for importing models through the app registry while it is loading

2016-09-06 Thread Carl Meyer
Hi Aymeric,

On 09/04/2016 01:03 PM, Aymeric Augustin wrote:
> Hello,
> 
> Since Django 1.7, `get_user_model()` cannot be called at import time.
> Django contains a lot of methods that start with `UserModel =
> get_user_model()` while it would be more natural to declare it as a
> global variable in the module. This trips up users and the reason why
> it’s forbidden isn’t obvious.
...
> However, in many cases, it doesn’t matter whether the model is fully
> constructed or not. In the use case I described in my introduction,
> the code only needs a reference to the model class at import time;
> that reference won’t be used until run time, after the app-loading
> process has completed. (Performing SQL queries through the ORM at
> import time doesn’t work anyway.)
> 
> For this reason, I think it would make sense to add an API similar to
> `apps.get_models()`, but that would work while app-loading is still
> in progress. It would make no guarantees about the functionality of
> the class it returns until app-loading has completed.
> 
> I implemented a proof of concept here:
> https://github.com/django/django/pull/6547.

A few thoughts:

1) A kwarg to `get_model` seems fine, but I don't like the vague FUD of
`unsafe=True` (if it's really "not safe" we shouldn't add it / make it
public at all). How about something more specific like
`require_ready=False`?

2) I think a key question here is the nature and clarity of the problems
that arise if you try to make use of an un-ready model class. If the
failure mode here is the reintroduction of unpredictable heisenbugs
where certain related fields are quietly not present because the model
on the other end happens to not have been imported yet, I think that's a
strong argument for refraining from introducing this API. Is it possible
to make it so that even the model meta introspection APIs (and of course
also any attempt to query the db) fail quickly, loudly, and with a clear
error message if the app registry is not yet fully populated? If so,
then I think there's little risk to adding this API (and in fact I think
we could even make it the default behavior of `get_model`).

As for whether it's needed in real use, the only feedback I can offer at
the moment is that the import-order enforcement that kicked in with the
removal of app-loading deprecation shims in Django 1.9 is the primary
reason that Instagram has so far upgraded only as far as Django 1.8;
fixing the import-order issues appeared to require too much upheaval and
would have delayed the upgrade too much. I haven't had time to look
closely at those issues so I can't yet offer more detailed feedback on
how much the availability of this particular API would help.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/eabf51d0-712e-32f1-d8e4-320ddf7681ad%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: GitHub PR status within Trac

2016-07-22 Thread Carl Meyer
Hi Tobias,

On 07/22/2016 04:53 PM, Tobias McNulty wrote:
> I spent some time during the DjangoCon sprint today looking into
> dashboard.djangoproject.com  and how
> it calculates metrics. I was hoping to add some new metrics that mash up
> data from GitHub and Trac together. While technically possible, this
> breaks down when you want to link out to a list of the related tickets
> in Trac. For example:
> 
>   * A list of Accepted tickets with no open PR or an open PR that hasn't
> t been touched in X months
>   * A list of Accepted tickets with no PR and no attached patch that
> haven't been touched in  months
> 
> This got me wondering: Is checking for GitHub PRs via JavaScript the
> Right Way to do it? What if we had a cronjob update Trac periodically
> with PR status from GitHub?
> 
> I think it would be valuable to be able to query on PR status from
> within Trac, e.g., to help find in progress but stale/abandoned tickets.
> Cleaning up the work of someone else who's lost interest in a patch is
> often a good way to get into Django development.
> 
> I'm sure there are some holes in this idea, so I'm putting it out there
> for comment. Was something like this considered before, and if so, why
> wasn't it pursued?
> 
> If it hasn't been considered before, what are the obvious problems I
> might encounter?

Others know these systems better than I do, but just a couple thoughts:

1) While being able to query Trac by PR status would be useful, losing
the immediate feedback of "I just created my PR and reloaded the Trac
ticket, and there it is!" would be a significant loss, I think. Delays,
even of just a few minutes, in that sort of UI tend to introduce an "is
this actually working" uncertainty that leads to extra support queries.
So maybe even if you implemented something on the backend, we shouldn't
get rid of the JS code? Or maybe github push hooks could be used to keep
update latency low?

2) I think it was done this way originally because whoever did it was
scared of touching Trac's Python code (with reason), and it was simpler
to just do it in JS, not for any deeper reason.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e70012b5-dd03-526a-6f99-a03470332bae%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: change commit message format to present tense?

2016-06-24 Thread Carl Meyer
On 06/24/2016 03:04 PM, Tim Graham wrote:
> With the idea of saving characters in the first line, would "Fix #XXX:
> Message" be better than ""Fix #XXX -- Message" also? This saves two
> characters without any loss of readability as far as I see.

Seems reasonable to me.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/576DA5B8.5090404%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: change commit message format to present tense?

2016-06-24 Thread Carl Meyer
On 06/24/2016 12:15 PM, Jon Dufresne wrote:
> On Fri, Jun 24, 2016 at 10:48 AM, Carl Meyer  <mailto:c...@oddbird.net>> wrote:
> 
> To be clear, the recommended git style is not present tense, it is
> imperative mood. So it should _not_ be "Fixes #12345 -- Regulates the
> frobnicator", it should be "Fix #12345 -- Regulate the frobnicator."
> 
> 
> Do you have a link to an authoritative source stating this as the
> recommended style or is it just common knowledge?
> 
> I'm not arguing against the proposal (in fact, I agree with it), I'm
> just curious if there is documentation to support one style over the other.

The style that is most commonly recommended is described in the most
detail (with reasoning) in these blog posts:

http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
http://chris.beams.io/posts/git-commit/

That same style is recommended in the Git Book in
https://git-scm.com/book/ch5-2.html

It is "endorsed" by git itself in that git uses the imperative mood when
it generates a commit message (e.g. for a merge) -- it generates "Merge
branch foo" not "Merges branch foo" or "Merged branch foo."

Also the git project's own commit messages use this style.

Carl




-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/576D7AB0.3040502%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: change commit message format to present tense?

2016-06-24 Thread Carl Meyer
To be clear, the recommended git style is not present tense, it is
imperative mood. So it should _not_ be "Fixes #12345 -- Regulates the
frobnicator", it should be "Fix #12345 -- Regulate the frobnicator."

Carl

On 06/24/2016 11:45 AM, Tobias McNulty wrote:
> I'm in support of this as well. Is the suggestion to change the format to:
> 
> A) Fixes #12345 -- Add support for ... / Validate that  / etc.
> 
> OR
> 
> B) Fixes #12345 -- Adds support for ... / Validates that ... / etc.
> 
> OR
> 
> C) Something else?
> 
> I assume (A) but others may interpret this differently?
> 
> On Fri, Jun 24, 2016 at 1:25 PM, Markus Holtermann
> mailto:i...@markusholtermann.eu>> wrote:
> 
> I don't mind either way. If everybody seems to use present tense these
> days, yeah, let's do that as well. As long as the general style of the
> commit messages stays: Fixes|Refs #12345 -- Make it work
> 
> My 2¢
> 
> /Markus
> 
> 
> On Fri, Jun 24, 2016 at 11:04:39AM -0600, Jacob Kaplan-Moss wrote:
> 
> I'm not entirely sure because my memory sucks, but odds are that
> I started
> the current standard of using past-tense.
> 
> FWIW I no longer care even at all, I think as long as commit
> messages are
> clear we I don't care what tense they are. Following the
> standard git way
> seems totally OK to me.
> 
> Jacob
> 
> On Fri, Jun 24, 2016 at 10:59 AM, Carl Meyer  <mailto:c...@oddbird.net>> wrote:
> 
> On 06/24/2016 10:55 AM, Tim Graham wrote:
> > A few contributors indicated in #django-dev that it could
> be beneficial
> > to change our commit message format to match git's
> guidelines of present
> > tense (instead of our current past tense convention) as it
> would be one
> > less thing to remember when contributing to Django.
> Besides consistency
> > with the current history, do you feel there are any
> benefits in
> > continuing to use past tense instead of present tense?
> 
> I was one of those contributors in #django-dev. Not only are
> past-tense
> commit messages non-standard for git (meaning they are one
> more thing a
> new contributor is likely to trip over), I also personally
> find them
> harder to write and make clear. Sometimes in a commit
> message you need
> to reference past (pre-commit) behavior and make it clear
> how the commit
> changes that behavior. I occasionally find that harder to do
> clearly
> when the entire commit message is supposed to worded in the
> past tense.
> 
> So I find all the advantages (except for consistency with
> past history)
> in favor of switching to the imperative mood.
> 
> Carl
> 
> --
> You received this message because you are subscribed to the
> Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an
> email to django-developers+unsubscr...@googlegroups.com
> <mailto:django-developers%2bunsubscr...@googlegroups.com>.
> To post to this group, send email to
> django-developers@googlegroups.com
> <mailto:django-developers@googlegroups.com>.
> Visit this group at
> https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> 
> https://groups.google.com/d/msgid/django-developers/576D6702.3080501%40oddbird.net
> .
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the
> Google Groups "Django developers  (Contributions to Django
> itself)" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to
> django-developers+unsubscr...@googlegroups.com
> <mailto:django-developers%2bunsubscr...@googlegroups.com>.
> To post to this group, send email to
> django-developers@googlegroups.com
> <mailto:django-developers@googlegroups.com>.
>

Re: change commit message format to present tense?

2016-06-24 Thread Carl Meyer
On 06/24/2016 10:55 AM, Tim Graham wrote:
> A few contributors indicated in #django-dev that it could be beneficial
> to change our commit message format to match git's guidelines of present
> tense (instead of our current past tense convention) as it would be one
> less thing to remember when contributing to Django. Besides consistency
> with the current history, do you feel there are any benefits in
> continuing to use past tense instead of present tense?

I was one of those contributors in #django-dev. Not only are past-tense
commit messages non-standard for git (meaning they are one more thing a
new contributor is likely to trip over), I also personally find them
harder to write and make clear. Sometimes in a commit message you need
to reference past (pre-commit) behavior and make it clear how the commit
changes that behavior. I occasionally find that harder to do clearly
when the entire commit message is supposed to worded in the past tense.

So I find all the advantages (except for consistency with past history)
in favor of switching to the imperative mood.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/576D6702.3080501%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: decorator_from_middleware changes

2016-06-21 Thread Carl Meyer
Hi Shai,

Apologies, I didn't fully process your suggestion or the reason for the
flag the first time. I think it would work, though I'd be tempted to set
the flag to False on the transition middleware by default, to provide
perfect back-compat for existing uses of decorator_from_middleware until
someone opts in by setting the flag to True or writing their own
new-style middleware that doesn't inherit the mixin.

I'm still not entirely happy with this approach for the reason Tobias
highlighted, though - it doesn't behave like someone might expect a view
decorator to behave, and I think probably it should have an API that
clarifies that what it's really doing is applying per-view middleware.

Still halfway tempted to go with the original "make middleware authors
responsible for this themselves, if they want to access
response.content" approach.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5769C70E.1010308%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: decorator_from_middleware changes

2016-06-21 Thread Carl Meyer
On 06/21/2016 10:53 AM, Carl Meyer wrote:
> If we're OK in the long term with making middleware-turned-decorator
> authors responsible for correctly handling TemplateResponse themselves,
> we should just go there right away via
> https://github.com/django/django/pull/6765 -- we can take care of
> backwards-compatibility quite nicely there in the transition mixin. In
> fact I'd say the overall backward-compatibility of this approach (#6765)
> is better than that of #6805 (the extra-middleware-annotation approach),
> due to converting all our builtin decorators-from-middleware to the new
> style in the latter.
> 
> The downside of #6765 is that the boilerplate for correctly handling
> TemplateResponse is pretty ugly [1], and any middleware that accesses
> response.content would pretty much need to do that, in case it might
> sometime be used in decorator_from_middleware.
> 
> To be honest though, I'm still not sure that isn't our best option. Most
> middleware I've written don't access response.content, they set headers,
> so they'd be unaffected. Some of Django's built-in ones do touch
> content, but we can deal with the ugly TemplateResponse boilerplate in
> our own middleware.

We could also leave MiddlewareMixin in place indefinitely, and people
who don't want to do advanced things in __call__ (like using context
managers) can keep inheriting from the mixin and writing process_request
and process_response methods until eternity. Then the mixin takes care
of TemplateResponse deferral and they don't need to worry about it.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57697232.7010501%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: decorator_from_middleware changes

2016-06-21 Thread Carl Meyer
Hi Tobias,

On 06/21/2016 06:22 AM, Tobias McNulty wrote:
> On Mon, Jun 20, 2016 at 10:51 AM, Carl Meyer  <mailto:c...@oddbird.net>> wrote
>> Possible disadvantages of this approach:
> 
>  
> 
>> 3. The new behavior may surprise some people accustomed to the old
>> behavior, since it means the effect of the middleware decorator won't
>> occur immediately where the decorator is applied. E.g. if you apply a
>> middleware decorator to a view, and then another custom view decorator
>> outside that, the custom view decorator won't see the effects of the
>> middleware decorator in the actual response (it would only see a new
>> entry in view_func._extra_middleware).
> 
> This one seems like the gotcha to me.

Yeah, I agree.

> What if, for example, I have a
> view decorator whose effects I specifically don't want to cache, but I
> do want to cache the underlying view; is it fair to require that the
> person write a middleware and then turn it into a decorator to be able
> to do that?

Caching happens to be a bad example, because AFAICT nobody should ever
use the cache_page decorator (or cache full responses right after the
view function, as you describe here) due to
https://code.djangoproject.com/ticket/15855 -- unless I suppose they are
pre-adding all the correct Vary headers themselves.

But I think the general point is a good one nonetheless.

> Are we effectively saying, "for view decorators to behave
> the way you might expect, implement them as middleware"? It seems odd,
> to me at least, that I should care what the underlying implementation of
> a decorator is before I use it. It also violates the 'strict in/out
> layering' goal, albeit for decorators instead of middleware. (A similar
> argument could be said of exceptions -- what if I want to trap an
> exception raised by a middleware-turned-decorator?)
> 
> It might be okay if the decorators themselves were explicit about what
> they were doing, for example @cache_page(3600) might become:
> 
> @add_middleware(CacheMiddleware, cache_timeout=3600)

Yes, I did think about this. It makes it more difficult to use, but it
does give a clearer picture of what's actually happening. We aren't
directly decorating the view function, we're just annotating it with an
extra per-view middleware.

> However, that's probably a bigger and unnecessary change.
> 
> Would it be possible to implement a combination of the approaches, i.e.,
> make the delay in applying the middleware-turned-decorator the exception
> rather than the rule, perhaps only for TemplateResponses and
> specifically for the purpose of supporting a deprecation plan? And then,
> long-term, leave it up to middleware/decorator authors & users to decide
> how best to implement/layer them, being explicit about the implications
> of rendering the response or perhaps more appropriately, "not rendering
> if you can avoid it" (i.e., your first strategy)?

I don't think it's worth doing this just for a deprecation path. If
we're OK in the long term with making middleware-turned-decorator
authors responsible for correctly handling TemplateResponse themselves,
we should just go there right away via
https://github.com/django/django/pull/6765 -- we can take care of
backwards-compatibility quite nicely there in the transition mixin. In
fact I'd say the overall backward-compatibility of this approach (#6765)
is better than that of #6805 (the extra-middleware-annotation approach),
due to converting all our builtin decorators-from-middleware to the new
style in the latter.

The downside of #6765 is that the boilerplate for correctly handling
TemplateResponse is pretty ugly [1], and any middleware that accesses
response.content would pretty much need to do that, in case it might
sometime be used in decorator_from_middleware.

To be honest though, I'm still not sure that isn't our best option. Most
middleware I've written don't access response.content, they set headers,
so they'd be unaffected. Some of Django's built-in ones do touch
content, but we can deal with the ugly TemplateResponse boilerplate in
our own middleware.

Carl


 [1]
https://github.com/carljm/django/blob/7d999844bd4b43b82a472634074156c24a4fc7cb/docs/topics/http/middleware.txt#L304

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/576970EC.50704%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: decorator_from_middleware changes

2016-06-21 Thread Carl Meyer
Hi Shai,

On 06/20/2016 11:07 PM, Shai Berger wrote:
> Well, we can change decorator_from_middleware to introspect the middleware 
> itself and decide accordingly:
> 
> -- if it does not have process_request or process_response as callable 
> attributes, and it is callable itself, it is a new-style middleware 

"Does not have process_request and process_response as callable
attributes" isn't reliable -- there's nothing preventing a new-style
middleware from having such methods and calling them from `__call__` (in
fact any middleware converted using MiddlewareMixin will fit this
description).

> -- if it does, we can define a new attribute '_dep5' (or something) whose 
> presence will indicate a dep5-compatible middleware. This attribute can be 
> defined on the transition mixin, so that people who use it for transition 
> don't 
> have to care
> 
> -- if it has callable process_request/process_response and doesn't have _dep5 
> then it is an old-style middleware
> 
> Note that decorator_from_middleware already used to instantiate the 
> middleware 
> class anyway, so the check can be applied to a middleware object, not to a 
> middleware factory.
> 
> The whole introspection behavior, including the _dep5 attribute and the code 
> that checks for it, can be dropped when old-style middleware is finally 
> dropped. If someone uses _dep5 manually, it will do them no harm later.

We can indeed introspect to try to auto-detect. It's a bit more complex
due to the different `__init__` signature as well, and it would fail in
case someone defined __call__ on an old-style middleware for some
reason, but I've already written the code for this in
https://github.com/django/django/pull/6765

The main reason I abandoned auto-detection for this second attempt is
that the behavior of these new-style view decorators is different enough
that I'd like to provide better flexibility for third-party projects.
Imagine a third-party project that currently provides both a middleware
and a view decorator, created by decorator_from_middleware. Now they are
adding Django 1.10 compatibility. If we auto-detect, and they inherit
their middleware from the transition mixin so it's cross-compatible,
decorator_from_middleware will suddenly detect it as new-style, leaving
them no way to continue to provide their old decorator with fully
backwards-compatible behavior.

Basically, I think middleware_decorator does something that's
fundamentally different enough from decorator_from_middleware that it is
clearer to make it a separate function and let people make explicit
choices about which one they are using.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57696C0F.5050804%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: decorator_from_middleware changes

2016-06-20 Thread Carl Meyer
On 06/20/2016 06:57 PM, Tim Graham wrote:
> I may have seen this idea in IRC but instead of a new
> middleware_decorator function did you consider varying
> decorator_from_middleware's behavior based on "if settings.MIDDLEWARE is
> None" similar to what BaseHandler does? We could add a check in
> decorator_from_middleware to raise an error if MIDDLEWARE is None and
> the middleware doesn't seem to be the new-style.

I did think about that, but I think having the output of a function like
this (which is usually called at import time) be non-deterministic
depending on settings is problematic on several levels.

At the most basic level, we usually try to avoid import-time settings
access, to avoid an import causing a "DJANGO_SETTINGS_MODULE is not
defined" error.

But it also seems wrong for a decorator-generator to generate a
completely different decorator based on settings. It's not uncommon that
a third-party library might provide both a middleware and a decorator
generated via decorator_from_middleware. Should the nature and behavior
of that third-party decorator change fundamentally depending on whether
it happens to be imported in a project that has (or hasn't) converted
from MIDDLEWARE_CLASSES to MIDDLEWARE? What about a third-party
old-style decorator from on old-style middleware, imported in a project
using MIDDLEWARE? There's no reason that decorator can't work the same
as it always has (the decorator itself doesn't have any dependency on
which type of middleware your project uses), but under this proposal it
would instead suddenly raise an error if you even try to import it.

I think this approach is more problematic and less backwards-compatible
(especially for third-party projects) than just introducing a new
function and deprecating the old one.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57689D2E.3060600%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


decorator_from_middleware changes

2016-06-20 Thread Carl Meyer
Hi all,

In porting decorator_from_middleware to support DEP 5 middleware, I ran
into one stumbling block: TemplateResponse. Normally with old-style
middleware, process_response runs after TemplateResponses have been
rendered. But a view decorator runs immediately wrapping the view, when
TemplateResponses will not have been rendered yet. So
decorator_from_middleware has special support for TemplateResponse -- if
it detects one, rather than calling process_response immediately, it
sets it as an on_render_callback.

But in DEP 5 middleware, request and response processing are not
separable -- they are part of the same function. This is the source of
the benefits of DEP 5: strict in/out layering, the ability to use
context managers in middleware, etc. But it also means that there is no
way to delay just the response-processing portion of the middleware
until after render.

My initial strategy [1] was to simply make middleware authors
responsible for this: a middleware that needed to access
response.content and wanted to be compatible with
decorator_from_middleware would be responsible to detect and handle
unrendered TemplateResponse appropriately.

In discussion in IRC, Florian Apolloner suggested an alternative: change
the implementation of decorator_from_middleware so that instead of
immediately applying all middleware methods as an actual view decorator,
make the decorator simply annotate the view callable with an
`_extra_middleware` list, and then add support for this annotation to
the base handler (effectively making it support per-view middleware).
The handler runs per-view middleware as if it were listed last in
MIDDLEWARE (that is, "closest" to the view). I've implemented this
suggestion in [2]. It has several advantages:

1. It consolidates middleware-handling logic in one place, the handler.
Currently this logic is effectively duplicated in the handler and in
decorator_from_middleware, and I've noticed several bugs in the current
decorator_from_middleware resulting from inconsistencies between the
duplicate implementations (e.g. if process_view or process_exception
returns a response, normally that response passes through
process_response, but in decorator_from_middleware it does not.)
Consolidation of this logic will improve consistency of middleware
behavior when listed in settings vs when applied via decorator.

2. It allows us to apply middleware-decorators in closer to the same way
that normal middleware is applied, e.g. response-handling can happen
after render() is called on TemplateResponses, solving the initial problem.

Possible disadvantages of this approach:

1. In order to avoid immediate breakage of decorator_from_middleware
when used with third-party middleware that don't yet support DEP 5, I've
implemented the new approach as a new function, `middleware_decorator`,
that will replace and deprecate `decorator_from_middleware`. This is
unfortunate churn, but I don't see another good way to be
backwards-compatible.

2. The decorators internal to Django that rely on
decorator_from_middleware (the csrf decorators, cache_page) will be
immediately converted to middleware_decorator, resulting in subtle
behavior differences from the current ones.

3. The new behavior may surprise some people accustomed to the old
behavior, since it means the effect of the middleware decorator won't
occur immediately where the decorator is applied. E.g. if you apply a
middleware decorator to a view, and then another custom view decorator
outside that, the custom view decorator won't see the effects of the
middleware decorator in the actual response (it would only see a new
entry in view_func._extra_middleware).

Thoughts welcome.

Carl

 [1] https://github.com/django/django/pull/6765
 [2] https://github.com/django/django/pull/6805

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57680303.8080002%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


feedback needed re exception handling in DEP 5 middleware

2016-06-13 Thread Carl Meyer
Hi all,

Any of you with a particular interest in middleware, DEP 5, and handling
of view exceptions, please take a look at the discussion on
https://github.com/django/django/pull/6618 and if you have strong
feelings about what route we should take, please weigh in either there
or here. (The code in that PR is less relevant, it doesn't represent the
latest plan of action that we need feedback on, which is outlined in the
discussion thread.)

Thanks,

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/575F2994.1000906%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Clearing prefetch related on add(), change(), remove()

2016-06-07 Thread Carl Meyer
On 06/07/2016 06:11 AM, Marc Tamlyn wrote:
> I may be "too close" to knowing the implementation of this feature to be
> able to comment on whether the behaviour is surprising to most people,
> but it doesn't surprise me. It's certainly analogous to that when you do
> `MyModel.objects.create()` it doesn't change an already executed
> queryset. There's a question of where you draw the line as well - what
> about `related_set.update()`?
> 
> I think it's worth documenting the behaviour, also noting that you can
> "force" the execution of a new queryset by chaining another .all().

Hmm, I have the opposite instinct. I don't find it analogous to the case
of some unrelated queryset object failing to update its internal cache
when the database changes. In this case we have a related-manager with
an internal cache, and we make changes _via that same manager_. I find
it quite surprising that the manager doesn't automatically clear its
cache in that case.

A much stronger precedent, I think, is the fact that Queryset.update()
does clear the queryset's internal result cache. In light of that, I
think the current behavior with prefetched-data on a related manager is
a bug that should be fixed (though it certainly should be mentioned in
the release notes).

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/575708A1.902%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Conditions DEP

2016-06-06 Thread Carl Meyer
Hi Connor,

On 06/06/2016 02:25 PM, Connor Boyle wrote:
> Should I stop working on the DEP then?

That's up to you, if it's useful to you as a design document. Obviously
there's no need for a DEP for you to release your own project. If the
project is a success and later becomes a candidate for inclusion in
core, the DEP might be useful then (but would likely need updating to
reflect experience gained and design changes in the meantime).

> Also, would it be at all
> appropriate to continue using this mailing list (for say, updates on the
> project and requests for review)?

Until you're at a point where you're ready to request consideration of
your project for inclusion in core, probably django-users would be a
better venue (also one with a much bigger audience!) Unless of course
you run into limitations of Django that prevent you from implementing
some part of your concept externally, then discussion of addressing that
limitation would certainly belong here.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5755E037.5060207%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Conditions DEP

2016-06-06 Thread Carl Meyer
Hi Connor,

On 06/05/2016 07:52 PM, Connor Boyle wrote:
> I'd like to make a proposal for the addition of a major new feature to
> Django. Back in late March of this year, I wrote an original Google
> Summer of Code proposal
>  that
> received some interest from several members of the community, but
> neither I nor this proposal were selected for GSoC. A community member
> (iirc it was Florian Apolloner) told me to put the proposal in the form
> of a DEP.

I read through the DEP - it's an interesting idea. I'm not seeing
anything in the proposal that inherently needs to be part of Django core
in order to be useful; it seems that it could just as well be
implemented as an installable library. When there's a new feature idea
that _can_ live external to Django core, we generally prefer that it
_does_ live external to core, at least for a while to prove the design
and demonstrate its practical advantage over similar alternatives.
Living outside core allows you to iterate much more rapidly on the
design, with new releases as often as you need them.

If in the future it becomes the clear winning implementation of a
pattern that is important enough that it should be available to all
Django users by default, we may decide to bring it into core.

Note that this isn't just a theory or something we tell people, it's
something that actually happens regularly, including with proposals from
core team members. See for example South, django-security,
django-transaction-hooks, django-channels...

So my recommendation is that you implement your proposal as a
pip-installable library and advertise it, see whether it gains traction
that way, and see what you learn about the design from having it
exercised by real users.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5755A614.7090404%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Should we require pytz for timezone support in Django?

2016-06-05 Thread Carl Meyer

> On Jun 5, 2016, at 6:09 PM, Josh Smeaton  wrote:
> 
> It sounds like we have consensus that we can make pytz a required dependency 
> when USE_TZ = True, and clear up the associated documentation. 
> 
> I think now we should try and find (rough) consensus on whether or not we 
> should have pytz as a required dependency in `install_requires`.
> 
> > If there is a significant number, and we want to continue to support their 
> > use-case, it's a non-zero added burden to require them to pull in a 
> > frequently-updated useless dependency. 
> 
> Let's assume there is a reasonable number of deployments with and without 
> timezone support. Even then I'm personally in favour of requiring pytz. 
> 
> django-admin.py startproject defaults to USE_TZ = True which means that new 
> projects that install django, start the project, and then run the internal 
> web server are going to get missing dependency problems.

That's an excellent point. I agree, we should just add pytz to install_requires 
if we require it for USE_TZ=True, otherwise the default new-project experience 
is unpleasant. 

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/12B46FC5-1326-4938-B611-86A7E195347A%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


Re: Should we require pytz for timezone support in Django?

2016-06-04 Thread Carl Meyer
On 06/04/2016 12:38 PM, Carl Meyer wrote:
> I just realized that I think something's gotten lost in this thread:
> Josh's original point was about requiring pytz _if you want timezone
> support_ (that is, if USE_TZ=True). That would I presume involve raising
> ImproperlyConfigured (or maybe having a site check raise an error?) if
> USE_TZ=True and pytz is not installed. I'd support this for sure: the
> "well, it might work OK without pytz if you're on Postgres" caveat is
> extra complexity for not much gain.
> 
> But this is not the same thing as putting pytz in install_requires,
> which makes it a required dependency for everyone using Django, even if
> they want USE_TZ=False.
> 
> I don't know how many USE_TZ=False projects there are. I always set
> USE_TZ=True, but I can imagine some small locally-oriented sites might
> have reason to prefer USE_TZ=False? If there is a significant number,
> and we want to continue to support their use-case, it's a non-zero added
> burden to require them to pull in a frequently-updated useless dependency.

FWIW: https://github.com/search?q=USE_TZ+%3D+False&type=Code

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57532EEC.7000700%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Should we require pytz for timezone support in Django?

2016-06-04 Thread Carl Meyer
I just realized that I think something's gotten lost in this thread:
Josh's original point was about requiring pytz _if you want timezone
support_ (that is, if USE_TZ=True). That would I presume involve raising
ImproperlyConfigured (or maybe having a site check raise an error?) if
USE_TZ=True and pytz is not installed. I'd support this for sure: the
"well, it might work OK without pytz if you're on Postgres" caveat is
extra complexity for not much gain.

But this is not the same thing as putting pytz in install_requires,
which makes it a required dependency for everyone using Django, even if
they want USE_TZ=False.

I don't know how many USE_TZ=False projects there are. I always set
USE_TZ=True, but I can imagine some small locally-oriented sites might
have reason to prefer USE_TZ=False? If there is a significant number,
and we want to continue to support their use-case, it's a non-zero added
burden to require them to pull in a frequently-updated useless dependency.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57532E4E.7090504%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Should we require pytz for timezone support in Django?

2016-06-04 Thread Carl Meyer
On 06/03/2016 03:55 PM, Philip James wrote:
> Is there consensus on making pytz required? (I, personally, am in favor,
> since I too can't remember the last project I did that didn't need it.)
> 
> If it's required, should it be vendored?

-1 on vendoring pytz. I think it's fine to just make it a required
dependency if that saves us significant work on a valuable new feature
or refactoring. I don't really see the hurry to make it a required
dependency just for its own sake, though.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57532114.20003%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-05-17 Thread Carl Meyer
On 05/17/2016 01:33 AM, Claude Paroz wrote:
> I can imagine that django.forms would then be responsible to tell the
> template engine to add the template source. Maybe this would need a new
> API in the template code?

Yes, it's certainly possible that we could address this with a new API
in the template engines. That's a more significant change than I had
thought we needed, but it may be the best option.

I'm happy to work on this issue more, but it's not likely to be until
the PyCon sprints.

Carl



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/573C1083.8030304%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-05-17 Thread Carl Meyer
Hi Simon,

On 05/17/2016 07:28 AM, charettes wrote:
> Did we consider defining a template loader that knows where to load the
> widget
> templates from instead of requiring 'django.forms' in INSTALLED_APPS with
> 'APP_DIRS': True in TEMPLATES?
> 
> Something along theese lines make more sense to me:
> 
> TEMPLATES = {
> {  
> 'BACKEND': 'django.template.backends.django.DjangoTemplates',
> 'DIRS': [],
> 'APP_DIRS': True,
> 'OPTIONS': {
> 'loaders': [
> 'django.forms.templates.loader.WidgetTemplateLoader',
> ]
> },
> },
> }

I think the solution needs to be at a higher level than DTL loaders,
because it should work also for Jinja2.

And this approach doesn't really help solve the backwards-compatibility
path, unless we'd automatically insert such a loader if it's not listed
explicitly.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/573C0FF5.5040801%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-05-16 Thread Carl Meyer
On 05/16/2016 01:48 AM, Claude Paroz wrote:
> Le samedi 14 mai 2016 16:03:57 UTC+2, Tim Graham a écrit :
> 
> (...)
> 
> I guess it will affect every project that uses the admin. I can't
> think of a simple solution other than adding a system check upgrade
> warning to detect this situation ('django.contrib.admin' in
> INSTALLED_APPS but not 'django.forms') and advise the user to add
> 'django.forms' to INSTALLED_APPS. Thoughts?
> 
> 
> I'm still answering with my naive hat: isn't it possible to simply
> always consider forms in django.forms without requiring anything new in
> INSTALLED_APPS? What would be the problem? (Sorry if I missed a
> discussion about that).

This is certainly possible, it's just (as Tim says) not clear where
exactly one would implement it without introducing an ugly bidirectional
coupling between forms and templates. It's one thing for form rendering
to now depend on templates, but for template engines (which currently
are configured explicitly and only know about the template directories
you tell them about) to also automatically and unconditionally know
about django.forms (even if you maybe aren't using forms at all -- maybe
you're using the template engine standalone for something else entirely)
doesn't seem right at all.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/573AB437.6020607%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-05-16 Thread Carl Meyer
The technical board has approved DEP 5, and since the patch [1] is also 
ready to go, I moved it straight to Final state. Thanks to everyone who 
contributed to the discussion, and especially to Florian and Tim for the 
patch.

Carl

  [1] https://github.com/django/django/pull/6501

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/22c30cb0-7a3e-4622-9302-294170ec92a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Template-based widget rendering

2016-05-16 Thread Carl Meyer
Hi Tim,

On 05/14/2016 08:03 AM, Tim Graham wrote:
> While testing this, I ran into a shortcoming with the fallback strategy
> for backwards-compatibility.
> 
> If you have a DjangoTemplates backend configured with 'APP_DIRS': True
> (as in the tutorial) and you try to visit /admin/auth/user/#/change/
> which renders the ReadOnlyPasswordHashWidget, the template system will
> crash with TemplateDoesNotExist because it can find the widget's
> template using the app_directories loader but not the
> 'django/forms/widgets/attrs.html' template that the first template
> includes. Since the first template is found using what's configured in
> TEMPLATES (which doesn't know about any of the built-in form templates),
> the standalone engine needed to find the built-in templates is ignored.
> 
> I guess it will affect every project that uses the admin. I can't think
> of a simple solution other than adding a system check upgrade warning to
> detect this situation ('django.contrib.admin' in INSTALLED_APPS but not
> 'django.forms') and advise the user to add 'django.forms' to
> INSTALLED_APPS. Thoughts?

Yuck. Is there only one admin widget that includes a built-in widget? If
so, I think we would maybe be better off just duplicating that one
built-in widget in the admin templates. Obviously that's not ideal, but
I think it's better than giving up on seamless backward compatibility.

Then obviously we need to add a note to the documentation clarifying
that if you have any custom form templates that need to include or
extend any of the default templates, you should put 'django.forms' in
INSTALLED_APPS so all of the templates can be found by the same template
engine.

This might be another argument in favor of deprecating the fallback and
actively pushing people to just get 'django.forms' into their
INSTALLED_APPS. But we can wait on that, like you said.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57394559.70508%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Template-based widget rendering

2016-05-15 Thread Carl Meyer
On 05/15/2016 11:01 PM, Curtis Maloney wrote:
> So this seems to leave us only part way to removing rendering decisions
> from the form code -- your widgets can be templated, but your form code
> still needs to decide which widgets...

Yep, one step at a time. With django-floppyforms templated widgets I've
used a template filter that allows switching a widget's template path in
the template that's rendering the form, placing more control into
templates. I think integrating something like that could make sense as a
next step.

(Since widgets are Python classes that control not only markup
generation but also data retrieval, control over widgets themselves
needs to remain in Python code. But the rendering template could be
given the option to customize the template used to render the widget.)

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5739564A.6010207%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-05-15 Thread Carl Meyer
On 05/15/2016 09:58 PM, Carl Meyer wrote:
> Yuck. Is there only one admin widget that includes a built-in widget? If
> so, I think we would maybe be better off just duplicating that one
> built-in widget in the admin templates.

That should read "built-in template," not "built-in widget."

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5739528E.3010909%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-05-12 Thread Carl Meyer
On 05/12/2016 09:39 AM, Tim Graham wrote:
> So this discussion doesn't stall the rest of the patch, I suggest
> keeping the fallbacks for now and deprecation them later if they cause
> confusion or other problems.

Yes, I think that makes sense.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5734BA8F.4000506%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-05-11 Thread Carl Meyer
On 05/11/2016 11:52 AM, Carl Meyer wrote:
> On 05/11/2016 11:30 AM, Tim Graham wrote:
>> What's your proposal for changing the default TEMPLATES? Using Jinja2 or
>> DTL?
> 
> At some point maybe we can adopt Jinja2 as a required dependency. Until
> then, the default startproject template can't use it, so I think DTL is
> the only real option.

Oops, I should clarify here that I don't actually plan to change the
default TEMPLATES setting at all; we don't need to. The default
startproject TEMPLATES already includes an engine with APP_DIRS: True; I
would just add 'django.forms' to the default INSTALLED_APPS.

But even that only really makes sense if we're planning to deprecate the
automatic fallback to the built-in templates, so we want to push people
to configure their settings to explicitly include them. If we plan to
leave the fallback around permanently, there's no need to change the
startproject template at all.

Here are the pros and cons as I see them for deprecating the fallback
(all the pros apply only after the deprecation process would complete):

- pro: cleaner and simpler default TemplateRenderer, with less complex
code and tests to maintain.

- pro: simpler mental model of form template rendering, where the form
templates work just like any other kind of template, they aren't
magically always available.

- con: requires an update to settings (for most projects, just adding
'django.forms' to INSTALLED_APPS) to keep forms rendering as they do now
(once the deprecation process completes).

I like conceptual simplicity, and I care more about where we end up than
about what the deprecation path requires (IMO the "con" in that list
disappears once everyone has upgraded through the deprecation process,
whereas the "pros" are permanent), so I lean towards deprecating the
fallback, but I really don't feel strongly about it at all. Claude
prefers keeping the fallback around permanently. What do others think?

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57337508.7060708%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-05-11 Thread Carl Meyer
On 05/11/2016 11:30 AM, Tim Graham wrote:
> I'm not sure about how common the need for custom widget templates are.
> Speaking for djangoproject.com and a few other small projects I
> maintain, I don't think these projects would make use of them but maybe
> if the feature is there, I might realize it would help in some places.

It's certainly not always needed; maybe "basic" is too strong a word.
Better to say that it's a very useful intermediate technique, especially
for more complex widgets with detailed UI needs.

We're not only talking about overriding templates for built-in widgets,
we're also talking about the ability to define your own custom widgets
with their own templates. If only the built-in widget templates are
available, you can't do that either. So given that templating is now the
norm for how to build widgets, we'd effectively be making it impossible
to create your own widget classes without changing the default form
renderer.

I'd flip it around and make the comparison with "standalone use of the
forms library without a configured TEMPLATES setting," which seems to be
the competing feature: is that really _more_ common than custom form
widgets? How many projects do you maintain that need that?

> What's your proposal for changing the default TEMPLATES? Using Jinja2 or
> DTL?

At some point maybe we can adopt Jinja2 as a required dependency. Until
then, the default startproject template can't use it, so I think DTL is
the only real option.

> Since others have supported the idea, feel free to push your ideas to
> the branch -- I won't be working on that branch for the rest of today.

Ok, will do.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57337178.5040003%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-05-11 Thread Carl Meyer
On 05/10/2016 03:37 PM, Carl Meyer wrote:
> I've
> updated the DEP with a couple minor changes to reflect the latest
> learnings from the implementation; you can see the latest changes at
> https://github.com/django/deps/compare/763530e1a9...master

Better version of this link (to exclude more recent changes to other
DEPs in master):
https://github.com/django/deps/compare/763530e1a9...77c93d46baf

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5733697F.2060309%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: GitHub Issues for DEP repository?

2016-05-11 Thread Carl Meyer
Hi Kevin,

On 05/10/2016 11:24 PM, Kevin Christopher Henry wrote:
> With all the talk of DEPs flying around I thought I'd finally take a
> look at one in detail.
> 
> I wanted to make a suggestion about it and realized that there wasn't
> really a good place to do so. The issue was too minor for a mailing list
> discussion, and there was no open pull request to comment on. My first
> instinct was to file a GitHub issue (and DEP 1 says "you can submit
> corrections as GitHub issues") but it seems that Issues has been
> disabled for this repository. Was that just an oversight?

Given that DEP 1 specifically mentions using GH issues to comment on
DEPs, I'm considering this a bug/oversight: I went ahead and enabled GH
issues on the DEPs repo. Thanks for pointing it out!

> More broadly, I think that DEP 1 makes this subject too complicated:
> "How you report a bug, or submit a DEP update depends on several
> factors, such as the maturity of the DEP, the preferences of the DEP
> author, and the nature of your comments. For the early draft stages of
> the DEP, it's probably best to send your comments and changes directly
> to the DEP author. For more mature, or finished DEPs you can submit
> corrections as GitHub issues or pull requests against the DEP
> repository. When in doubt about where to send your changes, please check
> first with the DEP author and/or a core developer."
> 
> It adds some friction to the process to ask someone to contact the DEP
> author directly to ask them whether or not they should contact them
> directly with their comments. I think it would be more straightforward
> to just use the usual complement of open-source tools that we all know
> and love: the developers mailing list, pull requests, and an issue tracker.

I agree with this: I think it's better to encourage keeping things in
public rather than encourage privately contacting DEP authors (as a DEP
author, I'd certainly rather have suggestions made in public than be
contacted privately, even in early draft stages). So I'd personally be
fine with a PR to amend this section to remove mention of private
contact. Jacob, I think you wrote this (or adapted it from PEP 1) -- any
thoughts?

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/573367C8.5090203%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Process DEP for "official" non-core projects

2016-05-11 Thread Carl Meyer
On 05/10/2016 08:58 PM, Andrew Godwin wrote:
> Following my decision to move Channels away from a 1.10 merge and
> towards an existence as a separate app for a bit while it matures, I
> would like to write up a process DEP for how we achieve this -
> specifically, what it takes to be adopted as a non-core app, what that
> means for the Django organisation, and what its goals should be.
> 
> Given recent events, I feel having a process that's agreed upon and
> specified is a good move for this, especially if we do it with something
> else again in future.

Makes sense to me.

> My rough outline is as follows:
> 
>  - A non-core project can be moved under the governance of the Django
> project if it is deemed an important feature that's in our interests to
> maintain and grow
> 
>  - We should be relatively convinced that it represents the best
> implementation of the feature that there is, that it is matured and has
> a stable release, and that there aren't competing implementations at a
> similar level
> 
>  - The decision-making structure is similar to that of a DEP, in that
> there must be a core "shepherd" who is the go-to person for it and makes
> sure it crosses the basic test, and a technical board vote on inclusion.
> 
>  - A project under Django governance will fall under the Django umbrella
> for security-related reports and fixes, and will maintain the Django
> security policy on the current major release only.

I'm not quite sure what this means. By "major release" here, you mean
"major release of the adopted project," not "major release of Django"?
So this means that security fixes for the adopted project will be
provided as patch releases on its current major release, but won't be
backported beyond that?

>  - Release schedule will not be fixed to Django's and no LTS pattern
> will be forced on it, and a limited backwards-compatibility policy of
> only "at least one major version" will be maintained.
> 
> I think it's worth working out how the whole thing is going to run if
> we're going to do it properly, and the DEP system seems like the best
> place for something like this. If we do go down the path of starting to
> split out Django into more packages, as well, it could provide a useful
> base for that to structure around (though we'd want to beef up
> security/backwards-compat timelines).
> 
> It's also probably sensible that whatever we come up with covers
> localflavor, as well, since that's sort of in this state already.

All of those points seem sensible to me, but I'd be interested in
hearing from people who've already been actively involved in localflavor
maintenance.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/573366A9.5020706%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-05-10 Thread Carl Meyer
On 05/07/2016 08:27 AM, Tim Graham wrote:
> Thanks, I have what I hope is a minimally mergable patch here:
> https://github.com/django/django/pull/6501

Thanks Tim (and Florian for all the previous work on this patch). I've
updated the DEP with a couple minor changes to reflect the latest
learnings from the implementation; you can see the latest changes at
https://github.com/django/deps/compare/763530e1a9...master

I reviewed this thread and I think the response was pretty much
positive; I didn't see any outstanding unaddressed concerns. If you have
any, please let me know! I plan to ask the technical board to approve
DEP 5 shortly.

> As noted on the PR, there a more things that should be before the 1.10
> feature freeze but I'm hoping I can ticket them out and get some help
> from the community after merging this first step so that I can continue
> spending most of my time reviewing patches.

I can't help with a new graphic for the docs, but I will take care of
the decorator_from_middleware update tomorrow.

I'm not entirely sure that a graphic is still needed with the new
system; its behavior is simpler than the old. But if a motivated and
appropriately skilled party provides a useful graphic, that'd be great
of course!

Carl

> On Friday, May 6, 2016 at 7:59:38 PM UTC-4, Carl Meyer wrote:
> 
> I agree with Simon on both counts. We do usually continue to test
> deprecated code paths until they are removed, but I also think the
> duplication in cases of tests overriding MIDDLEWARE_CLASSES might
> not be
> necessary in _all_ cases; I think some discretion could be used
> depending on to what extent the middleware is incidental to the
> tests vs
> the direct subject of the test. But it might be simpler to just do them
> all than to make that determination.
> 
> Carl
> 
> On 05/04/2016 08:57 PM, charettes wrote:
> > Hi Tim,
> >
> > I think we should favor displaying a message in accordance with the
> > setting the user is using as it will make the transition less
> confusing.
> > In the case of the documented check message I think using the form
> > "MIDDLEWARE/MIDDLEWARE_CLASSES" would make it easier to read then
> > mentioning the two possible variants. We already alter the document
> > messages anyway to account for their dynamic nature.
> >
> > In the case of the tests I believe both code path should continue
> to be
> > tested. From the top of my head I can't think of an alternative to
> > subclasses using @override_settings. I suggest we make the *legacy*
> > tests class extend the MIDDLEWARE using test class and not the
> other way
> > around as it will make the MIDDLEWARE_CLASSES code removal clearer.
> >
> > Simon
> >
> > Le mercredi 4 mai 2016 19:59:05 UTC-4, Tim Graham a écrit :
> >
> > I've been working on this and wanted to raise a couple points for
> > discussion.
> >
> > How should we treat error messages in place like system checks
> where
> > we have phrases like "Edit your MIDDLEWARE_CLASSES" ... of course
> > the check can easily check both MIDDLEWARE and MIDDLEWARE_CLASSES
> > without much effort but should we make the error message
> "smart" and
> > display the version of the setting that the user is using?
> > Alternatively, we could always reference MIDDLEWARE (the
> > non-deprecated version) or use some variation like
> > "MIDDLEWARE(_CLASSES)" or "MIDDLEWARE/MIDDLEWARE_CLASSES"
> until the
> > deprecation period ends.
> >
> > Another point for discussion is whether we need to duplicate a
> lot
> > of tests so we test that the middleware continue to work with
> both
> > the old-style MIDDLEWARE_CLASSES and the new style MIDDLEWARE
> > response handling. I guess a subclass of anything that uses
> > @override_settings(MIDDLEWARE=...) that uses
> > @override_settings(MIDDLEWARE_CLASSES=...) might work. Just
> putting
> > it out there in case anyone has a better idea.
> >
> > On Monday, January 18, 2016 at 9:20:03 PM UTC-5, Carl Meyer
> wrote:
> >
> > I've updated DEP 5 with a new round of clarifications and
> tweaks
> > based on the most recent feedback:
> > https://github.com/django/deps/compare/62b0...master
> <https://github.com/django/deps/compa

Re: Template-based widget rendering

2016-05-10 Thread Carl Meyer
Hi Tim,

On 05/10/2016 07:10 AM, Tim Graham wrote:
> About the fallback engines, the main use case I have in mind (as Claude
> alluded to) is if you want to use django.forms "standalone" without the
> rest of Django. In that case, it seems like it would be nice not to
> require someone to configure settings.TEMPLATES. Here's an alternate
> proposal:
> 
> Creating a "django.forms.renderers.templates.DefaultTemplateRenderer"
> (exact name to be discussed) which uses the fallback engines and ignores
> settings.TEMPLATES. This could be the default renderer for the
> FORM_RENDERER setting, for backwards-compatibility and to allow
> django.forms standalone usage by default. For more advanced uses, set
> the setting: FORM_RENDERER =
> 'django.forms.renderers.templates.TemplateRenderer' (which uses
> django.template.loader.get_template and doesn't have any fallback engines).

Yeah, I considered this (my first version of my commit actually had two
different renderer classes like this). My concern is that I think this
proposal has the default backwards for what will actually be typical
usage. In my experience of using templated widgets for the last several
years (via django-floppyforms), the biggest value is the ability to
override specific widget templates with your own templates. So I think
overriding templates (within a normal Django project with TEMPLATES
configured) is the "basic usage" and standalone use of the forms library
is an "advanced use," not the other way around.

The proposed "DefaultTemplateRenderer" doesn't allow any template
overriding at all, because it can _only_ load the built-in templates. I
think in the long run it would be a mistake to have the default
FORM_RENDERER setting be a renderer that doesn't allow easily overriding
templates, and I don't think that we should allow the transition
concerns to override reaching the right long-term solution after a
transition path.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/573226C2.9050807%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Withdrawing Channels from 1.10

2016-05-10 Thread Carl Meyer
On 05/10/2016 10:39 AM, Andrew Godwin wrote:
>  - Being almost purely an addition to Django, even though it technically
> inserts a new layer, makes it more well-suited to live externally than
> many other features. While the external package will have to
> monkey-patch a few things, it'll be relatively minor.

Are there small changes that could be made in Django 1.10 that would
reduce the monkeypatching burden? In the channels repo it seems the only
remaining monkeypatch in hacks.py is for staticfiles' runserver; are
there other monkeypatches I'm missing?

The staticfiles case is a known problem with management command
overriding, and I don't have a good solution. I guess it's a choice
between monkeypatching vs having to document a required ordering of
staticfiles vs channels in INSTALLED_APPS.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57321974.7060207%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: DEPs (was: Making Django more PaaS-friendly)

2016-05-10 Thread Carl Meyer
Hi Anssi,

Thanks for the clarification, that all makes sense. A few comments below:

On 05/10/2016 12:00 AM, Anssi Kääriäinen wrote:
> It's not so much the DEPs that have been written (though the channels
> DEP is going to be post-design instead of supporting the design). It's
> more about those features that don't have a DEP at all. For example
> database schemas and subquery expressions come to mind (these are
> ongoing work, PRs available at GitHub). There are various design
> choices and the features are somewhat significant, yet the design
> hasn't been at all DEP driven.

I would say that a DEP is generally a good idea if the people working on
the feature feel that it would be helpful to them to organize and record
the various design decisions, or if people in the community are
expressing concern about the feature or there's significant disagreement
about the design decisions that isn't quickly and easily resolved. When
there's a proposal to change a core aspect of Django in a pretty
fundamental way (like with multiple-template-engines or middleware),
that's a good case to just start out right away with a DEP, because it's
important to make really sure the community is on board with the change.
Again, I think DEPs ideally should be a useful tool, not a hoop to jump
through.

> Also, having DEP requested for channels only at this point, and Andrew
> (a technical board member) assuming a DEP wasn't required should point
> out there might be a problem. This is not to criticize Andrew but the
> way DEPs are sometimes required, sometimes not.

I didn't intend to be "requiring" a DEP for channels as some kind of
pro-forma dotting of the I's and crossing of the T's for a decision that
in practice had already been made; if that's how my message in that
thread came across, then I miscommunicated. I (strongly) suggested a
DEP, because when I looked back at e.g the discussion thread from
December I saw a number of concerns expressed about channels that still
didn't seem to me to have been resolved (and it also seemed clear from
messages from you and Mark that they weren't) and it seemed to me that
channels still needed the sort of focused public resolution of questions
and concerns that a DEP discussion can provide.

But it's quite possible I was just wrong in that suggestion. If in fact
all the important decisions are already made, and the only thing
channels really needs to address the concerns is real-world proving and
testing, then perhaps a DEP at this point is a waste of time. (Though a
DEP could also be a useful place to record and summarize the discussion
around the results of such real-world testing.)

> I'm not seeing a problem with the DEP idea in general, with any
> particular DEP or with any single feature having or not having a DEP.
> I'm criticizing the approach where sometimes patches are called out
> randomly for DEP, often at a point where the DEP is just documenting
> what has been already decided, not supporting the decision process
> itself.

Yeah, I certainly agree that the way it worked out for channels wasn't
ideal.

> I probably picked the wrong thread to complain about this - if we are
> going to want DEPs for features similar to the ideas discussed in this
> thread, then the timing for requesting a DEP was accurate.

Agreed.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/573217E2.4010702%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Making Django more PaaS-friendly

2016-05-09 Thread Carl Meyer
Hi Anssi,

On 05/09/2016 06:53 AM, Anssi Kääriäinen wrote:
> I'm very much afraid we are in a situation where we have a documented
> DEP process which isn't actually enforced, nor does it document what
> we actually do.
>
> Python's PEP process is useful as the document evolves while the
> design evolves. What we do with DEPs is different, we first design
> something, then we document the design decision, often mostly because
> DEPs are a required document. We don't get what we could from DEPs as
> they are written too late.

I'm curious to hear more about this - could you give some example DEPs
where this was a problem? In the case of the two DEPs I was most closely
involved with (multiple template engines and rethinking middleware), I
don't think this is an accurate description of what happened. Both those
DEPs were written relatively early in the design process and evolved as
we learned from discussion and implementation.
They were done quite similarly to how the PEPs I've worked on were done.

I'm also curious about your use of the word "enforced." I don't really
see the DEP process as something that needs "enforcement" (and I'm
curious where you see it as having been lacking in "enforcement"). IMO
the only relevant "enforcement," really, is that the implementation of a
DEP which is rejected by the technical board (has never happened yet)
should not be merged.

DEPs, like PEPs, aren't supposed to be a legalistic policy: there is no
hard and fast rule about which features should have DEPs, and I don't
think we need a hard and fast rule. DEPs should happen when someone
observes that the discussion around a feature is getting quite complex
or contentious, and suggests that a DEP would be helpful. DEPs should be
a tool, not a barrier: a tool for helping to focus discussions on
contentious issues, and record the perspectives expressed in those
discussions, both for posterity and to help keep the discussion from
going around in circles.

> Could we try an approach where we *first* require a DEP, then have the
> design discussion?

In the PEP process, it's very typical to start into design discussions
before realizing that the issue is contentious/complex enough to require
a PEP, then someone collects the knowledge generated from that early
discussion into a first draft PEP, which then receives further
discussion. (In the Python world most of this pre-PEP discussion happens
on the python-ideas mailing list, so if you aren't subscribed there you
might not see it.) I think this thread is a great example of that
working precisely the way it should work: first post some general
thoughts, collect ideas and responses to those thoughts, then work that
into a DEP with a specific proposal, followed by more detailed
discussion of that specific proposal.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/573113D6.90503%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Template-based widget rendering

2016-05-09 Thread Carl Meyer
Hi Tim,

On 05/09/2016 08:03 AM, Tim Graham wrote:
> I've been working on testing and writing documentation for Preston's PR:
> https://github.com/django/django/pull/6498
> 
> About backwards compatibility, the current implementation which seems to
> be based on Carl's proposal requires either a) 'django.forms' in
> INSTALLED_APPS (so the APP_DIRS loader can find the DTL widget
> templates) or b) Jinja2 to be installed so that the
> TemplateRenderer.default_engine() initialization of Jinja2 doesn't fail.
> If these items were accepted and acknowledged as upgrade requirements,
> it wasn't clear to me.

Nope, I just overlooked the "Jinja2 is not installed" case there. In
that case we should just use a DjangoTemplates engine instead of a
Jinja2 engine as the fallback, which is what you've done in
https://github.com/django/django/pull/6498/commits/83e0138a85ada2a8139af8a35629b78ec5e539d8

> In trying to make the Django test suite pass without Jinja2 installed, I
> noticed that the 'django.forms' in INSTALLED_APPS requirement is a bit
> annoying because this requires any use of
> @override_settings(INSTALLED_APPS=[...]) that does widget rendering to
> now include 'django.forms'.

Yeah... so relatedly, you also mentioned in IRC and on GH that you don't
see why we should deprecate the automatic fallback in case you don't
have django.forms in INSTALLED_APPS and a loader with APP_DIRS=True. The
answer to that, IMO, is that checking specifically for "django.forms in
INSTALLED_APPS and APP_DIRS=True" is janky and inadequate, and I really
don't think it belongs in there permanently. In fact I'm wondering now
whether it even belongs in there temporarily as a deprecation path.

The problem with that check is that there are various legitimate ways
one could use the TemplateRenderer, and putting django.forms in
INSTALLED_APPS and using a loader with APP_DIRS=True is only one of
them. If you don't want to use APP_DIRS=True, it's also reasonable to
put a separate engine in TEMPLATES pointing to the form templates (this
might be a nicer solution for the Django test suite, for instance). Some
people might even want to supply all their own form templates and not
use any of the default ones, and that should be supported, too. By
automatically overriding with our own fallback template engine if we
don't detect "django.forms in INSTALLED_APPS and APP_DIRS=True", we
effectively prohibit any other approach. I thought that might be OK as a
temporary state of affairs during a deprecation path, but I definitely
don't think it's OK permanently.

I pushed an alternative approach in
https://github.com/carljm/django/commit/7d734cfb9da2f64e4bf59c55167c70748b3bd092
that removes the INSTALLED_APPS checking. Instead, it has the `render`
method unconditionally fallback to the built-in templates if a template
is not found via the engines configured in TEMPLATES.

I still think that in the long run, it's simpler and more predictable if
the developer is simply responsible to either set up TEMPLATES such that
the form templates needed are findable (startproject would set this up
by default), or use a custom subclass of TemplateRenderer that does
something entirely different. So I would still favor deprecating the
fallback (in which case we should also update the startproject template
so the fallback isn't needed for new projects). But I think this
fallback is more flexible and simple enough that we could consider
keeping it in place permanently, if you and others prefer that.

(I also removed the whole engine_name thing in that commit; it doesn't
really seem worth it. If you're subclassing TemplateRenderer, it's easy
enough to just override get_template and do whatever you like.)

If you like that approach, I can re-add some TemplateRenderer tests and
update the docs and push it to the main `15667` branch.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57310F15.2060401%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-05-06 Thread Carl Meyer
I agree with Simon on both counts. We do usually continue to test
deprecated code paths until they are removed, but I also think the
duplication in cases of tests overriding MIDDLEWARE_CLASSES might not be
necessary in _all_ cases; I think some discretion could be used
depending on to what extent the middleware is incidental to the tests vs
the direct subject of the test. But it might be simpler to just do them
all than to make that determination.

Carl

On 05/04/2016 08:57 PM, charettes wrote:
> Hi Tim,
> 
> I think we should favor displaying a message in accordance with the
> setting the user is using as it will make the transition less confusing.
> In the case of the documented check message I think using the form
> "MIDDLEWARE/MIDDLEWARE_CLASSES" would make it easier to read then
> mentioning the two possible variants. We already alter the document
> messages anyway to account for their dynamic nature.
> 
> In the case of the tests I believe both code path should continue to be
> tested. From the top of my head I can't think of an alternative to
> subclasses using @override_settings. I suggest we make the *legacy*
> tests class extend the MIDDLEWARE using test class and not the other way
> around as it will make the MIDDLEWARE_CLASSES code removal clearer.
> 
> Simon
> 
> Le mercredi 4 mai 2016 19:59:05 UTC-4, Tim Graham a écrit :
> 
> I've been working on this and wanted to raise a couple points for
> discussion.
> 
> How should we treat error messages in place like system checks where
> we have phrases like "Edit your MIDDLEWARE_CLASSES" ... of course
> the check can easily check both MIDDLEWARE and MIDDLEWARE_CLASSES
> without much effort but should we make the error message "smart" and
> display the version of the setting that the user is using?
> Alternatively, we could always reference MIDDLEWARE (the
> non-deprecated version) or use some variation like
> "MIDDLEWARE(_CLASSES)" or "MIDDLEWARE/MIDDLEWARE_CLASSES" until the
> deprecation period ends.
> 
> Another point for discussion is whether we need to duplicate a lot
> of tests so we test that the middleware continue to work with both
> the old-style MIDDLEWARE_CLASSES and the new style MIDDLEWARE
> response handling. I guess a subclass of anything that uses
> @override_settings(MIDDLEWARE=...) that uses
> @override_settings(MIDDLEWARE_CLASSES=...) might work. Just putting
> it out there in case anyone has a better idea.
> 
> On Monday, January 18, 2016 at 9:20:03 PM UTC-5, Carl Meyer wrote:
> 
> I've updated DEP 5 with a new round of clarifications and tweaks
> based on the most recent feedback:
> https://github.com/django/deps/compare/62b0...master
> <https://github.com/django/deps/compare/62b0...master>
> 
> Carl
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-developers+unsubscr...@googlegroups.com
> <mailto:django-developers+unsubscr...@googlegroups.com>.
> To post to this group, send email to django-developers@googlegroups.com
> <mailto:django-developers@googlegroups.com>.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/eb1cf3f4-c021-40f6-be65-35427b2bf5c5%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/eb1cf3f4-c021-40f6-be65-35427b2bf5c5%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/572D2FD8.8060804%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Thoughts on ASGI or Why I don't see myself ever wanting to use ASGI

2016-05-06 Thread Carl Meyer

On 05/06/2016 02:31 PM, Andrew Godwin wrote:
> 
> On Fri, May 6, 2016 at 1:19 PM, Carl Meyer  <mailto:c...@oddbird.net>> wrote:
> 
> On 05/06/2016 01:56 PM, Donald Stufft wrote:
> > User level code would not be handling WebSockets asynchronously, that
> > would be left up to the web server (which would call the user level code
> > using deferToThread each time a websocket frame comes in). Basically
> > similar to what’s happening now, except instead of using the network and
> > a queue to allow calling sync user code from an async process, you just
> > use the primitives provided by the async framework.
> 
> I think (although I haven't looked at it carefully yet) you're basically
> describing the approach taken by hendrix [1]. I'd be curious, Andrew, if
> you considered a thread-based approach as an option and rejected it? It
> does seem like, purely on the accessibility front, it is perhaps even
> simpler than Channels (just in terms of how many services you need to
> deploy).
> 
> Well, the thread-based approach is in channels; it's exactly how
> manage.py runserver works (it starts daphne and 4 workers in their own
> threads, and ties them together with the in-memory backend).
> 
> So, yes, I considered it, and implemented it! I just didn't think it was
> enough to have just that solution, which means some of the things a
> local-memory-only backend could have done (like more detailed operations
> on channels) didn't go in the API.

Ha! Clearly I need to go have a play with channels. It does seem to me
that this is a strong mark in favor of channels on the accessibility
front that deserves more attention than it's gotten here: that the
in-memory backend with threads could be a reasonable way to set up even
a production deployment of many small sites that want websockets and
delayed tasks without requiring separate management of interface
servers, Redis, and workers (or separate WSGI and async servers). Of
course it has the downside that thread-safety becomes an issue, but
people have been deploying Django under mod_wsgi with threaded workers
for years, so that's not exactly new.

Of course, there's still internally a message bus between the server and
the workers, so this isn't exactly the approach Donald was preferring;
it still comes with some of the tradeoffs of using a message queue at
all, rather than having the async server just making its own decisions
about allocating requests to threads.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/572D0890.5010703%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Thoughts on ASGI or Why I don't see myself ever wanting to use ASGI

2016-05-06 Thread Carl Meyer
On 05/06/2016 01:56 PM, Donald Stufft wrote:
> User level code would not be handling WebSockets asynchronously, that
> would be left up to the web server (which would call the user level code
> using deferToThread each time a websocket frame comes in). Basically
> similar to what’s happening now, except instead of using the network and
> a queue to allow calling sync user code from an async process, you just
> use the primitives provided by the async framework.

I think (although I haven't looked at it carefully yet) you're basically
describing the approach taken by hendrix [1]. I'd be curious, Andrew, if
you considered a thread-based approach as an option and rejected it? It
does seem like, purely on the accessibility front, it is perhaps even
simpler than Channels (just in terms of how many services you need to
deploy).

Carl

  [1] http://hendrix.readthedocs.io/en/latest

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/572CFC63.9040308%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Thoughts on ASGI or Why I don't see myself ever wanting to use ASGI

2016-05-06 Thread Carl Meyer
On 05/06/2016 11:09 AM, Aymeric Augustin wrote:
> I think it's important to keep a straightforward WSGI backend in case we crack
> this problem and build an async story that depends on asyncio after dropping
> support for Python 2.
> 
> I don't think merging channels as it currently stands hinders this possibility
> in any way, on the contrary. The more Django is used for serving HTTP/2 and
> websockets, the more we can learn.

This summarizes my feelings about merging channels. It feels a bit
experimental to me, and I'm not yet convinced that I'd choose to use it
myself (but I'd be willing to try it out). As long as it's marked as
provisional for now and we maintain straight WSGI as an option, so
nobody's forced into it, we can maybe afford to experiment and learn
from it.

ISTM that the strongest argument in favor is that I think it _is_
significantly easier for a casual user to build and deploy their first
websockets app using Channels than using any other currently-available
approach with Django. Both channels and Django+whatever-async-server
require managing multiple servers, but channels makes a lot of decisions
for you and makes it really easy to keep all your code together. And (as
long as we still support plain WSGI) it doesn't remove the flexibility
for more advanced users who prefer different tradeoffs to still choose
other approaches. There's a lot to be said for that combination of
"accessible for the new user, still flexible for the advanced user", IMO.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/572CD4B8.6010800%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: My Take on Django Channels

2016-05-06 Thread Carl Meyer
Hi Andrew,

Replying off-list just to say that I totally understand your frustration
here, and I wish I weren't contributing to it :( I hope I'm managing to
speak my mind without being an asshole about it, and I hope you'd tell
me if I failed.

Really glad Jacob stepped up on the DEP; I was thinking when I wrote my
last email that that'd be the ideal solution, but I didn't want to put
anyone else on the spot (I almost volunteered myself, but I don't think
I'm quite sold enough yet to be a good advocate). I hope the
conversation on the technical topics goes well (I think you've been
doing a great job with it so far on these latest threads) and a
satisfactory resolution is reached in time for a merge into 1.10.

Carl

> I think you're entirely right, Carl - I'm just getting frustrated with
> myself at this point for not realising sooner and trying to find ways to
> not do it - people only pay real attention to a patch as you're close to
> merging and emotionally invested in it, and it's a little exasperating. 
> 
> Jacob has graciously stepped in to help write one, and I am going to
> have a much-needed evening off from doing Channels stuff, I haven't had
> a break in a while.


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/572CD1C3.2090305%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: My Take on Django Channels

2016-05-05 Thread Carl Meyer
On 05/05/2016 04:37 PM, Andrew Godwin wrote:
> To be honest, I had entirely forgotten the DEP process existed until
> this thread started up; I'm not sure what to blame this on, but as a
> member of the tech board I haven't got an email about approving a DEP
> since last October, so it's been a while.

There has been more recent activity on several in-progress DEPs on this
mailing list, but it has been a while since one was accepted.

> Part of me does not want to aggravate my RSI by having to write and rush
> through a DEP in the next 10 days, but I can't deny that you are likely
> correct that it sends the right signal given that we have the process in
> place.
> 
> That said, a couple of decently-sized features (full text search,
> password validators) have landed recently without one, so I can't
> entirely feel justified dropping this from 1.10 given that it is fully
> written, has extensive documentation, a mostly-complete test suite and
> several fully-worked examples - far more context than a DEP would ever
> provide. It would feel like a bit of a kick in the teeth, to be honest.

I've no desire either to aggravate your RSI or kick you in the teeth! I
understand the multiple competing pressures here and won't stand in the
way of a merge into 1.10 sans DEP if that still seems like the best path
forward to you. It's not like a merge into alpha is the end of the line
in terms of possible design changes or updates (or even possibly
reverts). A DEP could even happen post-merge; that would be unusual but
perhaps still better than none at all.

I have a couple more comments, more in the line of general thoughts
about the whys and hows of DEPs.

I do think that DEPs have a significant value that goes beyond just
providing information that could be found elsewhere (e.g. in the
channels documentation). They collect that information (or references to
it) in one place, in a standard digestible format, and formally present
it to the community as a requested change, with rationale and rejected
alternatives (including a fair representation of the objections that
have been raised and your answers to them), and present a formal
opportunity for anyone with concerns to raise them (and give you a
reasonable place to later say "this is precisely when you should have
raised your concerns if you had them") and then also store that in a
stable place for future reference when someone comes by in two years and
can't understand why we did things the way we did.

(I'm not saying this to put further pressure on, just to defend the DEP
process against the implicit charge that it's possibly-useless make-work
when other documentation has already been written.)

There's been no clear delineation of what size features should have a
DEP. I think channels, multiple-template-engines, and
reworked-middleware (and migrations, for that matter) are all
rethinkings of long-standing core aspects of how Django works, which in
my mind makes them prime DEP candidates, whereas FTS and password
validation both seem to me like small-to-medium peripheral features that
I wouldn't necessarily have expected to have one.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/572BD5AB.1050007%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: My Take on Django Channels

2016-05-05 Thread Carl Meyer
Hi Andrew,

On 05/05/2016 02:19 PM, Andrew Godwin wrote:
> I will put my hand up and say that this sidestepped the DEP process, and
> that's entirely my fault. It was not my intention; I've been working on
> this for over two years, and only last year did I go public with my
> semi-final design and start asking for feedback; I should probably have
> taken it into a DEP then, but failed to.

This isn't a past-tense question; it's not too late to write a DEP, and
I personally think that a DEP should be written and approved by the
technical board before the channels patch is merged. I actually assumed
that one was still on its way; perhaps I missed some communication at
some point that said it wouldn't be.

I'm sensitive to the fact that you've already put lots of work into this
and time is short if you want to get it into 1.10. On the other hand,
this is precisely why the DEP process exists: to ensure that significant
changes to Django are carefully considered, in public, in a way that
allows those without time to dig into all the details to absorb and
consider the salient high-level points. I think that is precisely what
the channels work needs (in community/process terms), and I think we'd
be very poorly advised to push forward on merging it without an approved
DEP.

I don't think a channels DEP would need to delve into the details of
precisely which channel backends are currently available, etc; it would
mostly be about justifying the high-level design (and comparing it to
rejected alternatives for solving the same problems). It would focus on
the changes to Django itself, rather than implementation choices of the
initially-available external components. It could probably copy
liberally from (or just be short and heavily reference) the ASGI spec
and/or Channels docs; that's not a problem.

I'm excited about the potential of channels and ASGI, but I'm also
suspicious of arguments that it is urgent to merge into 1.10 at all
costs. I'm not necessarily opposed to that, if it's ready on time and
the community discussion around a DEP seems to have reached a
satisfactory conclusion. (I also think that the important thing is to
make sure the changes to Django itself aren't painting us into any
corners: as long as ASGI is optional in Django, the external support
components don't need to be fully mature yet; especially if we can do an
import dance to make them optional dependencies, which I think is
preferable.)

But I also think it would be far better to wait than to rush it in in
the face of reasonable unresolved concerns from the community, and
without an approved DEP. The argument has been that merging it "sends
the right signal to the community," but I have some concern that rushing
the merge could send negative signals about process consistency and
fairness that could easily outweigh any positive signals about Django
"having an async solution."


Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/572BBD7D.10302%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Should Template.render() return a safe string?

2016-04-28 Thread Carl Meyer
On 04/28/2016 11:46 AM, Aymeric Augustin wrote:
>> On 28 Apr 2016, at 19:30, Aymeric Augustin 
>>  wrote:
>>
>> It seems reasonable to assume that the result of rendering with
>> autoescaping enabled is HTML-safe — that’s the reason why
>> autoescaping exists.
> 
> Scratch that and let me try again:
> 
> It seems reasonable to assume that the result of rendering with
> autoescaping enabled is HTML-safe. The template is expected
> to be safe and values from the context were escaped before
> being interpolated into the template.
> 
> (This makes my argument a bit weaker but it still stands.)

Yes, that's convincing. I was thinking "we can't be sure of what a
template author might have done, so we should assume the output is
unsafe", but that's too cautious. With autoescape on, the assumption is
clearly that the output of the template render is supposed to be safe.
If someone uses |safe to bypass autoescape, they are asserting that
whatever they passed through it is also safe. So I think it's right to
mark the output safe, at least if autoescape is on.

Not sure about when autoescape is off. There's some advantage to making
the behavior simpler and always consistent regardless of the value of
autoescape. But I don't have a clear enough sense of the use-cases for
autoescape-off to know what the behavior should be in that case. If the
primary use case is "rendering a template that isn't destined for the
browser anyway, so doesn't need to be safe", then I think it would be
wrong to automatically mark that output as safe.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57225875.9080506%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Should Template.render() return a safe string?

2016-04-28 Thread Carl Meyer
Hi Aymeric,

On 04/28/2016 11:02 AM, Aymeric Augustin wrote:
> I just stumbled upon an interesting difference between Jinja2 and the
> Django template language.
> 
> |Template.render()| returns a safe string in Django. It's likely an
> accident of the implementation more than a design choice. It happens
> because |NodeList.render()| calls |mark_safe()| on its output, even at
> the top level.
> 
> I believe that |Template.render()| doesn't return a safe string in
> Jinja2. To achieve this effect, one would have to wrap the return value
> here
> 
> in |Markup()|.
> 
> It's unclear to me that one behavior has significant advantages over the
> other. It would be nice to resolve this inconsistency because it can
> cause non-trivial bugs such as torchbox/wagtail#2541
> .

ISTM that Jinja2's behavior is preferable. Given that both templating
languages allow turning off autoescaping (or bypassing it in other
ways), there's no a priori reason to believe that a string resulting
from rendering a template is necessarily a safe string.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57224377.2000403%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Making Django more PaaS-friendly

2016-04-27 Thread Carl Meyer
Hi Marcin,

On 04/26/2016 08:12 PM, Marcin Nowak wrote:
> But I would like to say my thoughts about "settings" itself.
> They were good (simple) before incuding nested dictionaries.
> After switching to dicts the settings handling went harder way.

I agree that settings grouped into dicts are harder to manage than flat
settings. Django has not made any "switch to dicts" for settings; in
fact, the last time we had to make a decision between a group of flat
settings with a common prefix vs a dict, we chose flat (the SECURE_*
settings).

> As a example I can tell about reconfiguring LOGGING for dev/live settings.
> We're tuning them via direct changes like
> LOGGING['loggers']['some-logger']['level']='...'
> which is very error prone. The flat settings are easier to maintain.

LOGGING is a special case; this dict format is not defined by Django,
but by Python's logging module and its "dictConfig" function.

> Nested settings are good for framework internals and should stay in that
> form, 
> but maybe solution for end users and ENV integration is flattening
> settings again, at the high-level?
> 
> Let's imagine something like INI file which is parsed somehow and
> converted to internal settings object.
> The INI file has a shallow structure (options grouped in sections), 
> can be easily extended / overrided / included (look at Buidout as an
> example),
> and ENV keys will be related to flat INI instead of nested settings.py.
> 
> Let's consider an example:
> 
> |
> [databases]
> default.url =postgres+psycopg2://user:pwd@localhost:5432/exampledb
> default.engine =django.db.backends.postgres_psycopg2
> 
> |
> 
> which can be simply reconfigured by ENV vars like:
> 
> DJANGO_DATABASES_DEFAULT_URL="postgres+psycopg2://user:pwd@localhost:5432/exampledb"
> DJANGO_DATABASES_DEFAULT_ENGINE="django.db.backends.postgres_psycopg2"
> 
> OR add some preprocessing and flexibility:
> 
> |
> 
> 
> [databases]
> default.url =${env:SOME-USER-KEY1}
> default.engine =${env:SOME-USER-KEY2}
> 
> 
> [env]
> SOME_USER_KEY1=a defaultvalue fordefaultdatabase URL
> SOME_USER_KEY2=a defaultvalue fordefaultdatabase engine
> 
> |
> 
> where [env] section will by filled automatically by Django from os.environ.
> 
> This also gives a possibility to extend settings.ini via development.ini, 
> where [databases] can be overridden with developer settings (without env
> access)

I've done something similar before, and I think it's fine, but this is
an example of precisely the sort of more-magical opinionated approach to
settings that I don't think belongs in Django core; it's perfectly
possible to implement it as a third-party package on top of Django's
current settings handling, for those who like it.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5720E786.7020306%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Making Django more PaaS-friendly

2016-04-13 Thread Carl Meyer
Hi James et al,

In general, I like the idea of adding a helper to Django to read
settings from the environment. I think this helper should be kept as
simple and non-magical as is reasonable. Thus:

- I'm in favor of a flexible helper function that can be manually used
in a settings file to read an explicitly-named env var and coerce it
using any arbitrary callable (that gives enough power to "coerce" using
something as simple as `int` or something as complex as
`parse_database_url` that turns a DSN string into a Django db dict),
with some handling of defaults. Below I give some usage examples of the
function I use for that, and link to its code.

- I'm opposed to anything more magical than that, e.g. anything that
tries to set up an all-dancing automatic mapping or translation between
env var names and setting names, or tries to enforce some particular env
var prefix. The minor gains in conciseness with this kind of thing
aren't nearly worth the losses in flexibility in how you construct your
settings file or name your env vars.

- I'm also opposed to conflating the issue of reading env vars with
orthogonal stuff like class-based settings or patterns for using
multiple settings files.

More comments (and sample code) below:

On 04/11/2016 12:26 PM, Aymeric Augustin wrote:
>> On 11 Apr 2016, at 19:39, Curtis Maloney  wrote:
>>
>> 1. All env vars can have a fallback value (the method in my case)
>> 2. You can mark an env var as not having a fallback value, and it will raise 
>> an error at start up if not set.
> 
> There’s an additional complexity here.
> 
> Usually, it’s best for settings read from environment variables:
> 
> a. To use a default value in dev, even if that means a small degradation in
> functionality. This allows developers to start working on the project without
> adding dozens of exports to their $VIRTUAL_ENV/bin/postactivate, and to add
> just the values they need when they work on specific parts like integrations
> with third-party systems.
> 
> b. To crash if no value is provided in prod, in order to catch configuration
> errors upfront.
> 
> One might think of switching the behavior depending on settings.DEBUG, but
> that won't work because the switch is required to load settings properly.

I totally agree with all this. My projects have a separate MODE setting
which is either 'dev' or 'prod', and I use a utility to read settings
from the environment which supports variable defaults (or lack of
default) by mode, exactly as you outline above. Specifically, it works
like this:


env = EnvParser('PRJ_MODE')

SECRET_KEY = env('PRJ_SECRET_KEY', default={'dev': 'dev-secret'})

The effect of this is that the mode ('dev' or 'prod') will be read from
the `PRJ_MODE` env var (usually `PRJ` would be a short code specific to
the project name). Then if the mode is 'dev', no `PRJ_SECRET_KEY` env
var is required, and 'dev-secret' will be used as the default. But if
the mode is 'prod', the server will fail to start up unless
`PRJ_SECRET_KEY` is found in the environment.

I wrote this to allow for any number of arbitrarily-named modes, but in
practice I've never used anything but 'dev' and 'prod' (staging and demo
sites are generally set up as 'prod'). Hardcoding to those two modes
would perhaps allow a nicer syntax for specifying defaults, by using
separate kwargs instead of the dict.

Anyway, FWIW, here's the code I use:
https://gist.github.com/carljm/69b8e351dac87f4c3f5b440632727fdb

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/570EA4C6.6080700%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Django template 'if ... is' feature design

2016-04-07 Thread Carl Meyer
Hi Steve,

On 04/07/2016 05:09 PM, Stephen Kelly wrote:
> Am I right to think that using 'is' with numbers is expected to work like 
> this in the DTL? :

The situation with integers is roughly the same as with strings, except
that I believe CPython interns _all_ small integers, not just those that
are literals in the code. This happens to make DTL appear a bit more
"consistent" with CPython for integers, but the correct advice is still
the same: don't ever use `is` with integers, whether in Python code or DTL.

The `is` operator tells you whether two variables refer to the same
object. This is sometimes a meaningful question to ask for instances of
classes with mutable state, or for singletons like None, True, and
False, but it's not a useful or meaningful question to ask about
immutable objects like strings and integers, because there is no
perceptible difference (other than from `is`, `id`, and perhaps memory
usage) resulting from whether the implementation happened to choose to
reuse the same object or not.

> In [120]: c["a"] = 1
> 
> In [121]: t = e.from_string("{% if 1 is 1 %}yes{% else %}no{% endif %}")
> 
> In [122]: print t.render(c)
> yes
> 
> In [123]: t = e.from_string("{% if 1 is a %}yes{% else %}no{% endif %}")
> 
> In [124]: print t.render(c)
> yes
> 
> In [125]: c["a"] = 1.0
> 
> In [126]: t = e.from_string("{% if 1 is a %}yes{% else %}no{% endif %}")
> 
> In [127]: print t.render(c)
> no
> 
> In [128]: t = e.from_string("{% if 1 is 1.0 %}yes{% else %}no{% endif 
> %}")
> 
> In [129]: print t.render(c)
> no
> 
> In [130]: t = e.from_string("{% if 1.0 is 1.0 %}yes{% else %}no{% endif 
> %}")
> 
> In [131]: print t.render(c)
> no
> 
> Would it be useful to add those as unit tests and document what can be 
> expected of 'if ... is'?

No to tests, because we would be adding tests for undefined and
unreliable behavior.

It might be worth adding a short documentation note. We largely want to
avoid documenting Python's behavior in the Django docs, but a short note
in the template `is` docs reminding people not to ever use `is` with
strings or numbers might be worthwhile.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5706EAB6.6070207%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Django template 'if ... is' feature design

2016-04-07 Thread Carl Meyer
Hi Steve,

On 04/07/2016 04:36 PM, Stephen Kelly wrote:
> Anyway, that makes at least two people telling me that 
> 
> In [81]: "True" is "True"
> Out[81]: True
> 
> and
> 
> In: t = e.from_string(
>  "{% if \"True\" is \"True\" %}yes{% else %}no{% endif %}")
> In: t.render(c)
> 
> Out: 'no'
> 
> is 'the same behavior' and seems to be what you expect.

The point is that you shouldn't ever expect any consistent behavior when
using `is` with strings (whether in Python code or in Django templates),
because that behavior is not defined by the Python language, it's
implementation dependent. It happens that the CPython implementation
interns short string literals, so they are the same object in memory,
but this doesn't happen for longer strings, or strings that aren't
generated from literals. Those "string literals" in the Django template
are not string literals from Python's perspective, so they are not
interned, so they are not the same object.

There's no reasonable way to "fix" this, because there is no reliable
Python behavior to try to match. Any fix would just be encouraging use
of `is` with strings; the right answer is "never use `is` with strings"
(or with integers, for exactly the same reasons).

In other words, the value of `"string" is "string"` in Python is
effectively accidental and almost certainly useless (it does correctly
and reliably tell you whether the two string objects are the same
object, but for immutable objects this information is
performance-optimization-dependent and useless), so there is no
"correct" behavior here for the DTL to match.

Carl


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5706E49B.6080302%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: must we include the source files for all minified JavaScript?

2016-04-06 Thread Carl Meyer
It seems like a reasonable request to me. It's just generally useful to
have the original source matching the minified code available, e.g. for
debugging an issue with that code. In theory you can go find the
corresponding non-minified code yourself, but it's not always easy to
find exactly the right corresponding version, and there's not much
reason to make that work necessary.

Carl

On 04/06/2016 10:58 AM, Tim Graham wrote:
> From a Trac ticket [0]:
> 
> 
> In Debian, we like to have the sources of everything that we ship and we
> consider that minified javascript files are not really scripts (which
> are their own source).
> 
> There are currently two problematic files in Django:
> 
> 
> js_tests/qunit/blanket.min.js
> django/contrib/admin/static/admin/js/vendor/xregexp/xregexp.min.js
> 
> 
> To remedy this I would suggest to either:
> 
>   * directly use a non-minified file like you do for many other
> javascript files (ex:
> django/contrib/admin/static/admin/js/SelectFilter2.js)
>   * store a non-minified file next to the minified file (same name
> without the ".min" part) 
> 
> [0] https://code.djangoproject.com/ticket/26474
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-developers+unsubscr...@googlegroups.com
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/acca9df5-2377-4e6c-b103-6167c9e4bd87%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/57054173.4040307%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: [Review Request] Added support for database delegated fields (like AutoField)

2016-02-11 Thread Carl Meyer
Hi Anssi,

On 02/11/2016 02:51 AM, Anssi Kääriäinen wrote:
> I created a proof-of-concept pull request for improved deferred fields
> loading. See https://github.com/django/django/pull/6118. In short, it is
> faster and cleaner (no more dynamically created subclasses for deferred
> loading!) than the previous implementation, but we have a couple of
> backwards incompatibilities:
> 1) |Model.__init__()| will be called with value DEFERRED for those
> fields which are deferred. Previously those fields were not given to
> |__init__()| at all.
> 2) The class level attribute Model._deferred must be removed (we
> can't have class level attribute as we don't have dynamic classes any more)
> 3) A potential name clash for fields when one has a class method and
> field with clashing names.
> 
> Of these the Model.__init__() case could be problematic for those users
> who have overridden __init__().
> 
> Before doing any further work on this we should decide if the
> Model.__init__() problem is bad enough to stop this cleanup, and if so,
> do anybody have any ideas of how to avoid the problem?

Overriding Model.__init__() is specifically called out in the
documentation as something to avoid, and there is no mention in the
documentation of what is passed to __init__() in the case of defer/only.
I think this is a fine change to just document in the release notes.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/56BCDB8F.4080408%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: add prefered/default protocol in the sites framework #26079

2016-01-25 Thread Carl Meyer
On 01/25/2016 09:01 AM, Tim Graham wrote:
> Oh, looks like django-hosts also has this problem. It looks like in some
> cases the URLs will be correct (due to the PARENT_HOST setting, I think,
> which is 'djangoproject.dev:8000' in the djangoproject.com dev
> settings), but if you are browsing a different host (e.g. the docs
> site), then those URLs use the value from the Site model.
> 
> Another thought: I've sometimes thought that the Site model violates the
> principle that you shouldn't put configuration in your database. I guess
> there's some usefulness to having a ForeignKey to the site, but... would
> it be feasible to offer a SITES setting that could be used instead? e.g.
> 
> SITES = {
> 1: {'scheme': 'http', 'domain': example.com, 'name': 'My Site'},
> ...
> }
> 
> That would solve the portability issue, I think, since you could easily
> override the necessary parts in development settings. Not sure if it'd
> be palatable to deprecation the models though. I don't think maintaining
> both versions would be worthwhile in the long run.

+1 to this, I've long felt that the Site model was an anti-pattern. I
don't know if it's worth deprecating and switching to a different
system, though; it'd be a relatively painful deprecation for those using
it, I would guess.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/56A6473F.50203%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-01-18 Thread Carl Meyer
I've updated DEP 5 with a new round of clarifications and tweaks based on 
the most recent feedback: 
https://github.com/django/deps/compare/62b0...master

Carl 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a8695453-51d5-470d-bbfe-367b5f741ccc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: re-thinking middleware

2016-01-18 Thread Carl Meyer
On 01/10/2016 04:54 PM, Raphaël Barrois wrote:
> I've got only one minor suggestion to the "backwards compatibility"
> section of the DEP.
> 
>> It currently states that "If the ``MIDDLEWARE`` setting is provided
>> [...], the old ``MIDDLEWARE_CLASSES`` setting will be ignored.
> 
> I suggest that, instead, this fails loudly if both ``MIDDLEWARE`` and
> ``MIDDLEWARE_CLASSES`` are set.
> This would prevent projects from keeping the two versions around and
> having to remember which one is currently in use.

Great suggestion, thanks. I've added it to the todo checklist on the PR:
https://github.com/django/django/pull/5949

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/569D7BC3.1010303%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-01-18 Thread Carl Meyer
On 01/11/2016 04:52 PM, Shai Berger wrote:
> On Monday 11 January 2016 01:54:58 Raphaël Barrois wrote:
>> I've got only one minor suggestion to the "backwards compatibility"
>> section of the DEP.
>>
>>> It currently states that "If the ``MIDDLEWARE`` setting is provided
>>> [...], the old ``MIDDLEWARE_CLASSES`` setting will be ignored.
>>
>> I suggest that, instead, this fails loudly if both ``MIDDLEWARE`` and
>> ``MIDDLEWARE_CLASSES`` are set.
>> This would prevent projects from keeping the two versions around and
>> having to remember which one is currently in use.
>>
> 
> If the failure is too loud, it makes it hard to use the same settings file 
> for 
> testing with multiple versions of Django -- a practice used by Django itself, 
> and I think also by some reusable apps. We've run into this issue with the 
> change of the database test settings.

Yes; I think a checks-framework warning (like what we do for
`TEMPLATE_*` vs `TEMPLATES`) is the appropriate level; likely to catch
an unintentional usage, but silence-able if the duplication is intentional.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/569D7B30.2020109%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-01-18 Thread Carl Meyer
Hi Mat,

On 01/12/2016 04:23 PM, Mat Clayton wrote:
> +1 as well, this is really great work, on both patches!
> 
> Modifying the middleware stack mid request would be very useful for us.
> 
> We currently run 3 copies of the same code, with essentially different
> urls.py and middlewares loaded, each one is used to run a variation of
> the site, for example www/mobile web and api, and have slightly
> different requirements, which are handled in middlewares, for example
> api calls are authenticated by access tokens and www/mobile use cookies.
> 
> Ideally we'd love to consolidate our deployment into a single set of
> python web workers, but to date its been a little too complex to be
> worth it. We could however use the url dispatching patch and PR1 to
> write a middleware which could load in the necessary middleware stack
> based on the requirements of each request.
> 
> So my preference therefore would be for PR1 as I can see a few edge
> cases it would enable solutions for.

I don't think it's worth penalizing the performance of the common case
in order to enable these edge cases. I think it should be possible to
add a "per-request middleware" feature atop PR #2 in a fully
backwards-compatible way without compromising performance in the common
case (essentially by compiling multiple alternative request-paths in the
handler instead of just one), but this should be handled as a separate
feature, built in a separate pull request; it's out of scope for DEP 5.
It solves a completely different problem from the one DEP 5 is aiming to
solve.

I also think there are fairly easy ways to solve your problem in the
meantime without the per-request middleware feature, for instance with
hybrid middleware that conditionally does something-vs-nothing (or
one-thing-vs-another) depending on some attribute attached to the request.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/569D79B2.60601%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-01-11 Thread Carl Meyer
On 01/10/2016 04:54 PM, Raphaël Barrois wrote:
> I've got only one minor suggestion to the "backwards compatibility"
> section of the DEP.
> 
>> It currently states that "If the ``MIDDLEWARE`` setting is provided
>> [...], the old ``MIDDLEWARE_CLASSES`` setting will be ignored.
> 
> I suggest that, instead, this fails loudly if both ``MIDDLEWARE`` and
> ``MIDDLEWARE_CLASSES`` are set.
> This would prevent projects from keeping the two versions around and
> having to remember which one is currently in use.

Yes, that's a good idea. I think I had vaguely in mind that this wasn't
possible because the global default for MIDDLEWARE_CLASSES is non-empty,
but I guess we can still check to see whether the actual value != the
global default.

> Beyond that, I love the overall design of this upgrade :)

Great!

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5693EE27.5000209%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: slow migrations

2016-01-09 Thread Carl Meyer
On 01/09/2016 09:29 AM, Shai Berger wrote:
> There are two kinds of data migrations, generally speaking.
> 
> One is the kind that is needed to facilitate some schema migrations. Changing 
> the type of some field, for example, usually involves creating the new field 
> (schema migration), transforming the data from the old field to the new 
> (data), 
> and then removing the old field (and perhaps some renaming; schema). This 
> kind 
> of migrations, indeed, can be just removed when squashing.
> 
> The other is migrations which *create* data -- fill in tables for database-
> implemented-enums, for example. If you remove these, you are going to break 
> your tests (if you do this and haven't broken your tests, your tests are 
> missing -- I'd go as far as calling them broken).

I consider this second kind of data migration to be marginally smelly,
and avoid it whenever possible. If an enum's possible values will change
infrequently, I'd much rather have it in code than in the database
(that's almost certainly more efficient, too). If its values will change
frequently, then it's less likely that I have a strong "default set",
and I'd be more likely to leave the creation of initial values to be an
explicit part of deployment setup (and have the tests also explicitly
create the values they need), rather than use a data migration to
enforce a particular initial set.

That said, I do recognize that people use data migrations to create
initial data, and that there are cases where it makes sense.

> The second kind is quite common. Having a built-in command that resets 
> migrations and ignores them is, IMO, encouragement to skip testing, and I 
> think we shouldn't do that.

I agree that a "resetmigrations" command is dangerous. For my own usage
I'd be more worried about losing custom SQL schema or indexes added via
RunSQL than I would about initial-data migrations, but either way it's
certainly potentially lossy. I'd only be -0 on adding it, but it would
need to come with strong warnings in the documentation.

> Some notion (as Carl, I think, mentioned) of 
> "squashable data migrations" -- essentially, telling the two kinds apart -- 
> would be helpful. but not solve the problem completely, because, ultimately, 
> the second kind exists. We need to figure out how to help users deal with 
> them.

Hmm, I'm not sure if there is any general way to "help users deal with"
non-squashable RunPython/RunSQL migrations -- did you have any
particular ideas in mind? If they aren't squashable, they aren't
squashable, and I don't see what Django can do about that that doesn't
require the user to think carefully about their actual migration set and
manually modify migrations. You can't even do something like move
initial-data migrations to the end of the set to allow optimizing away
more schema alterations, because the migration may create the initial
data in a way that only works at that particular moment in schema history.

I think we'll be doing pretty well (it would certainly make a big
difference for my projects) if we can just allow users to identify
migrations that can be ignored when squashing.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5691F59D.4050407%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: PostGres 9.5 Upsert

2016-01-09 Thread Carl Meyer
On 01/08/2016 05:13 PM, bliyan...@rentlytics.com wrote:
> Postgres 9.5 has added the functionality for UPSERT aka update or
> insert.  Any interest in aligning UPSERT on the db layer with the
> get_or_create or update_or_create functionality in django?  Sounds like
> my company would be interested in doing the work if the PR will get the
> traction.

That'd be great! I can't see any reason why a good PR for that wouldn't
be accepted. There's no reason to be using the ugly algorithms in
`get_or_create` or `update_or_create` when native UPSERT is available at
the DB level.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/569193B4.70508%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-01-08 Thread Carl Meyer
On 01/08/2016 11:38 AM, Carl Meyer wrote:
> On 01/08/2016 04:51 AM, Markus Holtermann wrote:
>> Nitpicking, I would also name the settings variable MIDDLEWARES 
>> (i.e.plural) as it is a list of middlewares, not just one.
> 
> Well, this is a matter of some debate :-) See e.g.
> https://github.com/rack/rack/issues/332, where a number of people
> fervently argue that "middleware" is a "mass noun" along the lines of
> e.g. "furniture", and that it is even incorrect to say "a middleware"
> (much like you would never say "a furniture"); instead we should always
> say "a middleware component."
> 
> I think those people are fighting a lost battle; the usage of
> "middleware" as singular is already well established in the Python
> world, even outside Django; people frequently talk about "a WSGI
> middleware."
> 
> That said, my ear still prefers "middleware" as both the singular and
> the plural (there are such words in English) and dislikes "middlewares."
> So I'd prefer to stick with MIDDLEWARE and haven't changed it in the
> spec. But if most people think MIDDLEWARES is better, I won't stand in
> the way.
> 
> We could also go with something like MIDDLEWARE_FACTORIES or
> MIDDLEWARE_PIPELINE and avoid the issue altogether :-)

To take it one level up, historically speaking, using the term
"middleware" to mean "something between the web server and a web app's
controller/view" is questionable to begin with; note the wikipedia page
for "middleware" doesn't even mention this use of the term once. So
another option is to find a different term entirely and let the word
"middleware" die out from the Django lexicon. One advantage of this
would be that we wouldn't have to keep talking about "new-style" vs
"old-style" middleware :-)

But on the whole I think this is fighting a lost cause too - there are
WSGI middleware, Flask has middleware, Rack has middleware. So wikipedia
and historical usage aside, our use of the term is by now already well
established in the web-frameworks world. I think changing to a different
word will probably create more confusion than anything.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/569009A7.3030003%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-01-08 Thread Carl Meyer
Hi Ryan,

On 01/08/2016 11:48 AM, Ryan Hiebert wrote:
> I hadn't considered the option of just returning ``get_response``.
> I really like it, because it avoids both raising an exception and
> importing a common symbol.
> 
> While class-based middleware factories couldn't just do the same thing
> in the __init__ method, they could do that in a __new__ method instead,
> and then we could make returning the ``get_response`` function the
> "one way to do it."

I'd be more sympathetic to this viewpoint if Django hadn't already had
the `MiddlewareNotUsed` exception for many years in this very role.
There's no reason to break it if we don't absolutely have to. And since
we've got it, and it works in any scenario, it may as well be the
preferred method.

Implementing `__new__` is something many Python developers aren't
familiar with and don't do regularly, so I think it's not a good choice
as the primary recommendation.

Also, disabling middleware at setup time isn't all that common, so an
import isn't a big deal.

That said, I expect plenty of people (probably myself included) to use
the "just return ``get_response``" trick, whether it's documented or
not, and I think that's totally fine.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5690059B.90201%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-01-08 Thread Carl Meyer
Thanks everyone for the reviews. I've pushed a new commit addressing
most of the issues raised:
https://github.com/django/deps/commit/62b0ee351727cb0e7ef41ba6dd2f3f7280a219de

More comments and replies (to various people in the thread) below:

On 01/08/2016 09:01 AM, Florian Apolloner wrote:
> PR1 (allows changing the middlewares during the middleware run): 
> https://github.com/django/django/pull/5591
> PR2 (static middleware chain, seems simpler and probably the way to
> go): https://github.com/django/django/pull/5949/

Yes, the latter is what Pyramid does and what I had in mind; in fact,
its performance benefits are one of the justifications given in the DEP
for why to use the factory approach rather than simple functions. I
don't think "changing middlewares per request" is a use case that's
worth the performance implications (and it's not a use case we currently
support anyway).

Thanks for the super-quick turnaround on implementation!

On 01/08/2016 10:35 AM, Aymeric Augustin wrote:
> The only (and minor) concern I have is about allowing function-based
>  middleware factories to return None.
> 
> In the spirit of “there should be only one way to do it”, I would 
> require raising MiddlewareNotUsed explicitly to disable a middleware.
> A middleware factory that returns None would cause an 
> ImproperlyConfigured exception. Otherwise middleware could be skipped
> by mistake, if the developer forgets to return the middleware from
> the factory. This is especially a concern for production-only 
> middleware that developers can’t run locally.
[snip]

Yes, you and Ryan are absolutely right, allowing `None` was a mistake
and I've removed it from the spec. For function-based middleware,
there's also the option to simply return the ``get_response`` callable
you were given, which has the same effect of "skipping yourself." That
falls out naturally from the concept, so I don't see any reason to
disallow it, but we don't need to mention it in the docs if we want to
stick to "one way to do it."

On 01/08/2016 04:51 AM, Markus Holtermann wrote:
> I would, however, include the exception handling in the examples 
> provided in section "Specification" as that is an integral part of 
> middlewares, too.

Good idea, I've made that change in the commit linked above.

> Nitpicking, I would also name the settings variable MIDDLEWARES 
> (i.e.plural) as it is a list of middlewares, not just one.

Well, this is a matter of some debate :-) See e.g.
https://github.com/rack/rack/issues/332, where a number of people
fervently argue that "middleware" is a "mass noun" along the lines of
e.g. "furniture", and that it is even incorrect to say "a middleware"
(much like you would never say "a furniture"); instead we should always
say "a middleware component."

I think those people are fighting a lost battle; the usage of
"middleware" as singular is already well established in the Python
world, even outside Django; people frequently talk about "a WSGI
middleware."

That said, my ear still prefers "middleware" as both the singular and
the plural (there are such words in English) and dislikes "middlewares."
So I'd prefer to stick with MIDDLEWARE and haven't changed it in the
spec. But if most people think MIDDLEWARES is better, I won't stand in
the way.

We could also go with something like MIDDLEWARE_FACTORIES or
MIDDLEWARE_PIPELINE and avoid the issue altogether :-)

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5690020C.8090106%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: re-thinking middleware

2016-01-08 Thread Carl Meyer
Hi Curtis,

On 01/08/2016 04:31 AM, Curtis Maloney wrote:
> In general, I love it.

Great!

> It's MUCH simpler for people to write and comprehend... requires no
> classes [but IMHO the callable class is "cleaner"] and allows for
> configurable middlewares easily...
> 
> I do wonder, though... how the anti-import-strings factions will
> react... I'm sure it can, at least, support direct callables being in
> the MIDDLEWARE list, not just strings?

Well, I guess I'm part of that faction, and I wrote the proposal :-)

The settings file is one place where import-strings are pretty much
unavoidable in Django, because you can't safely import anything into
your settings model that indirectly imports models, or certain other
parts of Django that require settings to be loaded. I don't _like_ this,
but it's the reality, at least until we get rid of settings as a global
singleton (if that ever happens).

We could support directly including middleware factories instead of just
import strings in your MIDDLEWARE setting, but I don't think that's a
good idea, as encouraging people to import stuff from their project into
their settings file is almost certain to end with it blowing up in their
face at some point.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/568FE95E.8060401%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


re-thinking middleware

2016-01-07 Thread Carl Meyer
Hi all,

Back at the Django Under the Hood sprints in November, Florian and I
started batting around some ideas for fixing some of the problems with
the existing middleware abstraction. Florian put together an initial
pull request with a POC for some of those ideas. I've now updated the
proposal (incorporating some implementation ideas from Pyramid's
middleware equivalent) and written it up in the form of a DEP. You can
find the DEP at

https://github.com/django/deps/blob/master/draft/0005-rethinking-middleware.rst

and I'm also pasting the full text below, to ease in-line commenting via
email.

Comments welcome!


DEP 0005: Re-thinking middleware


:DEP: 0005
:Author: Carl Meyer
:Implementation Team: Florian Apolloner
:Shepherd: Carl Meyer
:Status: Draft
:Type: Feature
:Created: 2016-01-07
:Last-Modified: 2016-01-07

.. contents:: Table of Contents
   :depth: 3
   :local:


Abstract


The existing Django "middleware" abstraction suffers from an absence of
strict layering and balanced in/out calls to a given middleware. This
DEP proposes an improved abstraction for wrapping the request cycle in
strictly layered pre-view and post-view actions.


Motivation
==

In theory, and per `the documentation`_, ``process_request`` will be
called for each incoming request, ``process_response`` will be called
for each outgoing response, and ``process_exception`` will be called in
case of an uncaught exception.

This description seems to imply the invariant that if
``process_request`` is called, either ``process_response`` or
``process_exception`` will later be called on that same middleware in
that same request cycle. Django itself has in the past included
middleware (the now-defunct ``TransactionMiddleware``) that implicitly
relied on this invariant.

In fact, due to the short-circuiting and exception-handling behavior of
various middleware methods, this invariant does not hold. It is possible
for a middleware to have its ``process_request`` method called, but then
never see its ``process_response`` or ``process_exception`` called for
that request (e.g. in case of an uncaught exception in a "later"
middleware method).

It is also possible for a middleware to never see its
``process_request`` method called for a given request (because an
earlier middleware's ``process_request`` returned a response), but still
have its ``process_response`` or ``process_exception`` method called on
that response.

This lack of strict in/out layering makes it impossible to safely
implement some types of middleware (such as ``TransactionMiddleware``),
and requires verbose defensive programming: e.g. even if
``process_request`` sets a certain attribute on the request,
``process_response`` on that same middleware can't assume that that
attribute will be present on the request it receives.

This is the primary problem that this DEP intends to solve.

.. _the documentation:
https://docs.djangoproject.com/en/stable/topics/http/middleware/


Acknowledgment
==

The proposed API in this DEP is modelled on Pyramid's `Tween`_ concept
(the author and implementor of this DEP developed a very similar idea
independently at a Django sprint before reading about Tweens).

.. _Tween:
http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/hooks.html#registering-tweens

Specification
=

This DEP introduces a new setting, ``MIDDLEWARE``, which contains an
ordered list of dotted paths to middleware factories.

A middleware factory can be written as a function that looks like this::

def simple_middleware(get_response):
# one-time configuration and initialization

def middleware(request):
# code to be executed for each request before
# the view is called

response = get_response(request)

# code to be executed for each request/response after
# the view is called

return response

return middleware

Or it can be written as a class with a ``__call__`` method, like this::

class SimpleMiddleware(object):
def __init__(self, get_response):
self.get_response = get_response

# one-time configuration and initialization

def __call__(self, request):
# code to be executed for each request before
# the view is called

response = self.get_response(request)

# code to be executed for each request/response after
# the view is called

return response

In prose instead of examples: a middleware factory is a callable that
takes a ``get_response`` callable and returns a middleware. A middleware
is a callable that takes a ``request`` and returns a ``response``. (Just
like a view! Turtles all the way down!)

The ``get_response`` callable provided by Django might be the actual
view (if this is the last listed 

Re: Vote on Jira as bugtracker

2016-01-07 Thread Carl Meyer
Hi Andrey,

On 01/07/2016 04:23 PM, Andrey Antukh wrote:
> On Fri, Jan 8, 2016 at 12:42 AM, Florian Apolloner
> The DRFv2 (as DRFv3) as far as I know is licensed using MIT permissive
> license that does not impide take the source and re-license it under
> other license. 
> 
> taiga.io  initially have started using the DRF just
> like a dependency but over time the dev-team found the needs monky-patch
> many and many parts of it... Later, the dev-team have decided just
> include it in the code base and remove useless (for taiga) code and add
> additional features. 
> 
> Much of code/improvements is contributed back to the DRF (v2), other as
> third party packages and some other is no, because the approaches are
> diverged and the changes are just to much opinionated/coupled to the
> taiga usage.
> 
> I hope I have solved your question.

I am not a lawyer, but it seems to me the problem is not the use of the
code (which the license does allow), but the re-licensing and the lack
of the original DRF license in your codebase.

You are allowed to use the code and re-distribute it, but AFAIK you are
not allowed to re-license it unless you are the copyright owner. And the
terms of the DRF license do require that the DRF license text be
preserved along with any redistribution of the DRF code, which (as far
as I can see) Taiga currently doesn't do.

So if I'm right (which I may well not be), I think the issue is easily
resolved by including the original DRF license text in your base/api/
directory, and clarifying that that license (not the AGPL) is the
license that applies to all code taken from DRF.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/568F0D05.3050901%40meyerloewen.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: slow migrations

2016-01-07 Thread Carl Meyer
On 01/07/2016 03:03 PM, Aymeric Augustin wrote:
> As far as I understand, the CPU cost comes from generating a full set of
> model classes for each step of the migration history. That’s consistent
> with the profile sent by Florian.
> 
> I usually end up throwing away the migration history and regenerating a
> new set of migrations when I get to that point. This requires truncating
> the django_migrations table manually and faking the new set of migrations.
> 
> If the project doesn’t use data migrations, squashmigrations may achieve
> the same effect. Sadly real-life projects tend to have data migrations
> whose only purpose is to run once in production. They prevent full
> squashing.

FWIW, I've also done a hybrid of these two options, where I generate
fresh initial migrations rather than actually using squashmigrations
(for the same reason, to avoid problems with data migrations), but then
I still keep the old migrations around for a transition period and use
the `replaces` attribute (the same one added automatically by
`squashmigrations`) on the new initial migrations. Then later (once the
new migrations are deployed everywhere) delete the old migrations and
the `replaces` attr.

Effectively this is similar to what you're doing, it just takes
advantage of the `replaces` feature to avoid manually fiddling with the
migrations table on each deployment.

If I (or anyone else) ever gets around to it, I think
https://code.djangoproject.com/ticket/24109 could make the actual
squashmigrations command usable for real projects, by letting you just
mark certain migrations as "please ignore when squashing".

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/568EE2B2.1040501%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


Re: Deprecations vs. backwards-incompatible changes for tightening behavior that hides probable developer error

2015-12-21 Thread Carl Meyer
Hi Tim,

On 12/21/2015 08:09 AM, Tim Graham wrote:
> I'd like to ask for opinions about whether or not deprecations are more
> useful than making small backwards incompatible changes when they move
> Django in a direction toward unhiding probable developer error.
> 
> An example from a past release is the validation of fields in
> select_related() [1]. This was done as a backwards-incompatible change
> in 1.8: invalid fields immediately raised an error. Would you rather a
> change like that go through the normal deprecation cycle (pending
> deprecation, deprecation, error)? My reasoning against it is that it
> continues to hide probable errors in new and old code (especially in the
> first release since PendingDeprecationWarning isn't loud by default) and
> the cost for users of fixing existing code is likely low compared to the
> overhead of a deprecation cycle in Django. Did anyone find this was not
> the case in their own projects?
> 
> The question came up again with the behavior of the default error views
> silencing TemplateDoesNotExist when a custom template name is specified [2].

Obviously the first question is whether the behavior in question is
documented. If it is, then a deprecation path is definitely required. On
the other hand, if the current behavior contradicts the documentation,
then it's a bug and can just be fixed.

But the two examples you cite have an edge case with behavior that is
unspecified in the documentation. The next question I would ask in those
cases is "Can I imagine a scenario in which someone is intentionally
taking advantage of the current behavior - it's not just hiding an error
in their code? How likely is that scenario? How reasonable is their use
case? And if we conclude that there may be people out there with a
reasonable use case for the current behavior, how hard would it be for
them to change their code to continue working as-is, if Django's
behavior is changed?"

So to take the TemplateDoesNotExist case with the built-in error views:

The one scenario I can imagine is that someone has written their own set
of reusable error views wrapping the default ones with a different
default template naming convention, but they still want to use their
custom reusable error views in projects that may or may not choose to
actually provide the custom error templates.

I think this use case is reasonable and the scenario is possible, but
I'll concede that it may be unlikely. But if someone were doing this, I
would consider it a perfectly reasonable use of the existing API
contract ("nonexistent templates fall back to the default error view
content"), not abuse of a bug.

To adapt their code to the new behavior, they would need to add a
try/except for TemplateDoesNotExist in their wrappers. Not totally
trivial (and might require them to write some new tests), but not
unreasonable.

So for me, this case falls in a murky area. I tend to think it's better
for us to err on the side of being more backwards-compatible when the
current behavior isn't clearly a bug, and someone might be reasonably
relying on it. But I concede that it's in a grey area, and wouldn't
stand in the way of a decision to just treat it as a bugfix.

In cases where we really can't construct any realistic scenario for
intentional use of the current behavior, then I think we just make the
change right away and call it a bug fix. I don't think it's a bad thing
to cause code that was previously silently broken to start breaking loudly.

I think `select_related` with invalid field names is closer to this end
of the spectrum. I _can_ construct a scenario for intentional use of
that, but it's very unlikely and feels much more like abuse of a buggy
API. So if I did earlier suggest that the select_related change should
have gone through a deprecation path, I hereby retract that suggestion :-)

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/567848C6.5060504%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature


  1   2   3   4   5   6   7   8   >