Re: The state of per-site/per-view middleware caching in Django

[h3]

I think for the moment, the easy fix for anonymous forms it either to
put them on a different page or
to load them with ajax.

This way the forms and thus the tokens gets generated only when
needed.

If caching and performances are a big concern, I think those
alternative are win/win solutions.

You solve your problem and remove load.

My 2¢

On Oct 21, 2:39 pm, Carl Meyer  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 10/21/2011 07:02 AM, Jim Dalton wrote:
>
> > 1. Fixing #9249 and #15855. I hear your philosophical concerns about
> > #9249 but the ubiquity of Google Analytics means we must do fine some
> > way to fix it (IMO). Addressing these two tickets would at least
> > ensure page caching wasn't actually broken. I'll try to jump in on
> > those if I have time later next week. #9249 in particular seems quite
> > close.
>
> > 2. Clarifying the documentation. I think an admonition in the page
> > caching section of the docs which outlined the present challenges a
> > developer might face implementing it would probably have done the
> > trick for me when I was first glancing at it. I can open a ticket on
> > that next week, again if I have time.
>
> > It'd be great if these two got in for 1.4.
>
> Agreed - any work you're able to put in on any of these is very welcome.
>
> > 3. Addressing the other stuff is I guess for now a sort of "some day"
> > goal. I continue to feel strongly that it's a worthy goal,
> > particularly given that CSRF and contrib.auth are such fundamental
> > parts of most projects and that they really are the only two things
> > that stand in the way of page caching being a viable option in many
> > projects. If anyone else gets inspired by this goal let me know,
> > otherwise I'm content for the time being to let it stew.
>
> I take your point that it might be possible to do a cache-tweaking API
> that could allow the cache to be more aggressive around auth and CSRF
> with less coupling (though you'd still end up sprinkling cache-specific
> stuff into auth and CSRF with your approach). I remain pretty skeptical
> about whether this is a good idea; it seems like it could significantly
> increase the surface area for bugs in the cache middleware
> implementation, and just generally make the implementation harder to
> maintain with correct behavior. (I have some painful experience in this
> area: CACHE_MIDDLEWARE_ANONYMOUS_ONLY is the one existing, and
> relatively simple, instance of the type of enhanced caching logic you're
> talking about, and I made some fixes to it in the 1.3 cycle that I then
> later had to fix again due to unanticipated side effects of the first
> change). But at this point this is all kind of hand-waving without code
> to look at.
>
> You might also consider what's possible to do outside of core as a
> third-party alternative to Django's caching middleware. When you're
> proposing major and somewhat experimental changes, that can be a
> powerful way to demonstrate that the idea is workable, and makes it a
> lot easier to pick up users and advocates; people are generally more
> willing to try out a third-party tool than to run or test with a patched
> Django.
>
> Carl
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk6hvEkACgkQ8W4rlRKtE2edBACfdsW7IHoDKBrpwzwGGMx+ww5g
> U+AAoLZLn1CA6c1644kzsnZRZ6xaW60B
> =QBjT
> -END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django CMS Developer

[Aymeric Augustin]

On 21 oct. 2011, at 17:31, Berto Abreu wrote:
> I wanted to share an opportunity for a full time career in NYC.


The topic of this mailing list is the development of Django itself.

As such, even if you were advertising for a position related to Django, your 
message wouldn't be appropriate.

-- 
Aymeric Augustin.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

i18n docs

[Aymeric Augustin]

Hello,

Maybe it's just me, but I find the i18n docs quite confusing, and I often 
struggle to locate the information I need:
https://docs.djangoproject.com/en/1.3/topics/i18n/

For instance, I don't understand why:
- the index page immediately dives into gettext technicalities;
- /topics/i18n/localization/ describes how to write .po files and how to use 
the localization features of Django (USE_L10N), two seemingly unrelated topics;
- implementation details of i18n are split between /howto/i18n/ and 
/topics/i18n/deployment/.

In the context of my timezone support branch, I'd like to clarify all this.

I regard internationalization as the combination of three things:
1 - USE_I18N = translation, per language
=> translating text in code and templates -- USE_I18N is an 
unfortunate choice for the translation setting, but that's history)
2 - USE_L10N = localization, per country
=> formatting dates, times, numbers -- possibly currencies and 
units in the future
3 - USE_TZ = time zone, per region
=> converting datetimes

I think there should be an index page and one page for each of these three 
topics.

What do you think? Have I missed something?

Thanks,

-- 
Aymeric Augustin.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: The state of per-site/per-view middleware caching in Django

[Jim Dalton]

On Oct 21, 2011, at 8:04 AM, Kääriäinen Anssi wrote:

> I do not know nearly enough about caching to participate fully in this 
> discussion. But it strikes me that the attempt to have CSRF protected 
> anonymous page cached is not that smart. If you have an anonymous submittable 
> form, why bother with CSRF protection? I mean, what is it protecting against? 
> Making complex arrangements in the caching layer for this use case seems like 
> wasted effort. Or am I missing something obvious?

First issue is that CSRF can matter for anonymous users. From here 
http://www.squarefree.com/securitytips/web-developers.html#CSRF:

Attacks can also be based on the victim's IP address rather than cookies:

• Post an anonymous comment that is shown as coming from the victim's 
IP address.
...
• Perform a distributed password-guessing attack without a botnet. 
(This assumes they have a way to tell whether the login succeeded, perhaps by 
submitting a second form that isn't protected against CSRF.)

So two very common uses cases for anonymous forms are log in forms and 
anonymous comment forms, both of which are potentially vulnerable. I guess I 
feel like it's quite common to have forms on a page these days even for 
anonymous users.

Second is -- and I don't know about this -- but I don't know how well CSRF 
handles authentication conditionally. Like if I have a page and let's say that 
page has forms in it for logged in users but nothing for anonymous user, can I 
conditionally exempt the formless page from CSRF? I have no idea, but buy 
default I presume it's on and I presume the cache is varying on it.

So, yes, you could probably optimize a lot of this to sort of skip around the 
CSRF issue and it's not a deal breaker. But my main argument has been the 
ubiquity of CSRF + user authentication in Django projects to me means a 
solution to both of these is a requirement for page caching to become easy and 
applicable in most scenarios.

> 
> The following is from the stupid ideas department: Maybe there could be a 
> "reverse cache" template tag, such that you would mark the places where you 
> want changing content as non-cacheable. You would need two views for this, 
> one which would construct the "base content" and then another which would 
> construct the dynamic parts. Something like:
> 

Your idea sounds a lot like the "server side include" or "two phased template 
rendering" approach that I know some people are doing. Here's an excellent 
example of this approach being used in EveryBlock (from two years ago):

http://www.holovaty.com/writing/django-two-phased-rendering/

And looks like some core devs have been involved at some point in this 
implementation of that concept:

https://github.com/codysoyland/django-phased

That looks almost exactly like your idea: "django-phased contains a 
templatetag, phased, which defines blocks that are to be parsed during the 
second phase. A middleware class, PhasedRenderMiddleware, processes the 
response to render the parts that were skipped during the first rendering."

I guess that was sort of what I was hinting at in my previous discussion about 
how to handle CSRF. In the link he is taking it to the next level (where even 
logged in users get the page and that stuff is done after.

Anyhow it's obviously a sensible conceptual approach. It would be a stretch to 
fit that into the existing page caching approach of Django obviously.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Sane defaults for Startapp and Startproject

[Carl Meyer]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Rich,

On 10/21/2011 10:21 AM, Rich Jones wrote:
> How would you guys feel about a manage.py startexample
> 
> which would create an example application? A simple, templated hello
> world with an example model and an example view and an example
> template?

Indifferent? I kinda feel like following the tutorial gives you this
already, and I'm not real excited about having another "sample project"
in core to maintain and update. OTOH I understand that a lot of people
learn best by tweaking existing working code, and if this were done
outside of core the people who most need it (beginners) are not likely
to be aware of it. Maybe there'd be a way to make it a downloadable part
of the docs rather than a management command?

I'm not going to work on this or commit it, but if other people
(including at least one core dev) are excited about it I wouldn't stand
in the way. So I guess that's a -0.

Carl
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6huGUACgkQ8W4rlRKtE2dAWQCg6mKdlRHDkk+sPYJvafjyms5C
Q2wAoN70oNfxh3vTzd4W7DqrUCzO/RD8
=mCZ7
-END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Sane defaults for Startapp and Startproject

[Rich Jones]

How would you guys feel about a manage.py startexample

which would create an example application? A simple, templated hello
world with an example model and an example view and an example
template?

R

On Oct 20, 4:50 pm, Aymeric Augustin
 wrote:
> On 20 oct. 2011, at 22:22, Carl Meyer wrote:
>
> >> In order to make deployment easier, I also recommend putting
> >> site-wide templates and templatetags in an application, and keeping
> >> TEMPLATE_DIRS empty, but that's a personal preference.
>
> > Just out of curiosity, why would you say it makes deployment easier to
> > put static assets and templates into an installed app, rather than using
> > TEMPLATE_DIRS/STATICFILES_DIRS? (Templatetags are a different issue;
> > those have to go in an app).
>
> I see two advantages:
> - everything is an app: there are no special directories to handle when you 
> build a package,
> - templates and static files work with the same settings in development and 
> production.
>
> These arguments aren't very strong:
> - you still have to ensure that static files and templates are properly 
> packaged within apps,
> - generally, you have different settings in development and production 
> anyway, and the overhead of changing TEMPLATE_DIRS/STATICFILES_DIRS is 
> negligible.
>
> That's why I said it's a personal preference, maybe even an aesthetic matter 
> :)
>
> Best regards,
>
> --
> Aymeric Augustin.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Django CMS Developer

[Berto Abreu]

Hello all!

I wanted to share an opportunity for a full time career in NYC. Below
you will find the full job description. Let me know if you or anyone
that you may know of is interested. Thank you in advance!
Platform Engineer

OPERATIONAL POSITION DESCRIPTION

OVERVIEW OF ROLE
Primary Responsibility:
The Senior Software Engineer will be responsible for backend web
development.  He or She will work closely with product, content and
front-end teams creating a content publishing platform to support
http://fuse.tv and additional digital products.

Reports to:
Director Platform Engineering

Manages:
This position does not have direct reports

Key Non-Reporting Relationships:
Key contact Description/Purpose
1.  Director Platform Engineering   Discussion of implementation
strategies
2.  Project Manager Requirements reviews and project timelines
3.  Application Engineering Manager Discussion of implementation
strategies and requirements reviews
4.  Multiplatform Manager   Requirements reviews and meetings on
additional device support

OPERATIONAL ACTIVITIES OF THIS POSITION
1.  Code Fuse's consumer facing and internal facing digital products
2.  Develop public Fuse APIs so 3rd parties can consume our content
3.  Consume 3rd party APIs such as Ooyala, Facebook and Twitter to
enrich our products
4.  Develop a CMS for the content team to publish content with user
roles and approval and scheduling workflows
5.  Integration of client-side code of HTML, CSS and JavaScript

KEY MEASURES OF SUCCESS
2.  Operating metric(s):

a.  100% on time digital products

QUALIFICATIONS
1.  Management Skills
a)  Able to manage time efficiently and meet deadlines

2.  Technical/Functional Competencies
a)  5+ years’ experience developing CMS systems using products like
Drupal, Django, WordPress or similar.
b)  Expert level knowledge of Ruby, Php or Python to build web
applications.
c)  MySQL database architecture and development
d)  Someone who takes pride in writing and documenting clean code and
excited about peer code reviews
e)  Expertise with developer tools like Firebug, Selenium or Charles
Proxy

3.  Key Experiences/Achievements
a)  Experience working in a team environment
b)  Interest or experience with discovering emerging tools and selling
teammates on their benefits
c)  Experience with version control software like SVN
d)  Experience implementing social APIs such as Facebook and Twitter
e)  Experience with defect tracking software such as JIRA and the agile
scrum development process
f)  Experience developing scalable cloud systems with AWS and EC2
g)  Experience with any of the following a plus: Ooyala, Vevo,
thePlatform, MongoDB, Cassandra, Lucene, Java, Hibernate, Android or
iOS development

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

RE: The state of per-site/per-view middleware caching in Django

[Kääriäinen Anssi]

I do not know nearly enough about caching to participate fully in this 
discussion. But it strikes me that the attempt to have CSRF protected anonymous 
page cached is not that smart. If you have an anonymous submittable form, why 
bother with CSRF protection? I mean, what is it protecting against? Making 
complex arrangements in the caching layer for this use case seems like wasted 
effort. Or am I missing something obvious?

The following is from the stupid ideas department: Maybe there could be a 
"reverse cache" template tag, such that you would mark the places where you 
want changing content as non-cacheable. You would need two views for this, one 
which would construct the "base content" and then another which would construct 
the dynamic parts. Something like:

page_cached.html:
... expensive to generate content ...
{% block "login_logout" non_cacheable %}
{% endblock %}
... expensive to generate content ...

You would generate the base page by a cached render view:

def page_view_cached(request, id):
if cached(id):
return cached_content
else:
... expensive queries ...
return cached_render("page_cached.html", context, ...)

The above view would not be directly usable at all, you would need to use a 
wrapper view which would render the non-cacheable parts:

def page_view(request, id):
# Below would return quickly from cache most of the time
cached_portions = page_view_cached(request, id)
return render_to_response("page.html", context={cached: cached_portions, 
user:request.user})

where page.html would be:
{% extends cached %}
{% block login_logout %}
{% if user.is_authenticated %}
Hello, user!
{% else %}
login
{% endif %}
{% endblock %}

That seems to be what is really wanted in this situation. The idea is quite 
simply to extend the block syntax to caching. A whole another issue is how to 
make this easy enough to be actually usable, and fast enough to be actually 
worth it.

 - Anssi


From: django-developers@googlegroups.com [django-developers@googlegroups.com] 
On Behalf Of Jim Dalton [jim.dal...@gmail.com]
Sent: Friday, October 21, 2011 16:02
To: django-developers@googlegroups.com
Subject: Re: The state of per-site/per-view middleware caching in Django

On Oct 20, 2011, at 6:02 PM, Carl Meyer wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi Jim,
>
> This is a really useful summary of the current state of things, thanks
> for putting it together.
>
> Re the anonymous/authenticated issue, CSRF token, and Google Analytics
> cookies, it all boils down to the same root issue. And Niran is right,
> what we currently do re setting Vary: Cookie is what we have to do in
> order to be correct with respect to HTTP and upstream caches. For
> instance, we can't just remove Vary: Cookie from unauthenticated
> responses, because then upstream caches could serve that unauthenticated
> response to anyone, even if they are actually authenticated.
>
> Currently the Django page caching middleware behaves pretty much just
> like an upstream cache in terms of the Vary header. Apart from the
> CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting, it just looks at the response,
> it doesn't make use of any additional "inside information" about what
> your Django site did to generate that response in order to decide what
> to cache and how to cache it.
>
> This approach is pretty attractive, because it's conceptually simple,
> consistent with upstream HTTP caching, and conservative (quite unlikely
> to serve the wrong cached content).
>
> It might be possible to make it "smarter" in certain cases, and allow it
> to cache more aggressively than an upstream cache can. #9249 is one
> proposal to do this for cookies that aren't used on the server, either
> via explicit setting or (in a recently-added proposal) via tracking
> which cookie values are accessed. If we did that, plus special-cased the
> session cookie if the user is unauthenticated and the session isn't used
> outside of contrib.auth, I think that could possibly solve the
> unauthenticated-users and GA issues.
>
> However, this (especially the latter) would come with the cost of making
> the cache middleware implementation more fragile and coupled to other
> parts of the framework. And it still doesn't help with CSRF, which is a
> much tougher nut to crack, because every response for pages using CSRF
> come with a Set-Cookie header and probably with a CSRF token embedded in
> the response content; and those both mean that response really can't be
> re-used for anyone else. (Getting rid of the token embedded in the HTML
> means forms couldn't ever POST without JS help, which is not an option
> as the documented default approach). You can mark some form-using views
> that are available to anonymous users as csrf-exempt, which exposes you
> potentially to CSRF-based spam, but isn't a security issue if you aren't
> treating authenticated submissions any differently from

Re: The state of per-site/per-view middleware caching in Django

[Jim Dalton]

On Oct 20, 2011, at 6:02 PM, Carl Meyer wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Hi Jim,
> 
> This is a really useful summary of the current state of things, thanks
> for putting it together.
> 
> Re the anonymous/authenticated issue, CSRF token, and Google Analytics
> cookies, it all boils down to the same root issue. And Niran is right,
> what we currently do re setting Vary: Cookie is what we have to do in
> order to be correct with respect to HTTP and upstream caches. For
> instance, we can't just remove Vary: Cookie from unauthenticated
> responses, because then upstream caches could serve that unauthenticated
> response to anyone, even if they are actually authenticated.
> 
> Currently the Django page caching middleware behaves pretty much just
> like an upstream cache in terms of the Vary header. Apart from the
> CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting, it just looks at the response,
> it doesn't make use of any additional "inside information" about what
> your Django site did to generate that response in order to decide what
> to cache and how to cache it.
> 
> This approach is pretty attractive, because it's conceptually simple,
> consistent with upstream HTTP caching, and conservative (quite unlikely
> to serve the wrong cached content).
> 
> It might be possible to make it "smarter" in certain cases, and allow it
> to cache more aggressively than an upstream cache can. #9249 is one
> proposal to do this for cookies that aren't used on the server, either
> via explicit setting or (in a recently-added proposal) via tracking
> which cookie values are accessed. If we did that, plus special-cased the
> session cookie if the user is unauthenticated and the session isn't used
> outside of contrib.auth, I think that could possibly solve the
> unauthenticated-users and GA issues.
> 
> However, this (especially the latter) would come with the cost of making
> the cache middleware implementation more fragile and coupled to other
> parts of the framework. And it still doesn't help with CSRF, which is a
> much tougher nut to crack, because every response for pages using CSRF
> come with a Set-Cookie header and probably with a CSRF token embedded in
> the response content; and those both mean that response really can't be
> re-used for anyone else. (Getting rid of the token embedded in the HTML
> means forms couldn't ever POST without JS help, which is not an option
> as the documented default approach). You can mark some form-using views
> that are available to anonymous users as csrf-exempt, which exposes you
> potentially to CSRF-based spam, but isn't a security issue if you aren't
> treating authenticated submissions any differently from
> non-authenticated ones.
> 
> Generally, I come down on the side of skepticism that introducing these
> special cases into the cache middleware really buys enough to be worth
> the added complexity (though I could be convinced that #9249 is worth it).

Thanks Carl. This is definitely a good, clarifying response to what I was 
mulling around about.

A few thoughts of my own to add here:

* You and Nihan are certainly right about upstream caches. Regardless of what 
we do here, we'll have to vary by cookie in the response header. This makes 
sense for a site that offers authentication: Django needs to check on every 
page view to see if the user is authenticated, so we can't have the upstream 
cache holding on to a page for us.

* Agreed about how the "smartness" comes at the cost of brittleness if the 
implementations are too tightly coupled. That said, I can squint and sort of 
see an implementation that could thread the needle here. It would require 
something like:

- An API in the cache middleware instructing it to ignore certain cookies for 
the purposes of caching (i.e. something along the lines of #9249).

- Some kind of "pre-fetch" hook in the cache middleware. Whether it's a flag in 
the request object, a signal or something else, give other systems the ability 
to look at a request before it hits the FetchFromCacheMiddleware and either 
allow or prevent the response from being pulled from the cache. E.g if there 
was a flag request.invalidate_cache that defaults to False, the contrib.auth 
app could, in combination with the above, pull the session id from 
consideration in the cache key and do an authentication check on its own, 
invalidating the cache on its own if the user is authenticated. The core idea 
is what you already suggested, I'm more illustrating here that this can 
conceivably be implemented as an API, making it less brittle.

- Some kind of "post-fetch" hook in the cache middleware, combined with a 
retooling of the CSRF middleware. This is getting in the clouds here a bit, but 
a hook on the opposite end of the fetch operation could allow the CSRF app to 
add its token after the response was pulled from the cache. I say we're in the 
clouds here because for something like this to work the CSRF would have to do a 
little