Re: Logging format decision

[Simon Willison]

On Jan 14, 1:57 am, Russell Keith-Magee 
wrote:
> The other issue that I think still needs to be addressed is the
> internal usage of logging by Django itself.

My biggest hangup with the logging implementation was figuring out
exactly when the logging configuration code should actually run. The
problem is that Django's startup execution order is poorly defined -
stuff relating to settings is lazily evaluated the first time it's
needed, and there's no real documented place to put code that needs to
be executed once (the fact so many registration things end up being
called from urls.py is a good indicator that this is a problem).

This has put me off using signal handlers in my own code in the past,
and it also came up with the logging work.

I think Django needs a defined and documented execution order on
startup. I'm not entirely clear why we've avoided this in the past -
I've read http://www.b-list.org/weblog/2007/nov/05/server-startup/ but
it hasn't convinced me that a defined startup order for things like
registration patterns, signal handlers, logging configuration is a bad
thing.

Personally, I'd like to see this happen as part of the instantiation
of a single Django "site object" which handles all requests to the
server. We almost do this at the moment - if you look at the code in
http://code.djangoproject.com/browser/django/trunk/django/core/handlers/
each handler basically instantiates an object once which then has its
get_response() method called for every incoming request. Unfortunately
the time at which this object is instantiated seems to be pretty ad-
hoc - how and when it happens depends on if you're running under WSGI,
mod_python or the development server.

Defining Django's execution order is a pretty big job, but it would be
a great thing to achieve for 1.3. The obvious thing to tie it to would
be INSTALLED_APPS - it could be as simple as having an API promise
that the first thing Django does when it "starts" is to run through
each application in INSTALLED_APPS looking for and importing an
autoload.py module. I imagine we'll find that the time at which models
are registered is critical (what if autoload.py wants to use a model
that hasn't been loaded in to the AppCache yet?) and may need to do
more than one pass through INSTALLED_APPS. Plenty of details to figure
out.

Cheers,

Simon
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Model validation incompatibility with existing Django idioms

[Simon Willison]

A couple of related tickets filed today about model validation:

http://code.djangoproject.com/ticket/12513
http://code.djangoproject.com/ticket/12521

The first one describes the issue best - the new model validation code
breaks the following common Django convention:

form = SecretQuestionForm( {"secret_question":"foo", "answer":"bar"} )
if form.is_valid():
p = form.save(commit=False)
p.user = request.user
p.primary_contact = somecontact
p.save()

The problem is that is_valid() notices that some of the required
fields in SecretQuestionForm (a ModelForm) have not been provided,
even if those fields are excluded from validation using the excludes=
or fields= properties. The exception raised is a
UnresolvableValidationError.

This definitely needs to be fixed as it presumably breaks backwards
compatibility with lots of existing code (it breaks a common
ModelAdmin subclass convention as well, see #12521). Can we just
change the is_valid() logic to ignore model validation errors raised
against fields which aren't part of the ModelForm, or is it more
complicated than that?

Cheers,

Simon
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Design and code review requested for Django string signing / signed cookies

[Simon Willison]

On Jan 4, 6:18 pm, James Bennett  wrote:
> Simon, the amount of pushback this is getting, and the changes which
> need to be made to start bringing it up to snuff, make me feel far too
> nervous about this being ready in time to make 1.2 at all. I know
> you've put in the effort to shepherd this along, but I'm starting to
> think it's time to push this to the 1.3 release cycle (especially
> since 1.2 alpha freeze is tomorrow, and I don't think there's any way
> it'll be even alpha-ready by then).

I certainly don't think we should check this in for the alpha freeze.

We do however need to consider the places in Django that are already
using hmac / md5 / sha1 (contrib.formtools and middleware.csrf for
example). Even if we don't add the signed cookies feature to 1.2,
fixing any problems with our existing use of crypto should not be
affected by the feature freeze. There's not much point in implementing
this logic in several different places, so I think we should keep
targeting the django.utils.signed module for 1.2.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Design and code review requested for Django string signing / signed cookies

[Simon Willison]

On Jan 4, 2:45 pm, Jordan Christensen  wrote:
> Is there a good way to make it forward upgradeable? Allow the
> developer to decide on the shorter SHA-1 hash or the (theoretically)
> more secure SHA-256?

There is - we can expand the BACKEND setting which is already in place
for signed cookies (but not for other clients of the Signer class). I
think we should do this. For one thing, it would mean we could provide
a backend which uses the Google keyczar Python library instead of the
code that we write. keyczar is properly audited, but depends on
PyCrypto so isn't appropriate as a required dependency of Django.

Another thing we can do is use SHA-256 but truncate the HMAC to 128
characters. Apparently it's perfectly fine to do this - it's being
discussed in the programming.reddit thread at the moment.

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Design and code review requested for Django string signing / signed cookies

[Simon Willison]

Had some good feedback on news.ycombinator and programming.reddit -
you can follow the threads here:

http://news.ycombinator.com/item?id=1030290
http://www.reddit.com/r/programming/comments/ald1m/calling_crypto_security_experts_help_review_the/

tptacek on news.ycombinator pointed out a timing attack based on our
use of an insecure string comparison (an attack which affected Rails a
while ago). We can fix that using a constant time string comparison
such as this one:

http://code.google.com/p/keyczar/source/diff?spec=svn414=411=414=unidiff=/trunk/python/src/keyczar/keys.py

ascii on programming.reddit has convinced me to ditch the sep=":"
argument and hard code the separator. Customising that doesn't feel
like a feature anyone will ever need. They also repeated the advice to
use SHA-256 - I think I'll almost certainly have to give up my quest
for shorter signatures :(

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Design and code review requested for Django string signing / signed cookies

[Simon Willison]

>From Jordan Christensen on Twitter: 
>http://twitter.com/thebigjc/status/7366243197

"@simonw why sha-1 instead of sha-256? NIST has recommended not using
SHA-1 in new systems: http://bit.ly/6bIf5h;

I chose sha-1 over sha-256 for reasons of signature length. A base64
encoded signature generated with hmac/sha1 is 27 characters long. The
same thing using hmac/sha256 is 43 characters long. If you're planning
on using signatures in cookies and URLs that's quite a big difference
(43 characters is more than half of the maximum 80 characters needed
to safely transmit URLs in plain text e-mails, e.g. for account
recovery links).

My understanding is that the collision weaknesses discovered in SHA-1
are countered by the use of HMAC. Here's Bruce Schneier on the matter:

http://www.schneier.com/blog/archives/2005/02/sha1_broken.html

"It pretty much puts a bullet into SHA-1 as a hash function for
digital signatures (although it doesn't affect applications such as
HMAC where collisions aren't important)."

Despite the confusing API name, we're doing HMAC here, not digital
signatures - so I think we're OK. If I'm wrong I'm sure a crypto geek
will set me straight pretty quickly.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Design and code review requested for Django string signing / signed cookies

[Simon Willison]

mp with the signature


The second class in the signed.py module is a subclass of Signer that
appends a unix timestamp to the string before it is signed. This
allows the unsign() method to specify a max_age - if the signed value
is older than that max_age, it is discarded.

Here's the code:

class TimestampSigner(Signer):
def timestamp(self):
return baseconv.base62.from_int(int(time.time()))

def sign(self, value, salt='', sep=':'):
value = smart_str('%s%s%s' % (value, sep, self.timestamp()))
return '%s%s%s' % (
value, sep, self.signature(value, salt=salt)
)

def unsign(self, value, salt='', sep=':', max_age=None):
value, timestamp = super(TimestampSigner, self).unsign(
value, salt=salt, sep=sep
).rsplit(sep, 1)
timestamp = baseconv.base62.to_int(timestamp)
if max_age is not None:
# Check timestamp is not older than max_age
age = time.time() - timestamp
if age > max_age:
raise SignatureExpired, 'Signature age %s > %s
seconds' % (
age, max_age
)
return value

As you can see, the timestamp is appended to the value before it is
calculated, and a max_age can be passed to the unsign() method and
will be checked before the string is returned.

Again, as a space saving the unix timestamp is a base62 encoded - this
shrinks it and makes it suitable for inclusion in a URL (for example).

Signing cookies
---

Signed cookies are provided using two new methods on core Django
objects: a get_signed_cookie() method on the Django request object and
a set_signed_cookie() method on the Django response. Here's what a
very simple Django view might look like that reads the name from a
signed cookie and sets that cookie if a new name has been provided in
a POST parameter:

def index(request):
name = request.get_signed_cookie('name')
set_cookie = False
if name in request.POST:
name = request.POST['name']
set_cookie = True
response = render_to_response('index.html', {
'name': name,
})
if set_cookie:
response.set_signed_cookie('name', name)
return response

The get_signed_cookie() method is implemented here:

http://github.com/simonw/django/blob/signed/django/http/__init__.py#L66

And set_signed_cookie() is here:

http://github.com/simonw/django/blob/signed/django/http/__init__.py#L388

Both of these methods take an optional 'salt' argument - but even if
you don't provide a salt, the name of the cookie will be used as the
salt. If you DO provide a salt the actual salt used will be
cookie_name + your_salt.

SECRET_KEY rotation
---

I'm still working out the details for this, but the final feature we
want to provide is a mechanism for rolling out a new SECRET_KEY
without breaking everything that has been signed with an old one. I
believe this is best practice, but I'm eager to hear if it isn't.

The plan is to support an optional OLD_SECRET_KEYS setting which is a
list of old secret keys which should still work for unsigning but
should not be used for signing.

The UpgradingSigner class here is a start at this code:

http://github.com/simonw/django/blob/signed/django/utils/signed.py#L142

A piece of Django middleware will be provided that quietly "upgrades"
any signed cookies that have been signed using one of the older keys,
replacing them with a cookie signed with the current SECRET_KEY. This
middleware will need to know the names of the cookies that should be
upgraded and what salt was used to generate them.

The process for rolling out a new SECRET_KEY then will be this:

1. Put the current secret key in OLD_SECRET_KEYS:

OLD_SECRET_KEYS = ['your-current-secret-key']

2. Put the NEW secret key in SECRET_KEY:

SECRET_KEYS = 'your-new-secret-key'

3. Turn on the Django signed cookie upgrading middleware:

MIDDLEWARE_CLASSES += (
'django.middleware.signedcookies.UpgradeOldSignedCookies',
)

4. Tell that middleware which cookies to upgrade:

SIGNED_COOKIES_TO_UPGRADE = (
('name', 'salt-for-name'),
)

Now wait a week while a bunch of cookies get upgraded, then remove the
key from OLD_SECRET_KEYS.

Sending feedback


Does this look sane? Have we overlooked anything? Is there anything we
can do to make this more secure by default?

I'll read any replies here, or you can e-mail feedback to simon AT
simonwillison.net. Please say if you don't want stuff sent to that
address to be shared in public.

Thanks,

Simon Willison

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: 1.2 Feature freeze

[Simon Willison]

On Dec 23, 8:34 am, James Bennett  wrote:
> * URLs-1 (get_absolute_url replacement)

The code's not ready - I have the get_url / get_url_path methods but I
don't have a good enough hold yet on what actually needs to be done to
replace the existing get_absolute_url behaviour and all of it's
idiosyncrasies.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Call for feedback: django.utils.signed and signed cookies

[Simon Willison]

Having talked to James about this I'm holding off on the commit until
we've had it reviewed by real cryptographers. I'll aim to get it in
before the 1.2 beta feature freeze.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Congratulations Alex and Russell on committing multidb!

[Simon Willison]

... and congratulations to Marc Garcia and Jannis Leidel for the i18n/
l10n improvements that just went in as well. Also congratulations to
everyone for the other awesome commits going in at the moment. There
are far too many congratulations to spell them all out at the moment
(with hindsight I wish I hadn't started!).

1.2 is going to rock.

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Congratulations Alex and Russell on committing multidb!

[Simon Willison]

And a big congratulations to all involved. Here's the changeset log
(on GitHub since Trac seems not to like being linked to at the
moment):

http://github.com/django/django/commit/836d297e68d6a63103780295adebf6eaf6779611

And here's the documentation:

http://docs.djangoproject.com/en/dev/topics/db/multi-db/

Really excited about this - can't wait to start putting it to good
use.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Call for feedback: django.utils.signed and signed cookies

[Simon Willison]

On Dec 22, 6:22 am, Russell Keith-Magee 
wrote:
> As far as the patch itself is concerned, looks good to me. My only
> other request would be a serving of dogfood - if we're going to
> include a signed cookie module, it would be nice to prove that it can
> actually be used by actually using it.

I'd love to see the cookie backend of the new django.contrib.messages
feature switch over to using get/set_signed_cookie. I'm happy to put
together the patch, or alternatively one of the people who worked on
that feature can do it.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Call for feedback: django.utils.signed and signed cookies

[Simon Willison]

On Dec 22, 12:52 am, Johannes Dollinger
 wrote:
> I'd like some more kwargs to Signer and TimestampSigner. Mostly what's  
> inhttp://dpaste.com/136418/(except the `separator` kwarg, which was  
> a bad idea as it depends on encode()): Signer(serializer=...) and  
> TimestampSigner(ttl=...).

The first few versions of the code had a bunch more stuff along the
lines of that dpaste (which expires in 6 days, so here's a gist copy
of it: http://gist.github.com/261572 )

After struggling with it for quite a while, I decided that having a
single class that handled signing, serialization and compression
wasn't the right approach - it was doing far too much. Instead, I
changed the design so the Signer's only responsibility was generating
signatures, appending them to strings and verifying that they were
correct. The encryption/serialization was extracted out and moved
directly in to the signed.dumps() and signed.loads() functions.

This solved a bunch of problems I was having with the code - too many
subclasses, confusing amounts of multiple inheritance for mixing
together different subclassed behaviours - and generally made
everything feel a lot more cohesive.

That's also why I added the BACKEND setting for setting a cookie
signing backend - that way, users with specific ideas about how their
signed cookies should work can write their own class that does pretty
much anything they like.

Is there any functionality in particular that you think should be
resurrected from the more-complex-signing-classes? I'm very keen on
keeping the serialization and compression stuff out of Signer.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: What do people think about the get_absolute_url proposal?

[Simon Willison]

On Dec 8, 3:23 am, Rick Yazwinski  wrote:
> I think that this may be too simplified:
>         protocol = getattr(settings, "PROTOCOL", "http")
>         domain = Site.objects.get_current().domain
>         port = getattr(settings, "PORT", "")
>
> Many sites put load balancers and https hardward acceleration in front
> of their web interfaces.  This would obscure the protocol and port
> info from Django.

Aha! I was pretty confused by this thread, since I didn't remember
writing the above code. It turns out that's the problem with making
proposals like this on a wiki...

http://code.djangoproject.com/wiki/ReplacingGetAbsoluteUrl?action=diff=10

My original proposal can still be seen here:

http://code.google.com/p/django-urls/source/browse/trunk/django_urls/base.py

It's basically this:

def get_url(self):
if hasattr(self.get_url_path, 'dont_recurse'):
raise NotImplemented
try:
path = self.get_url_path()
except NotImplemented:
raise
prefix = getattr(settings, 'DEFAULT_URL_PREFIX', 'http://
localhost')
return prefix + path
get_url.dont_recurse = True

So for the common case it invents a new DEFAULT_URL_PREFIX setting
which can be used to ensure get_url uses the correct domain. If you
want to use a threadlocal request for this instead that's fine - just
define your own get_url() method that does that.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Call for feedback: django.utils.signed and signed cookies

[Simon Willison]

I've made some changes based on the feedback in this thread:

http://github.com/simonw/django/commit/802952bbb8b763e65ee545c6a8f39524b20e147c
"Use sha('signer' + secret_key + salt) to derive the key for use in
the
signature() method, addressing feedback from the django-developers
list"

The default signature() method now looks like this:

def signature(self, value, salt=''):
# Derive a new key from the SECRET_KEY, using the optional
salt
key = sha_constructor('signer' + self.key + salt).hexdigest()
return base64_hmac(value, key)

The secret key (self.key here) is now never used directly. Instead, a
sha1 hash of the salt 'signer' plus the secret key plus any additional
salt is used as the key for the signature. sha1 is used here as
protection against weird key length extension attacks (like the one
that affected the Flickr API recently).

http://github.com/simonw/django/commit/4ed44c2bce5000d6c78c3a26b84d08f636b3589c
"RAISE_ERROR now capitalised to emphasize that it is a constant"

http://github.com/simonw/django/commit/20f3a693b99ec6af0f91eecb31046e8a07dc7151
"Signed cookies now automatically include the name of the cookie as
part of the salt"

http://github.com/simonw/django/commit/68c52f0b995447d93bce1db486b23a27b918da73
"Moved get_cookie_signer in to utils.signed"

New patch is attached to the ticket. Is there anything else I need to
address before checking it in?

http://code.djangoproject.com/ticket/12417

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Loading Backends

[Simon Willison]

On Dec 21, 2:22 pm, Marty Alchin  wrote:
> Looking over Simon's patch for signed cookies, I noticed that yet
> another feature needs to write its own "load a backend from a setting"
> function.

Yup - and as I copied and pasted it from somewhere else I thought
exactly the same thing. Is this kind of refactoring something we can
do after the 1.2 feature freeze? If so I think it would be worth
cleaning up at least some of this stuff.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Call for feedback: django.utils.signed and signed cookies

[Simon Willison]

On Dec 21, 2:40 pm, Russell Keith-Magee 
wrote:
>  * I'm not sure I like this being in django.utils. To me, it feels
> like something that should be in django.core - along with caching,
> serialization, etc, signing is a core piece of functionality that a
> website will need to implement. django.utils is full of necessary, but
> largely ancillary code.

I have to admit I don't 100% understand the difference between core
and utils - but I'm happy with moving it across to core if that makes
more sense.

>  * get_cookie_signer() strikes me as something that should be a
> general utility in the signing module, not something buried in
> django/http.

I'd love to see this code factored out as per Marty's suggestion in
another thread. I wrote a backend just for signed cookies because
that's the one part of the implementation that's "baked in" to core
Django APIs - but I expect other bits to be baked in to things like
session handling / auth / form wizards etc over time, so being able to
over-ride the Signer used for those in the future will be important.

>  * django.http.raise_error should be RAISE_ERROR, so it's obvious it's
> a constant

Good point, I'll fix that this evening.

>  * I'm not sure I follow why we need to use base62 (and therefore
> provide our own base encoding method) in the Timestamp signer.

OK, you caught me. This is part of my evil scheme to sneak baseconv.py
in to Django. I'm using it for the signed cookie timestamp to make it
as small as possible (same reason I'm using a base64 of the HMAC hash
rather than the regular hexdigest) - since cookies add overhead to
every single HTTP request I figure anything we can do to make them
smaller is a bonus. I considered using a custom timestamp in seconds
since (2009, 12, 1) but that felt a little weird.

baseconv.py is actually a lot more important for some of the other
ways I use signed cookies. A great example is "recover my account /
forgotten password" e-mails. I like sending a signed URL for those (no
need to store any state) but it needs to be under 72 characters to fit
in an e-mail:

http://simonwillison.net/recover/b23/kfQJafvGyiofrdGnuthdxImIJw/

Where b23 is the base62 encoded user ID and the bit after it is a
signature.

URLs like that often make use of timestamped signatures (I like my
recover account URLs to expire after 24 hours) so again it's useful to
keep them short.

baseconv.py is also useful for things like URL shorteners e.g.
http://swtiny.eu/EZj

>  * In the original mailing list thread, there was discussion about
> prefixing the use of SECRET_KEY with some extra data, like the module
> name. The 'salt' option is one way to achieve this, but I agree with
> Luke and Marty that there should be some mandatory salting.

I'll look at that this evening.

>  * The original mailing list thread (and the wiki) also contained
> discussion about how to deprecate/cycle SECRET_KEY values in the case
> a value was compromised. Having a working solution for this problem
> probably isn't necessary, but it would be good to know that

I've got a plan for that as far as signed cookies go - an optional
piece of middleware which looks for invalid signed cookies in the
request, checks them against an OLD_SECRET_KEYS list and upgrades them
if they can be decrypted that way. This can be implemented completely
separately from the current code, though the ideal implementation
would involve adding a few related methods to the Signer class.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Call for feedback: django.utils.signed and signed cookies

[Simon Willison]

I've uploaded the patch for adding signing and signed cookies to
Django:

http://code.djangoproject.com/attachment/ticket/12417/ticket12417.diff

You can also read the documentation directly on my GitHub branch:

http://github.com/simonw/django/blob/signed/docs/topics/signing.txt
http://github.com/simonw/django/blob/signed/docs/ref/request-response.txt#L224
http://github.com/simonw/django/blob/signed/docs/ref/request-response.txt#L561

Most of the code lives in django.utils.signed (the low-level signing
API) but I've also added a get_signed_cookie() method to HttpRequest
and a corresponding set_signed_cookie() method to HttpResponse:

http://github.com/simonw/django/blob/signed/django/http/__init__.py#L84
http://github.com/simonw/django/blob/signed/django/http/__init__.py#L406
http://github.com/simonw/django/blob/signed/django/utils/signed.py

The code has documentation and unit tests. The documentation isn't
100% complete - I need to improve the explanation of what signing is
and why it is useful and document the new COOKIE_SIGNER_BACKEND
setting which allows users to swap in their own cookie signing
behaviour should they need to.

Most importantly though, the implementation has not yet been peer
reviewed by real cryptographers. With that in mind, would it be
appropriate to check this in before the 1.2 freeze? We would certainly
get the code reviewed before the final 1.2 release.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: adding rosetta for enhanced user experience

[Simon Willison]

On Dec 14, 9:41 pm, Suno Ano  wrote:
> What do folks think about shippinghttp://code.google.com/p/django-rosetta
> with Django? Enabling it per default even? imho that app is totally
> worth being shipped with Django per default.

This came up a couple of months ago. Here's why I don't think it's
appropriate to ship with Django:

http://groups.google.com/group/django-developers/msg/bfd9c35fb95bb35a

Basically, it turns out most translators already have a preferred
toolchain and may well not want to use a web-based tool.

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

What do people think about the get_absolute_url proposal?

[Simon Willison]

This made it to the 1.2 feature list:

http://code.djangoproject.com/wiki/ReplacingGetAbsoluteUrl

If we want this in 1.2, it could be as simple as merging the get_url /
get_url_path methods in to the base Model class, rolling a few unit
tests and writing some documentation. It feels like we should discuss
it a bit first though - the proposal hasn't really seen much
discussion since it was originally put together back in September last
year.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Simon Willison]

On Dec 5, 4:20 pm, Russell Keith-Magee  wrote:
> Trust me - I don't want to do mindless busy work. However, we need to
> have some sort of answer for the admin interface - Django's admin is a
> big selling point for Django for some people, so we can't really
> introduce a huge new feature, and then say "but you can't use it in
> admin". I'm interested in making the least intrusive change that is
> possible without hamstringing future multi-db interfaces.

It strikes me that the admin issue should be solvable entirely in
django.contrib.admin without any changes to the multidb code itself.
Right now you can get most of the job done using a ModelAdmin
subclass:

from django.contrib import admin
from blog.models import Entry

class EntryAdmin(admin.ModelAdmin):
def save_model(self, request, obj, form, change):
obj.save(using='otherdb')

def queryset(self, request):
return Entry.objects.using('otherdb').all()

admin.site.register(Entry, EntryAdmin)

I haven't tested the above so it's probably missing a few cases
(save_fieldsets for example perhaps) but if we document it I think
it's a good enough solution for the moment. Even if we need to
refactor ModelAdmin a bit to ensure the right hooks are available it
still shouldn't be a massive change.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Simon Willison]

On Dec 4, 8:38 am, Simon Willison <si...@simonwillison.net> wrote:
> We now have two blog posts in each database. But, if I do the
> following:
>
> e1.save(using = 'db2')
>
> It has the effect of replacing e3 with the values from e2, since the
> primary keys collide with each other.

I meant to say - the fix is clearly to call save() with the
force_insert=True argument, but it's easy to miss that. Worth
explicitly highlighting in the multi-db docs I think.

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Simon Willison]

On Dec 3, 4:10 pm, Russell Keith-Magee  wrote:
> Alex Gaynor's GSoC project to add multiple database support to Django
> is on the final straight. The only piece of the puzzle that is left is
> updating contrib.gis - but this hopefully won't require any major
> changes outside of the gis tree itself.
>
> Therefore, I'd like to call for feedback and testing of the branch.

I tried this out yesterday - it's excellent. The combination of the
new DATABASES setting and QuerySet.using() turns out to be exactly
enough to get all sorts of interesting work done. It neatly fulfils
the three requirements set out in the original ticket (replication /
sharding / hosting different applications on different databases).

A couple of observations. Firstly, there's a gotcha involving saving
existing objects to a new database:

e1 = Entry.objects.using('db1').create(title = "A blog post",
body="...")
e2 = Entry.objects.using('db1').create(title = "Another blog post",
body="...")
e3 = Entry.objects.using('db2').create(title = "Another blog post",
body="...")
e4 = Entry.objects.using('db2').create(title = "Another blog post",
body="...")

We now have two blog posts in each database. But, if I do the
following:

e1.save(using = 'db2')

It has the effect of replacing e3 with the values from e2, since the
primary keys collide with each other.

This behaviour makes sense if you think about it for more than a
moment, but still probably counts as a gotcha. Worth mentioning in the
documentation at least.

I could see this pattern biting people since that's the obvious way to
copy an object from one database to another.

Secondly, a question: Is there an idea for how this might be made to
work with the admin interface? I was asked this question at DJUGL last
night and wasn't sure of the answer.

Fantastic work on this, very excited to see it merged to trunk.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

How hard would it be to make bits of Django compatible with async servers?

[Simon Willison]

I've been getting very excited about Node.js recently:

http://simonwillison.net/2009/Nov/23/node/

It's basically Twisted / Tornado but in JavaScript, and with the huge
advantage that, because it's brand new, it doesn't have any legacy
blocking APIs. Database access looks like this:

c.query("select * from articles").addCallback(function(rows) {
  pretty_print(rows);
});

Even filesystem access is non-blocking:

posix.cat("/etc/passwd").addCallback(function(content) {
  puts(content);
});

This made me think about non-blocking IO and Django. Theoretically,
many of the services that Django provides could be used in a non-
blocking environment such as that provided by Twisted or Tornado:

* URL routing shouldn't need to be blocking - it's just dispatching to
a function based on a regex (though this is stymied by thread locals)
* Form handling shouldn't need to block on I/O
* If we cache templates in memory instead of reading from disk each
time, rendering them shouldn't need to block on IO either
* SQL statement generation with the ORM shouldn't need to block either

That last one is particularly interesting to me. My understanding is
that much of the ORM refactoring for multi-db consisted of improving
the separation of SQL generation from actual database interaction.
Being able to do something like this would still be incredibly useful:

def my_async_view(request, response):
query = Blog.objects.filter(pubdate__year = 2009).exclude
(tag__slug = 'sport')
def done(rows):
response.render('blog.html', {'rows': rows})
db.execute_async(query, done)

The db.execute_async method would take a queryset and a callback,
execute the query asynchronously and pass the result to that callback.
I've invented an imaginary potential async replacement for the
blocking response=view(request) idiom.

I don't think async support should be baked in to Django core, but
it's a very interesting thing to think about. Should we eventually
break Django up in to smaller reusable libraries this kind of thing
would be an exciting use case.

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=.

Should docs.djangoproject.com link to 1.1 frozen docs by default, not SVN dev docs?

[Simon Willison]

Now that 1.2 features have started to merge (great work everyone
involved with the Email backends and ManyToMany refactorings, they
look fantastic) we're entering one of those relatively rare periods of
instability in trunk. This is exactly the kind of time that we want
most of our regular users to be sticking with the 1.1.1 release, while
the super-engaged community members try out the new stuff as it's
added.

There's just one problem: http://docs.djangoproject.com/ defaults to
displaying the development docs. In fact, there isn't even an option
to see frozen docs for 1.1 (just an option to see them for 1.0). These
are going to start being pretty confusing for 1.1.1 users, for example
the e-mail docs now cover backends:

http://docs.djangoproject.com/en/dev/topics/email/

We include "New in Django Development version" warnings, but it still
feels a bit odd. In a post-1.0 world, I think it would make more sense
for the documentation linked to directly from the
docs.djangoproject.com homepage to cover the most recent official
release, with a bit less emphasis on the development version's docs.
Basically what the Python docs at http://docs.python.org/ do.

I may have missed a previous conversation about this, if so please
feel free to direct me to the old thread.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Best place for code that processes stuff from settings.py once (for logging implementation)

[Simon Willison]

On Oct 11, 2:32 am, Simon Willison <si...@simonwillison.net> wrote:
> Is there a straight forward solution to this that I've been missing,
> or is it an ongoing problem? If it's a problem, can we fix it? What
> makes it so difficult to fix (I'm going to guess there's no easy
> solution, or we would have sorted it out ages ago)? Are there any
> previous discussions I should be reading to catch up on this?

On Twitter Malcolm pointed me here: http://code.djangoproject.com/ticket/5685
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

I'm now tracking the logging implementation in ticket #12012:

http://code.djangoproject.com/ticket/12012

I've started posting patches from my branch there. Again, this is just
exploratory code at the moment but I'm keen to receive as much
feedback on the implementation as possible.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Best place for code that processes stuff from settings.py once (for logging implementation)

[Simon Willison]

I've got to the point in the logging implementation where I want to
add support for logging related settings in settings.py.

The current idea (suggested by Ivan Sagalaev) is to allow for a
standard Django-style setting that look something like this:

LOGGING = {
 'django.db.sql': {
 'handler': 'django.utils.log.StdErrHandler',
 'level': 'DEBUG',
 },
 'django.errors': {
 'handler': 'logging.handlers.SMTPHandler',
 ...
 }
}

So each key is the name of a logger, and the value is a dictionary
that configures various aspects about how log events sent to that
logger be processed (what levels to pay attention to, what handler to
use etc). If a handler isn't mentioned in the LOGGING dictionary (it
will be empty by default) then all log messages sent to that handler
will be silently discarded.

To implement this, I need to write some code that executes only once,
iterates over the django.conf.settings.LOGGING dictionary and uses it
to make the relevant calls to the Python logging framework to set
everything up - the above should map to code something like this
executing:

logger = logging.getLogger('django.db.sql')
logger.setLevel(logging.DEBUG)
logger.addHandler(django.utils.log.StdErrHandler())
logger = logging.getLogger('django.errors')
logger.addHandler(logging.handlers.SMTPHandler(...))

Here's the problem: I need the above to happen once, on startup, some
time AFTER the user's settings have been loaded. I can't figure out
where the appropriate place to do this is.

The best example I've found of code that executes once in Django core
is the code in the Settings class constructor:

http://code.djangoproject.com/browser/django/trunk/django/conf/__init__.py#L62

This is where the time.tzset() call based on the TIME_ZONE setting
happens, for example.

Unfortunately, that code executes the first time anything attempts to
access a property on the django.conf.settings module (which is a
LazySettings module until the first time it is accessed). I could put
the logging stuff in there, but it feels like a bit of a cludge.

The other interesting happens-once execution point I found is the
AppCache class here, which is instantiated once the first time the
django.db.models.loading class is imported:

http://code.djangoproject.com/browser/django/trunk/django/db/models/loading.py#L15

It appears to use the terrifying Borg pattern to avoid being
instantiated more than once.

Neither of these seem like particularly suitable places to put code
that configures logger objects based on the LOGGING setting.

This relates to a wider problem I've had when working with Django
(which I've seen crop up all over the place, but I'm not sure there's
a best practice for handling) - what to do with Django application/
project code that should only be executed once. We had this problem
with class registration for the admin, and ended up hacking around it
with a call to admin.autoload() in the urls.py file - which seems to
have resulted in urls.py becoming a defacto standard place for putting
code that's meant to run once on startup.

For logging, I want advanced users to be able to write their own
Python logging configuration code (see example above) rather than
using the LOGGING setting. Should I tell them to put that in urls.py
as well?

I'm ashamed to admit it, but I tend to shy away from using signals in
my own code purely because I don't know where I should put the code
that registers them.

Is there a straight forward solution to this that I've been missing,
or is it an ongoing problem? If it's a problem, can we fix it? What
makes it so difficult to fix (I'm going to guess there's no easy
solution, or we would have sorted it out ages ago)? Are there any
previous discussions I should be reading to catch up on this?

Alternatively, should I just stick my log configuring code in the
Setting class constructor and leave well alone?

Thanks,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: Tutorial Refresh

[Simon Willison]

On Oct 10, 2:04 pm, Ned Batchelder  wrote:
> My strong feeling is that these two goals will quickly become impossible
> to reconcile.  I think the idea of a conference site is a good one
> (everyone will understand the problem domain, lots of interesting
> avenues to explore, it's not yet-another-blog), but aiming for it to be
> the actual site used for DjangoCon will not work in the long run.

I absolutely agree that aiming for the site produced at the end of the
tutorial to be the exact site used for DjangoCon will lead to both
goals being fulfilled poorly, but I still think they can be addressed
at the same time. Build out a basic conference site for the tutorial,
then use that as the basis for the full DjangoCon site (which can fork
off from the tutorial to add more features). I'd love to see the final
DjangoCon site being open source and demonstrating Django best
practices.

That said, until we hear from the DjangoCon organisers we won't know
if this is a good idea from their point of view.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

On Oct 10, 1:41 pm, Simon Willison <si...@simonwillison.net> wrote:
> Oops, yes you're right - I misread the numbers. A 2.5 second different
> would correspond to around 350,000 log messages (assuming the rate of
> 130,000/second from my microbenchmark) - I have no idea how many
> statements the unit tests execute in total. Once I've written a bit
> more code of course I could always find out using a logger...

I just hacked in a logger that increments a counter for every message
- it indicates that there are 158,527 SQL statements executed by the
test suite, which means we should expect to add slightly over a second
to the time taken for the overall unit tests based on the
microbenchmark.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

On Oct 10, 12:55 pm, Luke Plant  wrote:
> > master: 314.442s
> > logging: 317.096s
>
> > Since there's nothing at all in my logging code that will speed
> > anything up, I'm putting that down to chance (I did run the suite a
> > few times first to "warm up" the VM I used, but like I said, this
> >  is pretty much the opposite of science).
>
> Those numbers are the time in seconds, right? In which case your
> branch is slower, not faster, as I think you are implying.  It could
> still be down to chance, but its only 1% which would be an acceptable
> hit for me anyway.

Oops, yes you're right - I misread the numbers. A 2.5 second different
would correspond to around 350,000 log messages (assuming the rate of
130,000/second from my microbenchmark) - I have no idea how many
statements the unit tests execute in total. Once I've written a bit
more code of course I could always find out using a logger...

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

On Oct 9, 4:54 pm, Jacob Kaplan-Moss  wrote:
> One big question I have is about performance: have you done any a/b
> testing against trunk to see how much adding logging -- especially
> upon every SQL query -- impacts performance?
>
> For me, at least, performance is going to be the critical factor. It's
> clear that even a do-nothing logger will add some overhead. If we're
> talking fractions of percents here, fine... but if there's a
> measurable performance hit that's going to be a big problem, and it's
> probably best to be tracking speed as we go along here.

Nothing scientific yet - I've run the full unit test and got these
results:

master: 314.442s
logging: 317.096s

Since there's nothing at all in my logging code that will speed
anything up, I'm putting that down to chance (I did run the suite a
few times first to "warm up" the VM I used, but like I said, this is
pretty much the opposite of science).

Microbenchmarks I performed with the timeit module suggest we can log
to a NullHandler logger at a rate of over 100,000 log messages a
second, which is why I haven't paid performance much attention since
running the microbenchmark. Here's how to replicate it (using Python
2.6, which lets you pass a callable to the timeit.timeit function):

import logging

class NullHandler(logging.Handler):
def emit(self, record):
pass

logger = logging.getLogger('django')
logger.addHandler(NullHandler())
logger.propagate = False

time_for_a_million_messages = timeit.timeit(lambda: logger.info('a log
message'), number=100)
messages_per_second = 1 / (time_for_a_million_messages / 100)

I ran that just now and got 137,223 messages per second. I figure even
a really heavy Django page will execute maybe a couple of hundred SQL
queries, which would add about 0.0015 of logging time if no handler
was hooked up. Hence why I'm not too worried.

Is there a reasonable benchmark I can be using for this? The test
suite run time doesn't seem particularly finely grained. I know Jeremy
Dunck was talking about this a while back.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: Tutorial Refresh

[Simon Willison]

Put me down as a other +1 for a full site tutorial, especially one
that covers pip, virtualenv, unit testing and the like.

I have a suggestion for a site, too: How about the conference website
for the next DjangoCon? It's meant to be a community run conference so
the job needs to be done at some point, and building it as a community
tutorial project feels like it would be a nice fit.

A conference site is neat because it's a good example of a content
oriented site that isn't a blog, but can still take advantage of
generic views (and the admin, of course). It can be as simple or as
complicated as needed for the tutorial. We can even include API driven
features (recent tweets / photos from Flickr etc) which would be a
good way of demonstrating advanced concepts like caching.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

On Oct 9, 11:12 am, Luke Plant  wrote:
> >http://github.com/simonw/django/tree/logging
>
> Is there some easy way to see the patch?  Preferably a link which will
> just show the diff between the latest of your branch and trunk, but
> failing that, a recipe of commands using git (for those of us who
> haven't bothered to learn git properly yet).

Once things are a bit further along I'll start posting patches to the
ticket tracker, but for the moment you can see the changelog here:

http://github.com/simonw/django/commits/logging

There are only two relevant commits:

http://github.com/simonw/django/commit/b5227e1ac1d70b1f936ee69d6e347d8148df461e
http://github.com/simonw/django/commit/65ff505718538124ee2979fb60dfe1a37ca1b8bc

But since that means you have to patch together what's going on from
the two commits, I've pasted a full diff to trunk here:

http://dpaste.com/hold/104862/
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

For anyone interested in following this, I've just started a
(currently experimental) branch on GitHub:

http://github.com/simonw/django/tree/logging

So far I've added a 'django' logger which is silent by default, and
hooked up a new CursorLoggingWrapper to log all SQL statements from
the ORM to it. I've also modified the runserver command so you can use
the following to watch the execute SQL flow past on the console:

./manage.py runserver --loglevel=django.db.sql:info

None of the implementation code is intended to be production quality
yet (I'm just experimenting for the moment), but I'm very interested
in feedback on the approach.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Brand new projects should pass their tests (the django.contrib.auth thing from #7611)

[Simon Willison]

On Oct 6, 1:25 pm, Russell Keith-Magee  wrote:
> I think we need something else. In particular, I think we need to
> address the problem at a slightly deeper level - we need to
> acknowledge that we don't differentiate between application tests and
> integration tests within Django's test framework.

That's a really interesting idea - I've been thinking about it over
the past couple of days and it does seem to make sense. I'm having
trouble imagining exactly how it would work, but I can see that tests
which identify problems with the user's configuration are useful, but
should absolutely be kept separate from those tests that verify that
the application works as a standalone chunk of code.

Are we talking about something like this?

./manage.py test # run application tests
./manage.py test --integration # run integration tests

>  1. Reverse the decision of #7611, and make the current contrib.auth
> test suite a genuine collection of app tests.
>
>  2. Add extra tests to validate the integration case. These tests
> would be conditional, depending on exactly what has been deployed. For
> example: one test would look to see if
> contrib.auth.views.password_change has been deployed in the project.
> If it has, then the test would try to GET the page. The test passes if
> the page renders without throwing a missing template warning. However,
> if the view isn't deployed, the test is either not run, or is passed
> as a no-op. Essentially, the integration tests should be trying to
> catch every way that you could misconfigure your deployment of an
> application in a project.

Conditional tests sound a little bit brittle to me - I think I prefer
the model where a user explicitly requests the integration tests to be
run.

> Making this happen will require two pieces of infrastructure:
>
>  * Firstly, we need a way to make app tests completely independent of
> the project environment. We started down this path when we added
> TestCase.urls, and the patch on #7611 adds another little piece of the
> puzzle. What we really need is a way to address this consistently for
> _all_ application settings - including those provided by the
> application itself. However, it isn't as simple as creating a new
> settings object, because some settings - such as database settings -
> need to be inherited from the test environment.

This is really interesting, because it fits in with my personal
vendetta against settings.py as a whole - the single reason I dislike
settings.py is that I often find myself wanting to use different
settings for different parts of the same application. Running tests
feels like an extension of that desire (and is in many ways a more
obvious use case).

>  * Secondly, we need to make sure that we can easily establish if
> integration conditions need to be tested. reverse() already does much
> of the job here, but some helpers to make it easy wouldn't go astray.

As someone who likes to experiment with alternative ways of
constructing an application (see djng) I'm still not a big fan of
reverse, which tightly couples me to defining all of my URLs in
urls.py with references to real solid functions (rather than re-
dispatching through whatever crazy mechanism takes my fancy on that
particular day). Again, it might turn out that there's a nicer
alternative to integration tests that attempt to run conditionally on
configuration. I'm very unclear on how any of this would work though.

It sounds to me like splitting tests in to application v.s.
integration tests is going to be a lot of work - work that's worth
doing, but it's not going to be a quick fix. I'd like to find a quick
temporary fix for the auth-tests-fail-by-default problem that can at
least resolve this particular issue rather than holding out for a
large scale refactoring of the test framework. Shipping default
registration tests with the contrib.auth app feels like the easiest
option here, especially since default templates would be a useful
feature of that part of the framework anyway.

If we were to ship default templates, simply using {% extends base %}
and providing an argument to the generic views that allows the user to
specify their own base template would be enough to ensure easy
integration with a custom design (and hence work around what I assume
is the reason we didn't ship default templates in the first place).

Does that sound reasonable?

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Brand new projects should pass their tests (the django.contrib.auth thing from #7611)

[Simon Willison]

One of the things that has been established at DjangoCon is that, as a
community, we don't have a strong enough culture of testing. This is
despite Django shipping with some good testing tools (TestClient and
the like). Anything we can do to make testing more attractive is a
Good Thing.

In my opinion, one of the biggest psychological barriers to testing is
this:

$ cd /tmp
$ django-admin.py startproject foo
$ cd foo
$ echo "DATABASE_ENGINE='sqlite3';DATABASE_NAME='data.db'" >>
settings.py
$ python manage.py test
Creating test database...
Creating table auth_permission
...
EE..E...EEE..
==
ERROR: test_password_change_fails_with_invalid_old_password
(django.contrib.auth.tests.views.ChangePasswordTest)
...
TemplateDoesNotExist: registration/password_change_form.html

On a brand new project, the tests fail. This has been discussed
before:

http://groups.google.com/group/django-developers/browse_thread/thread/ac888ea8567ba466
http://code.djangoproject.com/ticket/7611

The decision was to wontfix it, since the test failures are correct -
if you don't have those templates set up, the auth application won't
work. That certainly makes sense, but the larger issue remains: a
brand new project doesn't pass its tests out of the box.

A number of potential solutions come to mind:

1. Remove django.contrib.auth from the default set of INSTALLED_APPS
2. Add django.contrib.admin to the default set of INSTALLED_APPS so
that the templates are available
3. Ship the registration/ family of templates in the contrib.auth app
4. Modify the contrib.auth tests to first alter TEMPLATE_DIRS to
ensure the templates are available
5. Remove the views from contrib.auth and move them to
contrib.authviews - which isn't installed by default
6. Something else entirely

I quite like option 1 - I don't think including contrib.auth in brand
new projects is particularly useful. We can change the documentation
to note that if you want to add contrib.admin you'll need to inclued
contrib.auth as well.

I don't like option 2.

Option 3 is a fair bit more complicated than it sounds - we would need
to ensure that those templates can still be used by the admin but with
the admin's outer design applied (probably by using {% extends base %}
and having base be a variable that is independently twiddled with
somehow).

Option 4 would break the idea that the auth tests should tell you if
the templates are misconfigured.

Option 5 would add yet another dependency to the admin.

Option 6 would be welcome if anyone has any ideas.

I think this issue is well worth solving. If we DO solve it, we could
even think about adding some stuff about running "./manage.py test" to
the tutorial.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Oct 5, 6:33 pm, Matt Brubeck <mbrub...@limpet.net> wrote:
> On Sep 24, 10:18 am, Simon Willison <si...@simonwillison.net> wrote:
> > This offers two APIs: sign/unsign and dumps/loads. sign and unsign
> > generate and append signatures to bytestrings and confirm that they
> > have not been tampered with. dumps and loads can be used to create
> > signed pickles of arbitrary Python objects.
>
> Unpickling data sent by the client seems dangerous, since it can
> execute arbitrary code on the server [1].  Obviously signing the data
> goes a long way toward preventing such attacks, but I'm still not
> comfortable with the idea that a leaked signing key could instantly be
> escalated to arbitrary code execution. (For example, if the config
> files are exposed through a misconfigured web server or accidentally
> checked into public source control.) If you use JSON or some other
> object serialization by default, then the damage from a leaked secret
> key would be much more limited in most cases.

You know what, I have to admit I hadn't really thought about JSON as
an alternative. Obviously I knew that unpickling was unsafe - that's
one of the reasons I'm so keen on signing - but it really doesn't make
sense to expose this kind of risk if it's not necessary. The pickling
trick is cute, but is the convenience of being able to pass any pickle-
able object around really worth the risk? I don't think it is - I
think I'll ditch the dumps/loads signed pickle methods in favour for
similar functionality that uses JSON instead. Other than dates being a
bit more annoying to pass around, I really don't think that telling
people they can only dumps/loads JSON-encodable data would be a huge
burden.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Branchy Django developement on GitHub

[Simon Willison]

Thanks for putting this together - that's a really useful summary of
commands. Any chance you could save it on the wiki somewhere?

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding an option to re-test only failed tests

[Simon Willison]

On Oct 2, 12:09 pm, Ned Batchelder  wrote:
> I would think the same logic applies to Django.  Nose needs to work with
> lots of different projects, so they can't own the Django details, since
> by that logic they'd also own the TurboGears logic, the Pylons logic,
> the Twisted logic, etc...  But the Nose plugin API isn't changing much
> now, so Django could include a nose plugin which would only have to
> change when Django details change.  And we'd get all sorts of nose
> features without having to implement them in the Django code.

That seems to make sense. I rather like the gesture here as well -
shipping plugins that get Django to work well with other parts of the
Python ecosystem sends a powerful message that Django wants to work
well with other parts of the Python ecosystem.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding an option to re-test only failed tests

[Simon Willison]

On Sep 30, 5:47 am, Russell Keith-Magee 
wrote:
> I'm yet to be convinced that Nose should be the default test runner
> for the simple reason that it doesn't come out of the box with Python.
> However, I agree that using Nose with Django should be as painless as
> possible. I included the TEST_RUNNER setting specifically for this
> reason.

Agreed - adding nose as a dependency for running unit tests would make
people less, not more likely to run them.

> Then there is the issue of whether Django should be responsible for
> shipping a Nose-compatible test runner, or whether that is Nose's
> responsibility. Historically, I've been of the opinion that it should
> be Nose's responsibility to ship a test runner, but I'm open to other
> opinions on this.

>From the point of view of encouraging the usage of nose, either would
work fine. I think this is fits in to the conversation at DjangoCon
about how we should go about encouraging Django users to explore the
wider Python ecosystem. The important thing is that we can have some
official (or semi-official) documentation somewhere saying "to access
enhanced test running commands, install nose and drop this string in
to your TEST_RUNNER setting." The string itself could point at
dango.something or nose.something, the important thing from my point
of view is that they don't have to grab a test runner script from
another third party, then figure out where to put it within their
project. If nose don't want to ship the test runner, I'd be fine with
putting it in django.contrib somewhere.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

So I've read through the Python Logging 101 (really useful) and the
logging docs:

http://plumberjack.blogspot.com/2009/09/python-logging-101.html
http://docs.python.org/library/logging.html

Here's my understanding of what we need to do for Django.

1. Create a 'django' logger somewhere in the framework that will
always be executed if part of Django has been imported - would the
django/__init__.py file work? Configure that with a NullHandler, and
set it NOT to propagate events to the root logger. That way we can
start using 'django.*' loggers throughout the framework but none of
the messages will be sent anywhere without further configuration. The
code should look something like this:

import logging

class NullHandler(logging.Handler):
def emit(self, record):
pass

logger = logging.getLogger('django')
logger.addHandler(NullHandler())
logger.propagate = False

NullHandler is a generally useful class (I'm surprised it's not part
of the core logging library), so we should probably put that in
django.utils.log. The NullHandler is there purely to prevent the one-
off "you haven't configured a handler yet" error that gets sent to
sys.stderr.

2. Start using loggers within the Django framework itself. The most
obvious place to start is SQL queries - at the point where we execute
SQL, we should do this:

logging.getLogger('django.db.sql').debug(sql_to_execute)

That logs the SQL to a 'django.db.sql' logger - which is a child of
the 'django' logger and hence will be swallowed by the NullHandler.

Another place we could use logging is to record cache gets and misses:

logging.getLogger('django.core.cache').debug('Cache miss for key %s',
key)

And for sending e-mail:

logging.getLogger('django.core.mail').info('Mail sent to %s',
to_email)

The logger names reflect the structure of Django up to a point
(django.db) but I don't think there's anything wrong with straying
from that convention (django.db.sql for example) provided the prefixes
are sensible.

3. Add a bunch of syntactic sugar for making log messages visible at
various points within Django. For example:

./manage.py runserver --log-level=django:info

Would cause the runserver to first configure the 'django' logger to
print all messages of level 'info' and higher to stdout.

./manage.py runserver --log-level=django:info --log-
level=django.db:debug

Same as above, but also configures django.db messages of debug or
higher to display (I'm sure the command line syntax could be
improved).

By default, runserver would display no log messages at all.

The same arguments could also work for ./manage.py test:

./manage.py test --log-level=django.db.sql:debug

Hmm... maybe the --log-level argument could work for ALL management
commands.

My philosophy here is that log messages should be ignored unless
explicitly told otherwise. Django gets run in various different ways -
runserver, test and in deployment under mod_wsgi/FastCGI/etc - and
it's not at all obvious what the default log output behaviour should
be. As long as we make it extremely easy to see log messages if we
want them I think having them off by default makes sense. It also
ensures no surprises for people upgrading from 1.1 to 1.2.

4. Add a way to configure loggers in Django's settings.py file, in
particular for use in production. I'm not sure if these settings
should be ignored when running under other circumstances (./manage.py
test etc) in favour of the command line option, or if they should
still take effect. I quite liked Ivan's suggestion:

LOGGING = {
 'django.db.sql': {
 'handler': 'django.logging.FileLogger', # WatchedFileLogger
copy
 'filename': '/tmp/django-sql.log',
 'level': 'debug',
 },
 'django.core.cache': {
 'handler': 'logging.handlers.HTTPHandler',
 'host': '...',
 'url': '',
 'format': ''
 },
 'django.errors': {
 'handler': 'logging.handlers.SMTPHandler',
 ...
 }
}

django.errors is where we get to replace the current "e-mail 500
errors to an ADMIN" functionality.

There should also be an option for advanced users to ignore the
declarative syntax entirely and just provide a bunch of code that
configures logging in the traditional way:

import logging
django_logger = logging.getLogger('django')
django_logger.setLevel(...)
django_logger.addHandler(...) # etc

I'm not sure where that custom code should live - right there in
settings.py? At any rate, I think it's vital that people can use the
low level logging configuration API if they want to.

I'd rather not encourage people to use the logging module's support
for .ini style configuration, just because Django already has its own
configuration standards. That's not to say people can't call that from
their own code if they want to, but I don't think we should emphasise
that ability.

5. Document how users should go about implementing their own loggers.
I think we should tell people NOT to put them within the 'django'
logger 

Re: Adding an option to re-test only failed tests

[Simon Willison]

On Sep 29, 7:34 pm, Rob Madole  wrote:
> TEST_RUNNER = 'django.contrib.test.nose.run_tests'
>
> There might be some futzy bits to make that actually work, but I think
> it'd doable.

I'd love to see this working. Obviously this would work just as well
implemented as an external project - but if it's as useful as it
sounds I'd personally be up for including it in core, if only to
promote the usage of nose amongst Django developers (and hence help
weaken the impression that Django doesn't integrate well enough with
the wider Python community).

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding an option to re-test only failed tests

[Simon Willison]

On Sep 29, 5:03 pm, Rob Madole  wrote:
> I've been using nose for our tests, and one of the features that I
> really like is the ability to run the tests again but filter only the
> ones that caused a problem.
>
> I'm thinking it would look something like this
>
> ./manage.py test --failed
>
> Does this sound worthwhile to anybody?

I don't understand how this works - does it persist some indication of
which tests failed somewhere? If so, where?

If we're talking about features from nose, the two I'd really like in
Django's test runner are --pdb and --pdb-failures:

--pdb = when an error occurs, drop straight in to the interactive
debugger
--pdb-failures = when a test assertion fails, drop in to the debugger

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Model.objects.raw() (#11863)

[Simon Willison]

On Sep 29, 2:24 am, SeanOC  wrote:
> During the Djangocon sprints I started to work on a patch which would
> add a nicer interface for dealing with raw SQL queries.  While there I
> talked to RKM about where it should fit into the ORM API and where in
> the code base should live.  I've finally gotten around to finishing
> the code I wrote during the sprint and have posted a patch to the
> related ticket (http://code.djangoproject.com/ticket/11863).  You can
> also get the code fromhttp://github.com/SeanOC/django.

I've had a bit of a think about this, and there are a couple of things
I'd be very keen on seeing supported by this feature. I have a nasty
feeling they won't be that straight forward though.

The first is that I'd like it to be compatible with deferred loading
of model attributes - I haven't looked in to it, but my hunch is that
this won't be too hard (it might even work in its present form without
any further changes).

The second one is probably a lot harder - it feels to me like this
feature would be much more useful if it could perform select_related
style graph population. One of the key use-cases for raw SQL is
improving performance - writing queries which perform better than the
ORM (SQL optimisations, or queries that hit an alternative view or use
a stored procedure or similar). It's very plausible that a developer
might want to write a custom query that populates a full graph of
objects, just like select_related() does. It would be really useful if
the QuerySet.raw() method let them do that.

I haven't looked at how much work it would be to support graph
population - it will certainly complicate the user-facing API, and it
might require a bunch of work at the ORM level. If it's just too much
hassle I think the feature would be worth including without it, but
it's certainly worth some further investigation.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

On Sep 29, 9:36 am, Vinay Sajip  wrote:
> Simon, I'm the author of Python's logging package. Sorry for the delay
> in replying, I've been away from this list awhile. I think the "Java
> heritage shines through" is just FUD.

Hi Vinjay,

Thanks for dropping by. Firstly, I'm sorry for disparaging of your
work. I mis-spoke when I said the logging module wasn't "nicely
designed" - I obviously don't believe that, since I'm very keen on
including it in Django. I should have picked my words much more
carefully.

The point I intended to make is that using the logging module
correctly requires quite a lot of knowledge on the part of the
developer - there's a lot to understand in there. When I talk about a
"thin wrapper" for Django really I'm really just talking about smart
defaults - ensuring that Django users can start using logging without
needing to know anything more than "import this and call the log.error
() method", but that the full power of the logging library is still
available once they need it.

So again, I apologise for my extremely poor choice of words - that was
decidedly unclassy. I'm excited that you're interested in helping us
integrate logging with Django in the best possible way.

Thanks,

Simon

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Model.objects.raw() (#11863)

[Simon Willison]

On Sep 29, 2:24 am, SeanOC  wrote:
> During the Djangocon sprints I started to work on a patch which would
> add a nicer interface for dealing with raw SQL queries.  While there I
> talked to RKM about where it should fit into the ORM API and where in
> the code base should live.  I've finally gotten around to finishing
> the code I wrote during the sprint and have posted a patch to the
> related ticket (http://code.djangoproject.com/ticket/11863).  You can
> also get the code fromhttp://github.com/SeanOC/django.

I'm a big fan of adding this feature - I firmly believe we should be
encouraging people to roll their own SQL where necessary (it's the
ultimate answer to complaints about missing features in the ORM), and
I don't think just telling them to instantiate their own cursor is a
good enough answer.

It's worth talking to Jacob about this - last year he had something
like this in a private branch, but I don't think he ever released it.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: Better HTML4 support

[Simon Willison]

On Sep 28, 6:57 pm, Rob Hudson  wrote:
> For lack of knowing about anything better, I keep falling back to
> Werkzeug's HTMLBuilder class[1].  Pulled out and stripped of comments,
> it weighs in at 77 lines of code...

That's not so bad. I was worried about pulling in a large dependency,
but 77 lines (probably dropped in to django.utils.builder or similar)
isn't particularly alarming. I'm still a bit sceptical of introducing
another markup abstraction just to solve this one problem. I wonder if
it could be used to tackle the RSS/Atom generation problem as well?

> I agree with what I think I'm reading here -- a goal being to give
> designers more fine grained control over the form and form elements at
> the template level.

Yes, that's exactly it. Designers should be able to do anything they
like with forms at the template level (hence the {% field form.name
class="foo" %} suggestion). Again, this makes me slightly cautious
about the HTMLBuilder approach since exposing that at the template
level is virtually impossible.

> I think you might want both 1 and 3.  (1) for those that want finer
> control or just don't want to use the underlying HTML wrapper, and (3)
> for those that do.
>
> Would it be something to consider adding special case tag, like
> comments, to represent the self closing slash depending on current
> context's doctype?  For example, something like {% / %}?

I'm really not in favour of 3. 1 I think is OK - as I mentioned above,
I added it to django-html as {% slash %} and it's actually not that
unpleasant. I think I'd rather avoid adding custom syntax like {% / %}
just to support this one tiny edge case (I strongly expect most Django
developers won't ever want to write code that's doctype agnostic - and
as Max mentioned, HTML5 grandfathers in the /> syntax - the tag will
be used strictly by perfectionists).

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: Better HTML4 support

[Simon Willison]

On Sep 26, 7:33 pm, Simon Willison <si...@simonwillison.net> wrote:
> 1. a {% selfclose %} template tag:
>
>     <br{% selfclose %}>
>
> {% selfclose %} outputs either blank or " /" depending on the doctype.

I've added an experimental {% slash %} tag to django-html to address
the reusable XHTML/HTML templates problem. It's actually not as
horrible as I first thought - you end up writing template code like
this:

<br{% slash %}>



It's not exactly beautiful, but it's the simplest thing that could
possibly work. Most Django developers who aren't writing code designed
to be used with differing doctypes won't ever have to use it. In the
absence of a better idea I'm pretty happy with it.

http://github.com/simonw/django-html

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: Better HTML4 support

[Simon Willison]

On Sep 28, 5:20 pm, Max Battcher  wrote:
> I really don't see what the fuss here is about. If we are worried about
> forwards-compatibility, HTML 5 takes care of it. If we are worried about
> better backwards-compatibility with HTML 4, everyone else is saying that
> the future is now and the focus should be HTML 5...
>
> What is this argument really about?

http://www.djangoproject.com/ calls Django the web framework for
"perfectionists with deadlines". This is a perfectionist issue.

If the problem was incredibly hard to solve or involved breaking
backwards compatibility I'd drop this, but I don't think it's a
particularly big or difficult change. The django-html approach even
gives us a useful extra feature - it allows template developers to add
new attributes to form widgets without needing changes made to the
underlying Python form definitions:

{% field form.name class="foo" onfocus="bar()" %}

It's not just me that gets annoyed by this - when I'm teaching Django
to client-side engineers this tends to come up a lot - and I find the
answer a bit embarrassing. It's basically the only place in Django
that the template author can't control the markup, and good client-
side engineers are pretty picky.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: Better HTML4 support

[Simon Willison]

On Sep 27, 10:49 am, veena  wrote:
> my bits to discussion about supporting various (X)HTML versions.
>
> 1) Problem with (X)HTML in python code (in applications)
> I discovered this python packagehttp://pypi.python.org/pypi/html/1.6
> It allows you to write "python like HTML syntax" and generates HTML or
> XHTML.

Something like that might be an option for cleaning up the way markup
is generated within the forms framework itself, but ultimately I think
adding an entirely new Python-based generation method just to output a
few strings wouldn't be worth the added dependency.

> 2) Problem with (X)HTML in templates
> I think there should be parser, which parse template just before
> caching templates. Code could be messy HTML or (XHTML) or invalid HTML
> (like undisclosed tags, attributes without quotations marks etc.) and
> from this make pretty HTML or XHTML according to html coder settings
> by {% doctype %}

The performance overhead rules out this kind of approach for Django
core - any post-processing run against the output of the templates
would be executed on every single Django request, and HTML parsing and
rewriting is a very expensive operation. That's not to say a third
party module couldn't be written to do this kind of thing for people
who really want to clean up their output (I remember there being an
HTMLTidy middleware a few years back that did this) but it wouldn't be
suitable for inclusion in core.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: CSRF - next step?

[Simon Willison]

On Sep 26, 7:44 pm, Alex Gaynor  wrote:
> So I'm still a little unclear on what this shortcut does that
> direct_to_template doesn't already?

It's a bit less weird. direct_to_template lives somewhere I can't
remember (so I rarely import it), has an argument called
"extra_context" that's actually just the regular context, sets up a
surprising 'params' key in the context dictionary that contains the
function's **kwargs, executes any of the context arguments that are
callables before passing them to the template (even though the
template language usually does that anyway) and has an argument called
mimetype when as-of Django 1.0 content_type is the preferred name for
that parameter [1]. direct_to_template is also an odd name - it makes
sense for a generic view, but it feels strange when you're using it to
render a template from within your view code.

Personally, I want a shortcut called django.shortcuts.render which
looks like this:

def render(request, template, context=None,
content_type=DEFAULT_CONTENT_TYPE):
return render_to_response(template, context or {},
context_instance=RequestContext(request),
mimetype=content_type
)

Much less surprising than direct_to_template, and much easier to
import. I use this in basically every single one of my view functions
so I'd prefer a name that is concise over one that is more
descriptive. That's assuming I can't get any buy-in for my
TemplateResponse concept[2].

Cheers,

Simon

[1] (On further inspection it looks like my attempt to get mimetype
replaced by content_type throughout Django 1.0 was something of an
abject failure, since mimetype is used by both render_to_response and
direct_to_template)
[2] http://github.com/simonw/django-openid/blob/master/django_openid/response.py
- I really like the idea of custom HttpResponse classes that are
lazily evaluated just like QuerySets are, but I've got enough stuff to
evangelise for 1.2 already ;)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: Better HTML4 support

[Simon Willison]

On Sep 26, 10:17 am, Max Battcher  wrote:
> Furthermore, support for XHTML "5" (which is indeed a part of the HTML 5
> standard) shows that XHTML 1's principles are still around and still
> respected. Django's XHTML output can't be "out of date" if XHTML 5 is
> considered a successor to XHTML 1.

Opinions differ, but in the interest of keeping this thread about HTML
support in Django I'm going to leave that discussion here - let's
assume that it would be a Good Thing for Django should provide support
for outputting both HTML and XHTML, and focus on how we can achieve
that.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: Better HTML4 support

[Simon Willison]

On Sep 26, 5:48 am, Rob Hudson  wrote:
> First, let's not let this turn into an argument about HTML vs XHTML.

Oops, sorry!

> People have their preference one way or the other, and I believe
> Django should be agnostic.

Completely agree - that's why I'm in favour of a Django doctype switch
rather than a fullscale switch to HTML 5.

> django-html Shortcomings:
> * Fixes the symptom, not the bug. [1]  I think the fix should not do
> string replacement, but output the correct HTML in the first place.
> (I realize this is the best one can hope for as a 3rd party app, but
> if something were to come into core it would have the ability to fix
> things properly.)

Yes - I looked briefly at how much work was involved in doing this and
it's not insubstantial, which is why I opted for string replacement
just to demonstrate the API. I'm confident the exact functionality of
django-html could be replicated in a less messy way by patching core,
but it would require quite a bit of refactoring of django.forms.

> * Doesn't "fix" the rest of the areas where Django outputs XHTML.  Is
> it possible?

I think the key problem is template filters, which often produce XHTML
(linebreaksbr for example). This could be solved by allowing template
filters selective access to the template context, which I'm in favour
of in the absence of a compelling argument against. This could be done
in a way that allows existing filters to continue to work - maybe
something as simple as this:

register.filter('cut', cut, takes_context=True)

Which emulates the register.inclusion_tag API.

> 2. Add a new setting, e.g. settings.DOCTYPE, that various parts of
> Django and 3rd party apps use to render HTML.
>
> Advantages:
> * Simple and straightforward
>
> Shortcomings:
> * Yet another setting
> * Doesn't allow for template level decisions of doctype by designers

Really not keen on that - designers should be able to pick their
doctype, and there are some cases where specific pages of a site (or
of a reusable app) might need to use a specific doctype - MathML
output still requires XHTML for example.

I've somewhat foolishly volunteered for a bunch of 1.2 related hacking
already (CSRF, logging, signing) but I remain actively interested in
this, so I'll try to keep an eye on things and see if there's anything
I can contribute.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: Better HTML4 support

[Simon Willison]

On Sep 26, 6:50 am, Jerome Leclanche  wrote:
> This is a non-issue.



Obviously I disagree - this is a tiny thing that has bugged me ever
since newforms. It's also something I find myself constantly
apologising to front-end developers about, who for the most part love
the Django template language.

After 9 years In the marketplace, I think XHTML has pretty much lost.
It turns out that XML serialisations of HTML aren't nearly as useful
as people thought they would be (running an XML parser against the web
won't get you anywhere fast, and there are plenty of excellent HTML
parsing libraries now), and strict XML error handling is completely
unsuitable for the practical considerations of the web. HTML 5 is the
final nail in the coffin - the refocusing of the W3C on that over
XHTML 2 is an acknowledgement that XML is no longer the future of the
Web. I actually think Django's XHTML output makes us look a bit out of
date.

> Correct me if I'm wrong, but  is valid html syntax. It's parsed
> as valid by every html parser, and I'm positive this is the entire
> point of xhtml: staying backwards-compatible with html.

If you want backwards compatibility you need HTML 5, not XHTML.

HTML 5 is backwards compatible - it even goes as far as specifying the
error handling routines that browsers should take when faced with
invalid code (routines that have been reverse-engineered from existing
browsers) and properly specifying all of the APIs that browsers
support but were never actually part of a specification, such as
XMLHttpRequest and innerHTML.

XHTML 1.0 is kind-of backwards compatible, but it's a bit of a fudge.
It's really meant as a transitional stage between HTML 4 and the
stricter XHTML 1.1. As an example of a fudge, the  self-closing
tags actually mean something different in SGML (HTML's parent
language) - they're a short-hand feature called Null End Tags which
should technically result in extra > signs appearing in the text, but
thankfully no browser actually supports enough of SGML for that to be
a problem.

The moment you start serving XHTML with the correct content-type
(application/xml+html) all sorts of nasty things happen - in addition
to strict error handling the way the JavaScript DOM works
fundamentally changes - document.getElementsByTagName becomes
getElementsByTagNameNS for example.

XHTML 1.1 backwards incompatible - the spec forbids the use of the
text/html content-type and insists on application/xhtml+xml or
application/xml, which breaks completely in IE and causes other
browsers to use strict error handling (which you really, really don't
want).

XHTML 2.0 was completely backwards incompatible (tags were removed,
new incompatible tags added), a major factor in the W3C ending
official work on it.

> That said, I'm not against a {% doctype %} tag or anything. However, I
> do think having an xhtml default is a Good Thing due to its backwards
> compatibility.

I see XHTML as a dead end, so I'd personally prefer to switch to an
HTML 5 default - but the {% doctype %} thing makes switching to HTML 5
painless enough that I wouldn't mind the existing XHTML behaviour
remaining as the default (purely to avoid causing issues with existing
applications).

> I'm also not entirely fond of the idea of having 100
> different language outputs in Django. Why stop only after xhtml and
> html?

Because Django is a Web framework, and XHTML and HTML are the key web
languages. I don't think we need to worry about support for those two
leading to a slippery slope.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Sep 25, 4:44 pm, Johannes Dollinger
 wrote:
> Regarding parity, let me advertise a Signer object again:
>
>         signer = signed.Signer(
>                 key=settings.SECRET_KEY,
>                 expire_after=3600
>         )
>
>         s = signer.sign(v)
>         v = signer.unsign(s)
>
>         signer.set_cookie(response, name, value)
>         signer.get_cookie(request, name)
>         # or:
>         signer.get_cookies(request)[name]
>
>         # transparently sign cookies set via response.set_cookie()
>         # and unsign cookies in request.COOKIES in place:      
>         @signer.sign_cookies
>         def view0(request):
>                 ...    
>         @signer.sign_cookies('cookie', 'names')
>         def view1(request):
>                 ...
>
> This would make more options and customization feasible (by  
> subclassing):

OK, you got me. You obviously know my weakness for customisation by
subclassing :) Having a Signer class is a smart idea, I'll kick it
around a bit and see how it looks.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Sep 25, 3:39 pm, Simon Willison <si...@simonwillison.net> wrote:
> While that makes sense for caching, I couldn't say if it makes sense
> for signatures or not - when we sign something, will we always know
> the point at which we want that signature to expire? I don't know.

Here's a good argument for signing things with the creation-timestamp
rather than the expiration-timestamp - it leaves the door open for a
mechanism whereby historic SECRET_KEYs are stored. When we see a
signed string, we can use its timestamp to decide which of our
historic keys should be used to validate it.

BIt of an edge case (I can't say if we'd ever want to do this) but
it's an example of something that's not possible with expire-at
timestamps.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Sep 25, 3:10 pm, Marty Alchin  wrote:
> The timestamp would then be the actual expiration
> time, rather than the time it was signed, so the API can look like
> this instead (with a key added per prior discussion).
>
> >>> s = signed.sign('key', 'value')
> >>> v = signed.unsign('key', s)
> >>> s = signed.sign('key', 'value', expire_after=24 * 60 * 60)
> >>> v = signed.unsign('key', s)

Baking an expiration time in to a signed value and checking for it
when we unsign is definitely possible, but I'm a bit torn between the
two options. I prefer setting the expiry time when I read the cookie
rather than when I set it, but I can't say exactly why. I think it's
based on bad experiences with memcached - I've frequently found a need
to set things in the cache for ever and decide whether or not to treat
them as stale on retrieval (for implementing things like serve-stale-
on-error) which has made my instinct be to bake the time-of-creation
in to the thing and leave the is-this-stale decision until as late as
possible.

While that makes sense for caching, I couldn't say if it makes sense
for signatures or not - when we sign something, will we always know
the point at which we want that signature to expire? I don't know.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Sep 25, 1:57 pm, Marty Alchin  wrote:
> I think this is good for everywhere the raw signing API is accessed,
> perhaps to the point where that API can't even be used without a key
> (a namespace, really - honking great idea!). Helpers on top of that
> API could do without asking for that separately, if they can retrieve
> a reasonable key from other forms of input, such as a cookie name or a
> query-string argument name.

I rather like that idea.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Sep 25, 1:57 pm, Marty Alchin  wrote:
> I wish there was
> a way to sign the expiration as well, so people couldn't artificially
> extend the life of the cookie, but since that doesn't come back in the
> request, there'd be no way to validate it.

We can do that by baking the timestamp the cookie was set in to the
signed cookie value, than doing our own check against that and
discarding the cookie if it's expired. This pattern (signatures that
expire) is common enough that I think it would be worth supporting in
the low level django.utils.signed module - I've used timestamped
signatures for things like "recover your account" links that expire 24
hours after being requested.

The API would look something like this:

>>> s = signed.sign('value', timestamp=True)
>>> v = signed.unsign(s, expire_after=24 * 60 * 60)

A SignatureExpired exception would be raised if the signature was
older than the expire_after argument (SignatureExpired would subclass
BadSignature)

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Sep 25, 12:02 pm, Luke Plant  wrote:
> In other web apps (I think Wordpress?), there have been problems
> associated with use of secret keys when the same key is used for
> different purposes throughout the application.
> ...
>  - we add unique prefixes to the SECRET_KEY for every different
>    place it is used.  So for the e-mail confirmation link, we use
>    HMAC("email-confirmation" + SECRET_KEY, message)
>  - also add the ability to do SECRET_KEY rotation, as Simon
>    suggested.  This suggests we want a utility wrapper around hmac
>    that looks like hmac(unique_key_prefix, key, message) and handles
>    all the above details for us.

I share your concern. In the signed.py module I address your first
point by allowing an additional "extra_key" parameter:

>>> signed.dumps('hello', extra_key='ultra')
'UydoZWxsbycKcDAKLg.1XYDpILo5xqSwImfa3WuJJT4RPo'
>>> signed.loads(_, extra_key='ultra')
'hello'

However, this is available on dumps and loads but not on sign and
unsign - we should definitely implement it at those points instead.

I'm going to ask on some web security mailing list about best practice
for rotating keys. Do you have any further information on the
WordPress problems?

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Sep 25, 4:48 am, Russell Keith-Magee 
wrote:
> By way of greasing the wheels towards trunk: if the outcome of this
> mailing list thread was a wiki page that digested all the ideas,
> concerns and issues into a single page, it will make the final
> approval process much easier.

Great idea: http://code.djangoproject.com/wiki/Signing

I've attempted to summarise the discussion so far. I've also added a
couple of notes about potential improvements to the proposed
django.utils.signed API, in particular a note that having low-level
support for signatures that have a timestamp baked in and hence can
expire might be a good idea.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Sep 24, 8:22 pm, Benjamin Slavin  wrote:
> Unfortunately, this approach won't work.
>
> A malicious client can just send "key" rather than "key__Xsigned" and
> you'll never know that the cookie hasn't gone through validation.
> This also means that you can't look for cookie values that match a
> specific format (ex: body.signature) because a malicious client could
> just drop the signature portion.
>
> As always, we can't trust the client. :-(

Good point - that rules that approach out.

> 1) request.unsign_cookie('foo') -- This breaks the parallelism with
> existing cookies.  Sometimes we'll be doing request.COOKIES['foo'] and
> sometimes we'll be doing request.unsign_cookie('foo').

If we were going to do that, it would make sense to NOT have set_cookie
(... sign=True) as the API for setting one. We could achieve
parallelism with something like this:

response.sign_cookie('key', 'value')
...
value = request.unsign_cookie('key')

You can still read request.COOKIES directly, but you'll get the raw,
signed value. That API doesn't look too ugly to me.

> 2) A decorator for views -- @unsign_cookies("foo", "bar") -- This
> doesn't allow any sort of fall-back (you can't customize what to do if
> a given cookie is improperly signed)

If a cookie is improperly signed I think you silently discard it, as
if it was never set. If we had logging this could always be logged as
well... we could fire a signal if we really think people might want to
further customise it.

> 3) COOKIES as an intelligent object -- We can overload .get so we're
> doing something like request.COOKIES.get('foo', signed=True) -- I
> think this has the best chance at an interface that keeps a consistent
> feel. It's completely backward compatible, though it breaks the
> traditional expectation of what you can do via the `get` method on a
> dictionary.

This isn't so bad, since we already have a precedent for this in
request.POST.get_list('foo'). request.COOKIES.get_signed(key) might be
OK.

At the moment I think my preference is for response.sign_cookie and
request.unsign_cookie, though I'm a bit worried that "unsign cookie"
doesn't obviously mean "get the cookie, check if the signature is OK
and return the value if it is". I like that unsign_cookie maps to the
low level signed.unsign API, but it might well be that most users
never use the low level signing API and the cookie stuff is the only
bit of it they ever see.

Cheers,

Simon

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Simon Willison]

On Sep 24, 7:37 pm, Marty Alchin  wrote:
> Another option would be to have request.COOKIES be a custom
> dictionary, with an extra .get_unsigned(key) method that would work
> like .get(), but validates the signature along the way. That behavior
> can't be added straight to __getitem__() though, because that has no
> way of knowing whether the cookie was signed to begin with.

Hmm... I hadn't considered that. I was thinking that the unsigning
could be transparent, so by the time you access request.COOKIES['key']
the value had already been unsigned (and if the signature failed the
cookie key just wouldn't be set at all, as if the cookie never
existed). But as you point out, this doesn't work because you can't
tell if the cookie was signed or not in the first place.

We could fix this with a naming convention of some sort:

response.set_cookie('key', 'value', sign=True)
- results in a Set-Cookie: key__Xsigned=value header

But that's pretty ugly. Not sure what to do about this one -
request.unsign_cookie('key') might be an option, as at least that
reflects the set_cookie / sign / unsign API a bit. Not ideal by a long
shot though.

> eyes. Yeah, I suppose we could perhaps make that a subclass of
> ValueError or TypeError, but I would worry about people wrapping it up
> into something else that might cause problems.
>
> try:
>     value = signed.unsign(signed_value).decode('utf-8')
> except ValueError:
>     # Whoops! UnicodeDecodeError winds up here as well!

That's a great argument against subclassing ValueError - I hadn't
considered the unicode case. You're right, if anything it should
subclass SuspiciousOperation instead.

Cheers,

Simon

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Adding signing (and signed cookies) to Django core

[Simon Willison]

As mentioned in the thread about cookie-based notifications, at the
DjangoCon Sprints I raised the subject of adding signing (and signed
cookies) to Django core.

I've found myself using signing more and more over time, and I think
it's a concept which is common enough to deserve inclusion in Django -
if anything, its use should be actively encouraged by the framework.

It's also something that's hard to do correctly. At the sprints Armin
pointed out that I should be using hmac, not straight sha1, for
generating signatures (something Django itself gets wrong in the few
places that implement signing already). Having a cryptographer-
approved implementation will save a lot of people from making the same
mistakes.

Signed cookies
==

On top of signing (which I imagine would live in django.utils) I'd
like to add a signed cookie implementation. Signed cookies are useful
for all sorts of things - most importantly, they can be used in place
of sessions in many places, which improves performance (and overall
scalability) by removing the need to access a persistent session
backend on every hit. Set the user's username in a signed cookie and
you can display "Logged in as X" messages on every page without any
persistence layer calls at all.

I think signed cookies should either be a separate API from
response.set_cookie or should be provided as an additional argument to
that method. I'm not a fan of signing using middleware (as seen in
http://code.google.com/p/django-signedcookies/ ) since that approach
signs everything - some cookies, such as those used by Google
Analytics, need to remain unsigned.

So the API could either be:

response.set_signed_cookie(key, value)

Or...

response.set_cookie(key, value, signed=True)

(I prefer the latter option)

Proposed signing implementation
===

I'd be happy to donate my signing code from django-openid to the
cause, which was written to be usable entirely separately from the
rest of the django-openid codebase:

http://github.com/simonw/django-openid/blob/master/django_openid/signed.py
http://github.com/simonw/django-openid/blob/master/django_openid/tests/signing_tests.py

This offers two APIs: sign/unsign and dumps/loads. sign and unsign
generate and append signatures to bytestrings and confirm that they
have not been tampered with. dumps and loads can be used to create
signed pickles of arbitrary Python objects.

Here's what the API would look like with this library:

>>> from django.utils import signed
>>> signed.sign('hello')
'hello.9asVJn9dfv6qLJ_BYObzF7mmH8c'

The signature is a URL-safe base64 encoded digest of the hmac/sha1. I
used base64 rather than .hexdigest() for space reasons - base64
digests are 27 characters, hexadecimal digests are 40. When you're
including signatures in cookies and URLs (especially account recovery
URLs sent out in plain text, 80 character wide e-mails) every byte
counts.

>>> signed.unsign('hello.9asVJn9dfv6qLJ_BYObzF7mmH8c')
'hello'
>>> signed.unsign('hello.badsignature')
Traceback (most recent call last):
...
BadSignature: Signature failed: badsignature

BadSignature is a subclass of ValueError, meaning lazy developers
(like myself) can do the following rather than importing the exception
itself:

try:
value = signed.unsign(signed_value)
except ValueError:
return tamper_error_view(request)

>>> signed.dumps({"a": "foo"})
'KGRwMApTJ2EnCnAxClMnZm9vJwpwMgpzLg.mYepoYkzWwXRmsCTVJm3Mb0HHz4'
>>> signed.loads(_)
{'a': 'foo'}

Again, the pickle is URL-safe base64 encoded to take up less valuable
cookie space and generally make it easier to pass around on the Web. A
nice thing about URL-safe base64 is that it uses 64 out of the 65 URL-
safe characters (by URL-safe I mean characters that are left unchanged
by Python's urllib.urlencode function) - the remaining character is
the period, which I use to separate the pickle from the signature.

signed.dumps takes a couple of extra optional arguments. The first is
compress=True (default is False) which zlib compresses the pickle if
doing so will save any space:

>>> import this # to get an object worth compressing
...
>>> len(signed.dumps(this.s))
1207
>>> len(signed.dumps(this.s, compress=True))
637

By default, all signatures use Django's SECRET_KEY. If you want to
sign with a different key, you can pass it as an argument to the
various functions:

>>> signed.sign('hello', key='sekrit')
'hello.o6MKehoOfZ2b2FU84wzibW6IWxI'
>>> signed.unsign(_, key='sekrit')
'hello'

The dumps and loads methods also take a key argument, as well as an
additional optional extra_key argument for if you want to generate
different signatures for different parts of your application (useful
for the extra paranoid):

>>> signed.dumps('hello', extra_key='ultra')
'UydoZWxsbycKcDAKLg.1XYDpILo5xqSwImfa3WuJJT4RPo'
>>> signed.loads(_, extra_key='ultra')
'hello'

We'd want to get a proper cryptographer to give this the once-over
before adding it to core, but I'm generally happy with 

Re: CSRF proposal and patch ready for review

[Simon Willison]

On Sep 22, 9:24 pm, Luke Plant  wrote:
> > 2. I'm not at all keen on the implementation as a middleware
> > (especially a view middleware, which doesn't play well with generic
> > views and redispatching to other view functions, both patterns I like
> > a lot).
>
> Could you explain a bit more about the difficulties with generic views?  

Sorry, I meant to type "class based generic views" - but actually
that's not really what I meant either. My problem with view middleware
is that it makes it much harder to use the pattern where view
functions re-dispatch to other view functions. The classic example is
a site with a URL structure that looks like this:

/category/sub-category/sub-sub-category/arbitrarily-deep-nesting/news/

I've built stuff like this in the past by essentially discarding
urls.py - I instead map "^(.*)$" to my own view function which then
splits request.path on slashes and re-dispatches based on its own
custom logic. View middleware doesn't really fit that model - it
assumes that the top level URL configuration will resolve to the
callable that's actually going to implement the view, but that isn't
necessarily the case for complex sites.

A more common example is view functions which re-dispatch to other
functions:

def homepage_view(request):
if request.user.is_authenticated():
return logged_in_homepage_view(request)
else:
return default_homepage_view(request)

A view middleware will just see "homepage_view" - it won't know which
of the actual view functions is going to be called.

I'm a big fan of class-based generic views which can involve quite a
bit of subclassing and re-dispatching to other methods (like my
RestView class http://www.djangosnippets.org/snippets/1071/ ) which is
another reason view middleware makes me uncomfortable.

Note that this is a philosophical objection to process_view middleware
as a whole, not anything specific to CSRF.


> 1) Implement your csrf_protect_form method.  That could easily add your
> requirement to lock down to individual forms and timeouts.  It would need
> cooperation from a new template tag, or additional optional arguments to the
> current template tag. It might also need an additional context processor in
> settings, but as an opt-in solution that's OK.
>
> I think the solution that manipulates request.POST sounds OK - yes a hack, but
> providing this method is not the default, most people won't have to deal with
> any problems with it.

Excellent. I'll have a go at building that against your branch and see
how it comes out.

> The only question mark in my mind is what happens with multiple forms on a
> page (e.g. when you have a login box on every page).  It might not be an issue
> - the target of the login box will be another view anyway - but it needs
> thought.

I think that case works OK. It would end up looking something like
this:


  
  
  {% csrf_token "login" %}



  
  {% csrf_token "rate" %}


Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: CSRF proposal and patch ready for review

[Simon Willison]

On Sep 19, 4:56 pm, Russell Keith-Magee 
wrote:
> End users should be allowed to be as lazy as they like, but
> their laziness shouldn't open security holes in an app that Django
> ships, since the contrib apps (and admin in particular) are the
> obvious first port of call for any systematic CSRF attack.
>
> So - let's have both. A middleware enabled by default protects the
> rest of the views. However, we can also have a view decorator lets us
> protect admin (and other contrib apps) explicitly.

I'm a big +1 for that.

>  * The SafeForm method of reporting errors as part of form validation
> is much nicer than the 403 page, IMHO. However, I'm not sure what we
> can do to provide form-level CSRF error handling without introducing a
> whole bunch of other fragility.

I've been scratching my head over this one - inline error messages and
(in particular) not using a form submission are the single thing that
makes me favour SafeForm over the template tag approach for my own
stuff. That's why I designed SafeForm to complement the forms
framework - correctly handling CSRF feels like a form validation
problem to me, and reusing Django's redisplay-the-form-with-errors
flow is a really neat thing to be able to do.

The main reason I really like preserving form data is that it means
CSRF failures are less of a problem, allowing us to be much more
strict about how they work (setting form tokens to expire after a few
hours, tying tokens to individual forms etc). This means we can reduce
the damage caused in the case of a token leaking somehow. This becomes
particularly important if tokens are going to be used to protect GET
requests, which some applications may want to do (Flickr have CSRF-
protected GETs for their logout and change-language links, for
example) - GET tokens are more likely to leak in other people's
referrer logs or to intermediate proxies.

There is a way we could achieve form validation, but it isn't
particularly pretty. Say the CSRF check is performed by a decorator
(which has an accompanying middleware for applying that decorator to
the entire site). The problem we have is that in the case of a
failure, the fact that the check failed needs to be communicated
through to the form /inside/ that view. The form doesn't have access
to the request, so we're stuck.

But... the form does have access to request.POST. The dirty solution
would be for the decorator to rewrite request.POST to include a new
key:

request.POST.mutable = True
request.POST['__validation_must_fail'] = True
request.POST.mutable = False

django.forms.Form.is_valid() could then be hard-coded to always fail
if '__validation_must_fail' is present in the data dictionary. I've
avoided including the string "csrf" in the form key because this
feature feels like it could be useful for other things as well (I
might use it for testing out my form error styles). I don't think it's
a security vulnerability - if someone fakes a "__validation_must_fail"
key in their own submission I don't see that there's anything bad they
can do.

It's a pretty monstrous hack, but it would allow a CSRF protecting
decorator to inform a form inside a view that the check had failed and
the form should not validate.

I wouldn't suggest this as the default behaviour for the overall CSRF
protection scheme - it would leave hand-rolled forms unprotected, and
I imagine there are new error cases it would introduce. That said, it
would placate users like myself who want the nicer invalidation
messages and are willing to do a bit of work to achieve them. Maybe
something like this:

@csrf_protect
def view(request):
"""A view that fails the CSRF check with a full page 403,
as seen in Luke's current middleware"""

@csrf_protect_form
def view(request):
"""A view that won't 403 directly, but will ensure that any
calls to form.is_valid() fail due to the presence of a
__validation_must_fail key"""

Thinking more pie-in-the-sky-ly, maybe this approach could be
generalised further, to specify a mechanism by which request.POST can
include a set of validation errors that should be shown by the form.
This might have other uses in things like wizards where validation
errors may want to persist between different pages.

>  * The requirement for using RequestContext is somewhat annoying. I
> can't see any obvious way to piggyback CSRF data onto the base Context
> itself, but perhaps we can make RequestContext a more prominent
> default? The most obvious solution here is to make render_to_response
> use RequestContext by default.

Yes, yes and thrice yes - my only complaint about RequestContext is
how horrible the syntax for using it is:

render_to_response('template.html', {
'name': 'Foo',
'blah': 'baoenuth',
}, context_instance=RequestContext(request))



Yeah, I'd like a builtin shortcut like that - used like this:
  render(request, 'template_name.html', {'foo':bar })
The biggest problem, for me, is finding a decent name - since

Re: CSRF proposal and patch ready for review

[Simon Willison]

On Sep 18, 12:09 am, Luke Plant  wrote:
> OK, here is my response.  I hope this doesn't turn into a personal my-code-vs-
> your-code match (especially as most of "my code" is really other people's code
> and ideas :-) but I want to make sure we do this right, as it potentially has
> big implications, so the following will be "sleeves rolled up" (getting into
> details) and "gloves off" when it comes to criticisms of the code itself :-)

Completely up for that - this is really excellent stuff. Django needs
the best possible solution for this problem, so the more smart
discussion around it the better.

> Advantages of SafeForm
> --
>
> I can see the value of having contrib apps secured whatever the user has in
> their settings.py
>
> Unfortunately, that is the only advantage of any significance that I can
> see.

There are a couple of other advantages that I haven't document
anywhere:

1. SafeForm has no dependency on Django's session framework. This is A
Good Thing as I personally avoid sessions in my projects (no need to
access state on every request if you can get away with not doing it).

2. SafeForm fails relatively gracefully in the event of CSRF check
failures - since it's tied in to the form validation flow, it displays
an inline error message but you don't lose your form data. This is my
personal favourite characteristic of SafeForm, partly because it
allows you to be much more strict with your CSRF tokens - you can
create tokens that expire relatively quickly without risking too much
harm to your users.

> Disadvantages of SafeForm
> -
>
> 1) Big changes are required to use it.

Yes, very true. The form tag / RequestContext solution is a major
improvement there.

>  * quite commonly, significant changes to the template:
>    * Often you have to explicitly add the csrf token field.  And you've
>      got to worry about whether you need to or not.

Thankfully if you do add it when it's not needed nothing bad happens -
Django silently blanks out {{ form.csrf_token }} (I've complained
about this before but here it's actually useful).

>    * If you weren't displaying form.non_field_errors before (which can be
>      common if you don't need it), you have to add it.

Again, this actually isn't too bad - if you forget to add
non_field_errors, you'll just lose the CSRF validation message - your
form will still preserve data and can be resubmitted. I noticed Flickr
fail silently like that if you fail a CSRF check (by tampering with
their magic_cookie) - they simply redisplay the form with your data
filled in, without any kind of error message.

> 2) More than one way to do CSRF protection.
>
> Although you provide a low level mechanism for forms, it still means you have
> Two Ways To Do CSRF Protection, and they are quite different (from the users
> point of view).  Once you've added in the multiple forms in a  problem,
> you've now got Three Ways To Do CSRF Protection, all of which will be needed
> in quite common circumstances.

I count Four Ways - I see no reason to remove the Django 1.1
middleware method, which is a convenient way of adding CSRF protection
to an entire site, even if it's a bit ugly. Adding a new CSRF
protection mechanism is likely to significantly increase awareness of
the issue whatever we do, so I wouldn't be surprised to see a lot of
people start turning on the old middleware as a stop-gap measure. So
yes, this isn't exactly ideal.

> 3) More than one way to do Forms
> ...
> Every user now faces a choice - use Form or SafeForm?  Form seems to be
> simpler; the API is possible to play with at the interactive prompt (see
> tests/regressiontests/forms.py, for example), unlike SafeForm which requires
> an HttpRequest object; and SafeForm sounds like something for the paranoid,
> ultra-secure types.  So you can bet that tutorials and code across the
> internet are going to use Form.

I dunno, this feels like something that can be mostly addressed with
documentation and community pressure. I'm perfectly willing to spend
the next year explaining CSRF to anyone who cares to hear about it -
it's shocking how poorly understood it is within the industry as a
whole.

> Because they have switched to using SafeForm, that form will now
> include the CSRF token, which is leaked to external sites, which is a security
> vulnerability.

I'm not enormously worried about this one - again, I don't feel we'll
be able to solve the CSRF problem without some serious end-user
education. If we're really worried about this we could set the default
token expiry time to half an hour to at least limit the potential
damage a bit.

(I know end-user education is rarely the answer to anything, but at
least we're dealing with developers here).

> 4) CSRF is baked in to view functions.
> ...
>  * Developers can't 'swap out' the implementation of CSRF protection.  With
>    the template tag method, this is easy to do.  If someone has custom
>    

Re: CSRF proposal and patch ready for review

[Simon Willison]

On Sep 17, 3:42 pm, Simon Willison <si...@simonwillison.net> wrote:
> All good points - the change in function signature naturally fell out
> of the CSRF work (since the form needs access to the request object in
> both cases) but you've convinced me that it's a step too far - I'll
> see if I can fix that.

I just pushed a new release which changes the constructor signature
back to the way it works in regular Django:

http://pypi.python.org/pypi/django-safeform/2.0.0

(Really should have used 0.1 / 0.2 etc for the version numbers, lesson
learnt for next time!)

I also added some unit testing sugar to address one of Luke's (many)
points, see the "Unit testing" section of the README file. I'm still
digesting the rest of his e-mail, plenty to think about there.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

On Sep 18, 6:21 pm, Eric Holscher  wrote:
> I have looked into Logging before for another project, and I found that
> SQLAlchemy's support seemed to be a pretty good model to follow. They define
> all of their loggers under the sqlalchemy namespace, and then you can
> configure different handlers for different things[1]:
>
> I think that this would be necessary to have in Django, so that for
> instance, I could listen to the django.orm logs, and not the django.http, or
> listen to them with different handlers/levels.

Yes, absolutely - this looks like exactly the right model.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

On Sep 17, 4:04 pm, Russell Keith-Magee 
wrote: 
> I've seen several attempts to wrap Java loggers in a "nicer"
> interface, and every one of them ended up hobbling some of the power
> features of the logger. There is also the issue of our wrapper playing
> nicely with the loggers already being used in the wild.

I should clarify - by "lightweight wrapper" I basically mean a pre-
configured log setup and a standard place to import the logger from -
and maybe a tiny bit of syntactic sugar if it will make the common
case more palatable. I'm mostly just interested in making the logging
module an encouraged technique within the Django world. It should
definitely play nicely with any already-existant logging code.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: CSRF proposal and patch ready for review

[Simon Willison]

All good points - the change in function signature naturally fell out
of the CSRF work (since the form needs access to the request object in
both cases) but you've convinced me that it's a step too far - I'll
see if I can fix that.

Cheers,

Simon

On Sep 17, 2:10 pm, Russell Keith-Magee <freakboy3...@gmail.com>
wrote:
> On Thu, Sep 17, 2009 at 3:11 PM, Simon Willison <si...@simonwillison.net> 
> wrote:
>
> > On Sep 15, 2:57 pm, Luke Plant <l.plant...@cantab.net> wrote:
> >> OK, I'll wait and see.
>
> > Here's the code:http://github.com/simonw/django-safeform
>
> >>  * it requires using Django's form system.  I've got plenty of views that
> >> don't (e.g. anything with a dynamic number of controls).  People not using
> >> django.Forms are left out in the cold, or having to learn another way to do
> >> things.
>
> > I've covered this in django-safeform by exposing a low level interface
> > to the CSRF token creation and validation routines (in csrf_utils) -
> > see the readme for documentation on this.
>
> >>  * it's off by default.  Turning it on by default will require making
> >> django.forms.Form subclass SafeForm, which will almost certainly require a
> >> huge and immediate upgrade effort, because SafeForm cannot have the same 
> >> API
> >> as Form.  If it's off by default, I see no point at all in this entire
> >> exercise.  If we don't arrive at a point where the POST form created in the
> >> tutorial is safe from CSRF, I think we've failed.
>
> > My priority for this is a little different: while I would dearly love
> > to see all Django applications secure against CSRF by default, I can't
> > see a way of doing it that doesn't break existing apps. More important
> > for me though is that the Django admin (and other reusable apps, such
> > as Pinax stuff) MUST be secure against CSRF out of the box, no matter
> > what the user puts in their settings.py file. If developers fail to
> > protect themselves (especially when the solution is well documented)
> > then at least it isn't our fault.
>
> > I think this is subtly different from the XSS escaping issue simply
> > because the solution to CSRF is much more intrusive.
>
> >> And it will still require changes to templates, just like the template tag,
> >> and it will also require changes to views, only significantly more complex
> >> than simply using RequestContext.  It's got at least as many and at least 
> >> as
> >> intrusive 'moving parts' AFAICS.
>
> > The SafeForm API actually simplifies views - you go from this:
>
> > def change_password(request):
> >    form = ChangePasswordForm()
> >    if request.method == 'POST':
> >        form = ChangePasswordForm(request.POST)
> >        if form.is_valid():
> >            return HttpResponse('Thank you')
> >    return render_to_response('change_password.html', {
> >        'form': form,
> >    })
>
> > To this:
>
> > @csrf_protect
> > def change_password(request):
> >    form = ChangePasswordForm(request)
> >    if form.is_valid():
> >        return HttpResponse('Thank you')
> >    return render_to_response('change_password.html', {
> >        'form': form,
> >    })
>
> > The SafeForm deals with the request.method == 'POST' bit, meaning one
> > less conditional branch within the view function itself.
>
> Did we discuss this bit at the sprint? I'm completely on board with
> everything else, but this bit makes me nervous:
>
>  * It calls is_valid() on the form during a GET (which isn't currently
> done as part of normal practice).
>
>  * It isn't possible to have any special view handling based on the
> request method.
>
>  * It implies that the method of construction the form will always be
> the same, regardless of whether we are GETting or POSTing. This isn't
> always true - in particular, I'm thinking of the construction of
> FormSets, which can use a queryset to populate the forms on GET, but
> don't require the queryset on POST.
>
> > I'm pretty happy with the SafeForm interface now that I've fleshed it
> > out - it's a lot less clunky than I was originally expecting.
>
> Another advantage that Simon hasn't mentioned - it requires no
> modifications to the tutorial. We can present the basic ideas of form
> handling in tutorial 4 without mentioning CSRF. We then have scope to
> introduce a new tutorial on security issues, that can both educate on
> the nature of XSS and CSRF attacks, plus what you have to do in order
> to protect against them.
>
> Yours,
> Russ Magee %-)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Replacing get_absolute_url, I am against it

[Simon Willison]

On Sep 15, 12:04 pm, Ivan Sagalaev  wrote:
> James Bennett wrote:
> > Except I can't help thinking this is an awfully arbitrary distinction
> > to draw. In effect you're saying that nearly every question about an
> > object should be answerable by interrogating it directly, *except* for
> > "what's a URL I can use for you?"
>
> May be I can explain this distinction with an example.
>
> We once had two different web sites about "events". They both had a
> "core" set of models but each one had their own set of views & urls. So
> for a core Event model a question "what's your URL" just didn't make
> sense. It had two different URLs depending on the project it was
> imported in.

For me, this one boils down to pragmatism v.s. purity.

>From a purity point of view, having models that know what their URLs
are when Django's concept of a URL is defined in the urls.py file is
an obvious violation of separation of concerns.

>From a pragmatism point of view, being able to ask an object it's URL
is incredibly convenient - and in 90% of the projects I've worked on
in the past few years each object has had one and only one URL (in
fact REST principles actively encourage this).

I'm OK with .get_url() and .get_url_path() (I still prefer the
semantics and names used in my original proposal to any of the
proposed alternatives) only being useful 90% of the times, with
projects where objects can have more than one URL needing to find some
alternative method.

As for running code, allow me to point to http://code.google.com/p/django-urls/
:)

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: SoC merge plans?

[Simon Willison]

On Sep 13, 12:52 pm, Russell Keith-Magee 
wrote:
> Although the branch isn't ready for merge yet, the code that is
> present is all up to spec. I think the sprints have produced workable
> solutions for the outstanding problems; fixes implementing the
> solutions we discussed should be forthcoming shortly.

With my cowboy hat on... what are the disadvantages to merging this to
trunk /now/ and fixing the remaining outstanding issues there? It
might make it easier to get the other SoC branches compatible with
this one (which sounds to me like it makes the most far-reaching
changes) and it would encourage the wider community to help check that
everything really is backwards compatible.


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Proposal for 1.2: built-in logging with django.core.log

[Simon Willison]

I think we should add logging to Django in version 1.2, implemented as
a light-weight wrapper around the Python logging module
(django.core.log maybe?) plus code to write errors to the Apache error
log under the mod_python handler and environ['wsgi.errors'] under WSGI
(meaning mod_wsgi will write to the Apache error log as well).

Benefits of logging as a core Django service


Adding logging to Django core would provide the following benefits:

1. We'll be able to de-emphasise the current default "e-mail all
errors to someone" behaviour, which doesn't scale at all well.

2. Having logging in the core framework will mean people start
actually using it, which will make it easier for people to debug their
Django apps. Right now adding "print" statements to a Django app is a
common debugging technique, but it's messy (you have to remember to
take them out again) and error prone - some production environments
throw errors if an app attempts to write to stdout. It's also not
obvious - many developers are surprised when I show them the
technique.

3. Logging in Django core rather than a 3rd party app will encourage
reusable applications to log things in a predictable way, standard
way.

4. 3rd party debugging tools such as the debug toolbar will be able to
hook in to Django's default logging behaviour. This could also lead to
plenty of additional 3rd party innovation - imagine a tool that looks
out for logged SQL that took longer than X seconds, or one that groups
together similar log messages, or streams log messages to IRC...

5. Built-in support for logging reflects a growing reality of modern
Web development: more and more sites have interfaces with external web
service APIs, meaning there are plenty of things that could go wrong
that are outside the control of the developer. Failing gracefully and
logging what happened is the best way to deal with 3rd party problems
- much better than throwing a 500 and leaving no record of what went
wrong.

6. Most importantly from my point of view, when a sysadmin asks where
Django logs errors in production we'll have a good answer for them!

7. As a general rule, I believe you can never have too much
information about what's going on with your web application. I've
never thought to myself "the problem with this bug is I've got too
much information about it". As for large log files, disk space is
cheap - and pluggable backends could ensure logs were sensibly
rotated.

Places logging would be useful
==

- Unhandled exceptions that make it up to the top of the Django stack
(and would cause a 500 error to be returned in production)
- The development web server could use logging for showing processed
requests (where currently these are just printed to stdout).
- Failed attempts at signing in to the admin could be logged, making
security audits easier.
- We could replace (or complement) django.connection.queries with a
log of executed SQL. This would make the answer to the common question
"how do I see what SQL is being executed" much more obvious.
- Stuff that loads things from INSTALLED_APPS could log what is being
loaded, making it much easier to spot and debug errors caused by code
being incorrectly loaded.
- Likewise, the template engine could log which templates are being
loaded from where, making it easier to debug problems stemming from an
incorrectly configured TEMPLATE_DIRS setting.
- We could use logging to address the problems with the template
engine failing silently - maybe some template errors (the ones more
likely to be accidental than just people relying on the fail-silent
behaviour deliberately) should be logged as warnings.

Most of the above would be set to a low log level which by default
would not be handled, displayed or stored anywhere (logging.info or
similar). Maybe "./manage.py runserver --loglevel=info" could cause
such logs to be printed to  the terminal while the development server
is running.

Problems and challenges
===

1. The Python logging module isn't very nicely designed - its Java
heritage shines through, and things like logging.basicConfig behave in
unintuitive ways (if you call basicConfig twice the second call fails
silently but has no effect). This is why I suggest wrapping it in our
own higher level interface.

2. There may be some performance overhead, especially if we replace
mechanisms like django.connection.queries with logging. This should be
negligble: here's a simple benchmark:

# ("hello " * 100) gives a 600 char string, long enough for a SQL
statement
>>> import timeit, logging
>>> t = timeit.Timer('logging.info("hello " * 100)', 'import logging')
>>> t.timeit(number=100) # one hundred statements
0.00061702728271484375
>>> t.timeit(number=100) # one million statements
6.458014965057373

That's 0.0006 of a second overhead for a page logging 100 SQL
statements. The performance overhead will go up if you attach a
handler, but that's 

Re: CSRF proposal and patch ready for review

[Simon Willison]

On Sep 15, 2:57 pm, Luke Plant  wrote:
> OK, I'll wait and see.

Here's the code: http://github.com/simonw/django-safeform

>  * it requires using Django's form system.  I've got plenty of views that
> don't (e.g. anything with a dynamic number of controls).  People not using
> django.Forms are left out in the cold, or having to learn another way to do
> things.

I've covered this in django-safeform by exposing a low level interface
to the CSRF token creation and validation routines (in csrf_utils) -
see the readme for documentation on this.

>  * it's off by default.  Turning it on by default will require making
> django.forms.Form subclass SafeForm, which will almost certainly require a
> huge and immediate upgrade effort, because SafeForm cannot have the same API
> as Form.  If it's off by default, I see no point at all in this entire
> exercise.  If we don't arrive at a point where the POST form created in the
> tutorial is safe from CSRF, I think we've failed.

My priority for this is a little different: while I would dearly love
to see all Django applications secure against CSRF by default, I can't
see a way of doing it that doesn't break existing apps. More important
for me though is that the Django admin (and other reusable apps, such
as Pinax stuff) MUST be secure against CSRF out of the box, no matter
what the user puts in their settings.py file. If developers fail to
protect themselves (especially when the solution is well documented)
then at least it isn't our fault.

I think this is subtly different from the XSS escaping issue simply
because the solution to CSRF is much more intrusive.

> And it will still require changes to templates, just like the template tag,
> and it will also require changes to views, only significantly more complex  
> than simply using RequestContext.  It's got at least as many and at least as
> intrusive 'moving parts' AFAICS.

The SafeForm API actually simplifies views - you go from this:

def change_password(request):
form = ChangePasswordForm()
if request.method == 'POST':
form = ChangePasswordForm(request.POST)
if form.is_valid():
return HttpResponse('Thank you')
return render_to_response('change_password.html', {
'form': form,
})

To this:

@csrf_protect
def change_password(request):
form = ChangePasswordForm(request)
if form.is_valid():
return HttpResponse('Thank you')
return render_to_response('change_password.html', {
'form': form,
})

The SafeForm deals with the request.method == 'POST' bit, meaning one
less conditional branch within the view function itself.

I'm pretty happy with the SafeForm interface now that I've fleshed it
out - it's a lot less clunky than I was originally expecting.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Long-term direction for shortcuts

[Simon Willison]

On Aug 3, 3:53 am, chairface  wrote:
> Also, it is only a single video, but that video
> is a pretty good resource for the topic of Django's future.  It was
> the topic of the entire hour.  Are you implying that it's a poor
> source for people wondering where Django is going?  It was explicitly
> designed to be exactly that kind of source.

I'm sorry if I gave that impression, but it really wasn't my
intention. The talk was titled "Django Heresies" partly because I
wanted to emphasize that it was NOT a discussion of Django's intended
future - it was more an exploration of things which one person (me)
thought we should reconsider as a community and open up for further
discussion. It was meant to be controversial!

> Simon has not justified why his particular preference for imports
> necessarily meets even the common case and why he, or somebody else,
> cannot put their own choices for common imports into a single that they
> import (pretty normal "reduce repetition" pattern in Python).

All I was saying is that I find myself repeating the same imports at
the top of every views.py - and wondering if there's a better option.
For my own projects I inevitably end up creating my own shortcuts.py
module. I wish I didn't have to.

As an aside, I've been experimenting with the single import pattern
(which I rather like, but everyone else seems to hate) in djng:

http://github.com/simonw/djng/blob/953eb33390972cbdd0ac0a52e3b23bfdd55e2cfe/example_forms.py

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multi-DB Update

[Simon Willison]

On May 2, 12:19 am, Malcolm Tredinnick 
wrote:
> I'd prefer dictionaries to strings, because strings are tough to modify
> dynamically -- as has already been demonstrated a few times in Django's
> history.

That's a pretty strong argument in favour of dictionaries - in which
case it might be good to switch the CACHE_BACKEND setting over to
using dictionaries instead of a DSN-style string at some point in the
future - consistency seems like it would be a good thing here.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multi-DB

[Simon Willison]

On Apr 30, 12:25 am, Alex Gaynor  wrote:
> My question would be how is using a DSN and letting those be passed directly
> to using() any advantage over letting someone pass a dict of those options
> to using(), or a connection object itself.

No advantage at all - I'm interested in being able to hand a
dynamically constructed connection to using(), and I don't
particularly mind if it's a DSN, a dictionary or a connection object
I've created already.

My principle arguments for DSNs are the other two - we already use
them for cache backends, and it's what SQLObject and SQLAlchemy do.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multi-DB

[Simon Willison]

On Apr 25, 7:11 am, Alex Gaynor  wrote:
> Simon uses a new DSN based syntax for specifying DBs, however I find this
> pretty akward in the presence of our existing way for defining DB settings.

I had a few reasons for suggesting DSNs:

* We already use DSN-style syntax for cache backends - it's weird to
do it there (for something that isn't actually a database) but not for
our database settings. Of course, that inconsistency could be fixed by
splitting up the settings for cache backends instead.
* SQLObject and SQLAchemy both use DSNs.
* If we implement my suggestion that a using() method can take a
connection object OR a connection alias string (defined in
settings.py) OR a DSN, it's much easier to dynamically construct a
database connection at runtime.

I'm particularly interested in that last usage scenario. Personally
I'm not a big fan of settings.py - over the past four years I've found
myself at various times wanting to dynamically alter almost every
setting in there - and there are some interesting edge cases where
creating a database connection at run time could be useful. Three off
the top of my head:

1. A phpMyAdmin style application for administering remote databases,
which might allow the user to enter their database details in to an
online form
2. A web-based installation mechanism, such as the one used by
WordPress
3. (really nutty one this) - a very high scale situation where an
application is partitioned across hundreds of databases, which each
one having the same set of tables. This is how WordPress-MU works (as
used by wordpress.com), with every blog getting its own duplicate set
of tables -  and it's surprisingly effective.

Creating a database connection at runtime doesn't require DSNs of
course, provided you can feed your own connection object to any using
() clause rather than being tied to using connections that have
already been configured in settings.py.

> Further a DSN method removes the presence of something like
> DATABASE_OPTIONS(unless we start adding query strings onto the end), which
> is very useful for things like the postgresql auto commit option, plus I
> plan to move the TEST_DATABSE_* settings into DATABASE_OPTIONS, since things
> like charset are DB dependent.

Again, we use query strings for cache backends so I have no problem
with using them for database options.

I wouldn't be heartbroken if DSNs weren't used in Django, but the
above should at least explain the reasons behind my suggestion.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 24, 2:18 pm, zellyn <[EMAIL PROTECTED]> wrote:
> Would it make sense to have the middleware/view decorator set a
> property on the request, and pass the request to all forms, and have
> *all* forms CSRF-protect themselves when the property is set? That
> would make it easy to add protection to externally-developed apps.

That's an interesting idea. I'm a bit cautious of requiring ALL forms
to take a request object though - Django's current form library is
decoupled from Django's request object, which is actually a really
useful property. I've used a "form" class to import CSV data in to
Django ORM models in the past for example, using the form logic to
handle the necessary type conversions.

Keeping django.forms decoupled from HttpRequest also ensurse it can be
used by other Python web frameworks that don't have the same request/
response model.

I'm fine with SafeForm depending on HttpRequest, but I'd rather not
introduce that dependency to BaseForm and Form.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Inner imports in core code

[Simon Willison]

On Sep 24, 1:40 pm, Simon Willison <[EMAIL PROTECTED]> wrote:
> As a result the performance overhead from having imports inside
> functions as opposed to at module level should be virtually non-
> existent.

Not entirely sure how I managed to miss your benchmark figures when I
read your mail, but I'm wrong here - there's clearly a performance
overhead involved in importing inside a function. It would be
interesting to see how much this affects Django performance as a whole
- I guess this is the kind of question the proposed metronome
benchmark will help answer.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Inner imports in core code

[Simon Willison]

On Sep 24, 8:37 am, David Cramer <[EMAIL PROTECTED]> wrote:
> I was digging through some code today, and I noticed imports are
> happening within a lot of functions. It was my knowledge that it works
> like so:
>
> import in a function is the same as ruby's load or php's include --
> its executed everytime the function is
>
> import in a module outside a function is like ruby's require or php's
> require_once -- its loaded and cached in memory

That's not the case. Try the following:

# importme.py
print "running importme"

def three():
  print "3 - from importme"

# test1.py
import importme
importme.three()

# test2.py
def one():
  import importme
  print "1"
  importme.three()

def two():
  import importme
  print "2"
  importme.three()

one()
two()

$ python test1.py
running importme
3 - from importme

$ python test2.py
running importme
1
3 - from importme
2
3 - from importme

As you can see, the initial print statement in importme.py only
executes once, even when "import importme" is run twice (from inside
functions).

As I understand it, Python's "import" statement checks sys.modules to
see if something has been imported already. If it hasn't, the module
is loaded for the first time and executed. If it has already been
imported (and hence is in sys.modules) then no extra code is loaded or
executed - the import statement merely ensures that a reference to
that module is available in the function's scope.

As a result the performance overhead from having imports inside
functions as opposed to at module level should be virtually non-
existent. I don't think we need to worry about this - and like Jeremy
said, usually there's a reason for doing it (avoiding importing
something unless it's actually going to be needed, or avoiding
circular imports).

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 23, 11:23 pm, Simon Willison <[EMAIL PROTECTED]> wrote:
> CSRF attacks are a problem for systems where an action is only meant
> to be available to a specific logged in user. This user is inevitably
> identified by a unique cookie. This is normally a session cookie,
> hence many CSRF protection mechanisms key their hidden form token off
> the session cookie.

It turns out it's not that straight-forward after all:

http://icanhaz.com/csrfpdf (PDF link, "Robust Defenses for Cross-Site
Request Forgery")

The above paper introduces the "login CSRF" attack, where CSRF is used
to force a victim to log in to a site using /the attacker's/
credentials. The hope is that the user will then enter personally
sensitive information which the hacker can harvest later on.

Django's CSRF mechanism needs to be able to protect forms even in the
absence of a unique-to-the-user cookie, which means it needs a way of
setting its own cookies. We can either do this using the
form.protect() or form.render_response() methods I advocated earlier,
or we can use a middleware/view decorator combination. I think I'm
leaning towards the view decorator / middleware option now.

Cheers,

Simon

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 23, 11:51 pm, "Amit Upadhyay" <[EMAIL PROTECTED]> wrote:
> There is another option, a template tag. I would implement it as a
> middleware and a template tag. Template tag csrf_protect, will require
> CSRFMiddleware and django.core.context_processors.request, will add a
> input file containing something derived from {{ request }} and
> middleware will check and raise HttpForbidden.

Oddly enough that's exactly how ASP.NET MVC does it:

http://blog.codeville.net/2008/09/01/prevent-cross-site-request-forgery-csrf-using-aspnet-mvcs-antiforgerytoken-helper/

They use their equivalent of a view decorator rather than middleware,
which is what I'd suggest doing in this case as well (middleware in
Django is usually applied globally which isn't always what you want).

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 24, 1:04 am, Simon Willison <[EMAIL PROTECTED]> wrote:
> There's another option that avoids the need for any cookies at all:
> generating a persistent one-use-only token when a form is saved,
> storing that in the database and only allowing submissions that
> include a token that was previously assigned.

Scratch that - the tokens would still need to be assigned to an
individual user (almost certainly keyed off a cookie) as otherwise an
attacker could create their own tokens and use them to attack another
user.

It would work for sites protected using HTTP authentication rather
than cookies though, as you'd be able to attach each token to the HTTP
auth username. I don't think this is a case we need to address though.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 23, 11:23 pm, Simon Willison <[EMAIL PROTECTED]> wrote:
> CSRF attacks are a problem for systems where an action is only meant
> to be available to a specific logged in user. This user is inevitably
> identified by a unique cookie. This is normally a session cookie,
> hence many CSRF protection mechanisms key their hidden form token off
> the session cookie.

There's another option that avoids the need for any cookies at all:
generating a persistent one-use-only token when a form is saved,
storing that in the database and only allowing submissions that
include a token that was previously assigned.

This avoids any need for cookies at all, but has the serious
disadvantage that you end up with potentially thousands of useless
tokens stored in your database. You can clear these out periodically
but it's still overhead that it would be nice to avoid.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 23, 1:06 am, Simon Willison <[EMAIL PROTECTED]> wrote:
> Since all form.protect(response) actually does is ensure that there's
> a csrf cookie, you should only actually have to do it once for one of
> the forms.

Unsurprisingly, I've been totally over-engineering this. There's no
need for form.protect(response) at all - in fact, the form doesn't
need to have anything to do with the response object. This
dramatically simplifies things - in fact, it takes us back to the
original proposal of a SafeForm that just takes the request object as
an argument to its constructor.

CSRF attacks are a problem for systems where an action is only meant
to be available to a specific logged in user. This user is inevitably
identified by a unique cookie. This is normally a session cookie,
hence many CSRF protection mechanisms key their hidden form token off
the session cookie. That's also what Django's CSRF middleware does.

I like to be able to avoid sessions where possible, which is why I
didn't want SafeForm to use the session cookie and thought it would
need to set a cookie itself. But even if an application isn't using
sessions, it's still going to need to use one or more cookies that are
unique to the user.

I think we solve this with a setting, the default of which looks like
this:

CSRF_COOKIES = [SESSION_COOKIE_NAME]

SafeForm creates its csrf_token based on a hash of the cookie values
for the keys in that list. It's a list because some authentication
cases may rely on more than one cookie to uniquely identify the user
(rare but possible). By default, it will use the Django session
cookie. If you're not using sessions in your app (you're using a
signed cookie called 'current_user' for example) you can specify that
in the CSRF_COOKIES session.

So, the bit about the SafeForm needing access to the response was a
red herring.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 23, 10:17 pm, "Robert Coup" <[EMAIL PROTECTED]>
wrote:
> then when you get a form submission, base64-decode it, split at "/", check
> the hash matches by recalculating it, then use the proximity-to-timestamp
> and the remote_addr to check the form validity.

Anything that relies on remote_addr is flawed, because IP addresses
change all the time. I frequently load up a Google Groups thread on my
laptop, compose a reply on the train to work and submit it when I get
there - and since I've moved networks my IP address changes in between
loading the form and submitting it. There's also the risk of proxies
that load balance traffic through different IP addresses, not to
mention IP addresses that are shared by many people (including a
potential attacker).

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Denormalisation, magic, and is it really that useful?

[Simon Willison]

On Sep 23, 2:00 pm, "Marty Alchin" <[EMAIL PROTECTED]> wrote:
> Without some Python-based approach, all I could see is maybe adding a
> cross-platform "create trigger" API (ugh) to Django, which an
> application could then use to set up its triggers during syncdb.
> Otherwise, something like that forum app would have to implement a
> trigger for all available backends or just ship with instructions on
> how to set it up yourself.

Triggers also aren't supported in Drizzle ( https://launchpad.net/drizzle
), which Django will probably want to target at some point.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 23, 9:00 am, oggie rob <[EMAIL PROTECTED]> wrote:
> I just read this thread now, and by the end of it, things are looking
> pretty complicated. Is it worth a gut check to make sure this is
> worthwhile? Although the middleware might be a hack now, it seems
> sensible that it fits in request/response areas rather than in forms:
> you still need to go out of your way to add it anyway (so users won't
> necessarily "turn it on"); it takes a lot more code; add in the
> multiple forms per page question, and to me it seems like you've fixed
> a problem by introducing another.

Here's a useful case in point: the admin. Django's admin should ship
with CSRF protection turned on and baked in. Right now, I'm willing to
bet 95% of the Django admin sites out there are exploitable via CSRF
because the middleware wasn't turned on. This is really bad.

I'm positive we can figure out a better API for CSRF protection than
what we have at the moment. At the moment I'm focused on forms, but if
there's something we can do at the view level instead I'd love to see
some suggestions.

> Finally, it doesn't take much to make a pretty message - something
> like "You are under attack, close down your browser and try again"
> with images of flaming people & such - for the (lets be realistic)
> very rare cases when a CSRF attack occurs.

I'm worried about false positives. One example where this would happen
is if you were to change your SECRET_KEY (secret management is a whole
other issue we haven't addressed). That's why I like the validation
error approach - it's unobtrusive and doesn't unnecessarily scare
people. We should definitely log detected CSRF issues though (logging
= another issue).

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 23, 12:53 am, Tai Lee <[EMAIL PROTECTED]> wrote:
> On Sep 23, 9:27 am, Simon Willison <[EMAIL PROTECTED]> wrote:
>
> > The significant downside is that having a render() method on a form
> > that performs the same function as render_to_response feels really,
> > really strange. It's convenient, but it just doesn't feel right and
> > I'm not sure I can justify it.
>
> How would this work when you have multiple forms/modelforms/formsets
> on one page?

In that case, you'd have to fall back to the old way of doing things -
form.render() would be a shortcut, not the only way of doing this.
You'd probably end up with code that looked like this:

def complex_view(request):
form_one = FormOne(request)
form_two = FormTwo(request)
form_three = FormThree(request)
if form_one.is_valid() and form_two.is_valid() and
form_three.is_valid():
# Do something with their cleaned_data
return HttpResponseRedirect("/done/")
response = render_to_response('complex_view.html', {
'form_one': form_one,
'form_two': form_two,
'form_three': form_three
})
form_one.protect(response)
form_two.protect(response)
form_three.protect(response)
return response

Since all form.protect(response) actually does is ensure that there's
a csrf cookie, you should only actually have to do it once for one of
the forms. Maybe it shouldn't be a form method at all in that case -
maybe it should be a function called django.forms.protect(request,
response) (request so it can check if the cookie has been set already,
and response so it can set it if needed).

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Denormalisation, magic, and is it really that useful?

[Simon Willison]

On Sep 23, 12:21 am, "Justin Fagnani" <[EMAIL PROTECTED]>
wrote:
> In my experience at least, denormalization occurs
> a lot and leaves a lot of room for mistakes, so it's something a
> framework should handle if possible.

Just so it's on the record, I'd like any denormalisation tools in
Django to include a mechanism for re-syncronizing them should
something go awry (like direct updates being applied to the database
without keeping the denormalised fields in sync). This mechanism could
then be exposed as a ./manage.py command which could be called
periodically to verify and fix any incorrect data.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 23, 12:27 am, Simon Willison <[EMAIL PROTECTED]> wrote:
>     return form.render('add_article.html', {
>         'extra_context_args': 'Go here',
>     })

Using render_response as the method name might make this a little more
palatable:

 return form.render_response('add_article.html', {
 'extra_context_args': 'Go here',
 })

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 22, 9:41 pm, Brian Beck <[EMAIL PROTECTED]> wrote:
> - If there's some other way to spell form.protect(response).

Here's a crazy idea that might just work:

class AddArticleForm(forms.SafeForm):
headline = forms.CharField()
# ...

def add_article(request):
form = AddArticleForm(request)
if form.is_valid():
# Process the data in form.cleaned_data
return HttpResponseRedirect('/done/')
return form.render('add_article.html', {
'extra_context_args': 'Go here',
})

We're doing a bunch of interesting things here. First, it's
AddArticleForm.__init__ itself that looks at request.method and
decides if it's going to bind to the data (if request.method ==
'POST') or simply create a new blank form. Second, we're using
form.render() instead of render_to_response. form.render is a thin
wrapper around render_to_response that does the following:

1. Adds 'form' to the context - a "form_var='article_form'" keyword
argument could be used to change this default behaviour
2. Uses a RequestContext (with the request that was passed to
AddArticleForm.__init__) - I can't think of any reason not to
3. Creates the HttpResponse using render_to_response
4. Sets a CSRF cookie on the response, if necessary

This solves all of our problems in one shot (the need to sometimes set
a cookie, having access to the request, etc), with the added bonus of
reducing the amount of view boilerplate needed to use a form to the
absolute minimum.

The significant downside is that having a render() method on a form
that performs the same function as render_to_response feels really,
really strange. It's convenient, but it just doesn't feel right and
I'm not sure I can justify it.

Interesting option though.

Cheers,

Simon


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 22, 11:09 pm, Jan Oberst <[EMAIL PROTECTED]> wrote:
> I'd protect all my forms if there's a neat way to do it. Why would it
> only apply to logged-in users? I'm not using contrib.auth.

It doesn't need to only apply to contrib.auth logged in users, but it
should only be used for forms which are behind some kind of cookie-
based protection (auth is the most obvious example, but if you've
rolled your own authentication scheme you should be able to use
SafeForm as well).

There's no point in protecting a form which anyone can use (e.g. a
public "contact us" form) as the purpose of CSRF is for an attacker to
force you to perform an authenticated action that you don't want to
perform - deleting something from a CMS for example. If anyone can use
the form in question the attacker can just go and submit it
themselves.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 22, 10:21 pm, "Jacob Kaplan-Moss" <[EMAIL PROTECTED]>
wrote:
> This makes me think -- is it possible that CSRF protection at the form
> level is too low? Perhaps it's something that needs to be happening
> at, say, the view level? Some sort of decorator, and/or a tag to spit
> out the CSRF token in the template...

Interesting thought. It feels like the form is the right place for
this for a couple of reasons:

1. It involves adding an extra form field
2. When a CSRF check fails, it's polite to show a message. Form
validation is a good place for this.

The downside of doing it at the form level is the need to have access
to the request and (potentially) the response as well, for setting a
cookie.

Doing things at the view level (with a decorator) provides access to
both request and response objects, but doesn't provide access to form
fields or validation errors.

Maybe the answer is a combination of both - a form subclass and a
decorator on the view?

Will have to think about that.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

On Sep 22, 10:01 pm, Brian Beck <[EMAIL PROTECTED]> wrote:
> > > -- What about third-party app forms that aren't SafeForms, but need to
> > > be?  The situation dictates this, not the form author.
> > I think we keep CSRF middleware around to deal with that. We also very
> > actively encourage third party apps to adopt SafeForm as soon as 1.1
> > is out.
>
> But still, the situation dictates the need for SafeForm, not the form
> author.  If this becomes best practice, essentially *every* form will
> need to be initialized with a request.

One thing that might help out in this case would be the ability to
create a SafeForm from a regular Form (which might be an argument for
csrf protection as a feature of django.forms.Form rather than a
subclass). If the third party code is well written (it follows the
class-based generic view idiom for example, providing a get_form()
method that can be over-ridden) it should be straight forward to
intercept the form it creates and upgrade it to a SafeForm.

You've reminded me of another problem with SafeForm: how does it
interact with ModelForms? Is there a SafeModelForm as well? What about
FormSets?

Cheers,

Simon


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Proposal: django.forms.SafeForm - forms with built in CSRF protection

[Simon Willison]

CSRF[1] is one of the most common web application vulnerabilities, but
continues to have very poor awareness in the developer community.
Django ships with CSRF protection in the form of middleware, but it's
off by default. I'm willing to bet most people don't turn it on.

I don't believe middleware is the right way to approach this. It's too
"magic" - it involves code that parses and re-writes your HTML as the
response is being returned. It also means CSRF failures can't be
gracefully handled - the middleware can throw up a big ugly error
page, but ideally a CSRF failure would be treated the same way as a
regular form validation error.

I propose django.forms should include a SafeForm class, which is a
subclass of Form that includes built-in protection against CSRF. I
imagine the interface looking something like this:

from django import forms

class AddArticleForm(forms.SafeForm):
headline = forms.CharField()
author_email = forms.EmailField()
# etc

The interface to the form needs to be slightly different as the form
needs access to the full request object - CSRF validation requires
inspecting a cookie or session variable. Rather than passing
request.POST I suggest passing just request:

def add_article(request):
if request.method == 'POST': # If the form has been submitted...
form = AddArticleForm(request)
if form.is_valid(): # All validation rules pass
# Process the data in form.cleaned_data
# ...
return HttpResponseRedirect('/done/')
else:
form = AddArticleForm()

response = render_to_response('add_article.html', {
'form': form,
})
form.protect(response)
return response

Note the "form.protect(response)" at the end - it's a bit ugly, but we
need some way of setting a cookie on the response that can later be
used to verify that the form submission did indeed come from our own
code.

The template will need to include the hidden CSRF token fields:


{{ form.non_field_errors }}
{% for field in form %}

{{ field.errors }}
{{ field.label_tag }}: {{ field }}

{% endfor %}
{{ form.csrf_fields }}



A CSRF failure message (I still haven't worked out the ideal user-
facing copy for this, "Your form session has expired; please re-
submit" is my current favourite) will be included as one of the
non_field_errors.

I'm also not sure how to deal with generating valid HTML in the above
- {{ form.csrf_fields }} needs to be wrapped in a block level element
for irritating reasons.

I suggest using cookies for the additional token rather than sessions
as doing so removes the dependency on Django's session framework (a
good idea for large sites that don't want the overhead of storing
thousands of inactive sessions).

Why not build this in to django.forms.Form directly? Because CSRF is
only an issue for forms that are supposed to only be used by
authenticated users. Forms that don't require a cookie don't need to
be protected.

The other option would be for the django.forms.Form constructor to
take a "csrf_protection=True" keyword argument, but this doesn't feel
as neat as a subclass.

That's as far as my thinking has got at the moment. I don't see any
reason this needs to be developed from the start as a patch or branch
against Django - it can be done as an external project and merged in
to django.forms once the code is well tested and the API is finalized.

(I'm totally up for working on this, but I really, really need to ship
the new version of django_openid first.)

Cheers,

Simon

[1] Some CSRF links for the uninitiated:
* http://en.wikipedia.org/wiki/Cross-site_request_forgery
* http://simonwillison.net/2008/talks/amajax-security/
* http://shiflett.org/articles/cross-site-request-forgeries
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Template inheritance and {% include %} tags

[Simon Willison]

On Sep 18, 11:58 pm, SmileyChris <[EMAIL PROTECTED]> wrote:
> I think you missed the point, Simon.
>
> Michael isn't talking about self-referencing extending, he's talking
> about blocks in statically included templates ({% include "bit.htm"
> %}).

Ah yes, sorry - misunderstood the original e-mail.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---