Proposal - Handling arbitrary request Content-Types.

[Tom Christie]

Hey All,

  This is somewhat related to the ticket “Process HTTP PUT into 
request.FILES and request.PUT as done for POST” [1], although much broader 
in scope.

  In that ticket there’s a link to a related discussion [2] where Malcolm 
Tredinnick explains why a request.PUT that mirrors the current request.POST 
behaviour isn’t the right thing to do.  Obviously Malcom’s right that 
request.POST is a special case that’s limited to supporting browser form 
submissions, which means restricting it’s behaviour to requests with a 
method “POST” [3] and a content type of either 
“application/x-www-form-urlencoded” or “multipart/form-data” (and apparently 
“text/plain” as per HTML5) [4].

  However, it would still be useful if Django provided built-in behaviour to 
be able to support accessing the content of HTTP requests for requests other 
than just form POSTs, at some higher level than simply forcing the developer 
to use request.read() or request.raw_post_content.

  I don’t want to get too bogged down in implementation details at this 
point, more just gauge a response to the proposal - if it’s a “that sounds 
like a decent idea”, or if it’s a “ain’t never gonna happen in core” kind of 
thing, so I’ll (try!) to keep it reasonably high level for now...


So...

  Generic Content-Type support could be provided either by re-purposing the 
request.REQUEST attribute (obviously making sure it’s done in a backwards 
compatible way [*]), or by adding a new lazily evaluated request.CONTENT.

  The request.REQUEST/request.CONTENT would be used to return the parsed 
content of the request, as determined by a set of installed parsers.  A 
request.content_parsers attribute would also be added to the request [**], 
which would default to initializing from settings.CONTENT_PARSERS, and would 
used to determine which parser would be used to de-serialize the request.  
  
  Django could initially ship with a FormParser (And perhaps also some other 
useful defaults such as a JSONParser.)  The default for 
settings.CONTENT_PARSERS would probably just include the FormParser.

  The FormParser would handle “application/x-www-form-urlencoded” and 
“multipart/form-data requests”, and return a QueryDict containing both the 
content data and uploaded files, reusing the existing MultiPartParser in 
it's implementation. (and respecting the existing upload_handlers for file 
uploads.)
  
  You would probably also want to add a ‘content_parsers’ view decorator 
that could be used to set the list of parsers used on a given request at the 
view level.  That’d mean that usage would look something like this...

def standard_view(request, *arg, **kwargs):
  request.POST[‘foo’] # Does not work for form PUT requests
  request.REQUEST[‘foo’]  # Now works for form PUT requests

@content_parsers(JSONParser)   # NB. Decorator would take either a 
tuple/list or a single value
def json_view(request, *args, **kwargs):
  request.REQUEST[‘foo’]  # Works for application/json requests

@content_parsers(BinaryUploadParser)
def put_upload_view(request, *args, **kwargs):
  uploaded_file = request.REQUEST  # An UploadedFile object

(*NB, I’m not suggesting here that Parsers would be limited to returning 
pre-parsed objects, you might also create streaming parsers that return 
generator objects.)


  I like the idea because it follows the same pattern as request.POST but 
supplements the behaviour to support arbitrary methods and content-types, 
which is pretty core stuff for anyone trying to do any Web API stuff with 
Django.  Forcing the developer to drop down to 
request.raw_post_content/request.read() isn’t particularly helpful if we 
could provide a nice flexible out of the box solution.

  Obviously there’d be a lots to thrash out, and there’s also some fiddly 
complexity there (EG handling request.raw_post_content, see ticket 9054 [5]) 
but what do you peeps think of the idea in general?  Is it something that’d 
be valuable in core, would it only merit further discussion if there was a 
more explicitly detailed proposal, or if there was a reference 
implementation, or is it entirely out-of-scope?  All of what I’ve mentioned 
could equally be implemented as Middleware, so would that make more sense 
than implementing it in core, and if it was done as Middleware would it 
still be useful for inclusion into Django?

  I realise it’s a fairly big proposal (esp. for a first post to the list) 
so feel free to go ahead and shoot me right down!  :)

  Cheers,

Tom


Possible interface
==

settings.CONTENT_PARSERS# List of parsers, defaults to 
(contentparsers.FormParser,)

HttpRequest.parsers # property, defaults to initializing from 
settings.CONTENT_PARSERS
HttpRequest.REQUEST *or* .CONTENT   # lazily evalulated, like request.POST

class ContentParser(object)  # Interface for parser objects.
  handles_request(request)
  parse(request)

class FormParser(ContentParser)  # Multipart and urlencoded behaviour, 
probably returns QueryDic

HttpRequest.read() issues

[Tom Christie]

It's not very obvious from the docs or source if HttpRequest.read() can 
always be safely treated as a limited input stream, or if the developer 
needs to respect HttpRequest.META['CONTENT_LENGTH'].

As far as I can tell the intention is that it can always be treated as a 
limited stream, that seems to at least be the implication in 
WSGIRequest.__init__, in which case it looks to me like there are two bugs.

1. This code should not raise an Assertion Error...

>>> from django.test.client import RequestFactory
>>> req=RequestFactory().post('/', {'foo':'bar'})
>>> req.read()
Traceback (most recent call last):
  File "", line 1, in 
  File 
"/Users/tomchristie/workspace/django-rest-framework/env/lib/python2.6/site-packages/django/http/__init__.py",
 
line 296, in read
return self._stream.read(*args, **kwargs)
  File 
"/Users/tomchristie/workspace/django-rest-framework/env/lib/python2.6/site-packages/django/test/client.py",
 
line 51, in read
assert self.__len >= num_bytes, "Cannot read more than the available 
bytes from the HTTP incoming data."
AssertionError: Cannot read more than the available bytes from the HTTP 
incoming data.

After all, running under the dev server I can do this just fine without 
causing an exception:

def test(request):
return HttpResponse("Read data: '%s'\n" % request.read())

(In the first case the underlying stream isn't being wrapped in a 
LimitedStream, in the second case it is)

2. Isn't the use of LimitBytes in MultipartParser.parse() now redundant?

If it isn't the intention that HttpRequest.read() can be treated as a 
limited stream then shouldn't this be documented, and in any case wouldn't 
it be better if it was always a limited stream - there's some 
duplicated behavior with parsing the CONTENT_LENGTH in WSGIRequest, 
HttpRequest and MultipartParser that looks like ti could be avoided.

Am I just fundamentally misunderstanding something?

Cheers,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: HttpRequest.read() issues

[Tom Christie]

I've created two tickets for this, with patches and tests...

http://code.djangoproject.com/ticket/15762 - WSGIRequest should wrap the 
test client wsgi.input in LimitedStream
http://code.djangoproject.com/ticket/15763 - MultiPartParser's LimitBytes is 
now redundant.

It's possible that I've misunderstood and you can't assume that it'll be 
okay to do request.read(BUFFER_SIZE) without respecting CONTENT_LENGTH,
although if that's the case then presumably that's an issue itself?

(Eg. because it means that you can't hand the request over to some arbitrary 
parser that just treats it like any other file-like object)

Cheers,

  t.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: HttpRequest.read() issues

[Tom Christie]

> So, OP should not be trying to read more than CONTENT_LENGTH.

>From the underlying stream, sure.  The question is if it's okay to do on the 
HttpRequest object.  It's an issue because now that HttpRequest exposes a 
file-like interface to the stream some users are likely to do things like 
this:

request_data = json.load(request)

And expect that everything should work fine.

> Django should provide a limited stream to prevent user reading more
> than CONTENT_LENGTH either by returning an empty end sentinel,
> or perhaps if wanted to be pedantic, raise an error.

And it does currently provide a limited stream, in some cases, but for (very 
reasonable) performance reasons, only when it appears neccessary.

> Now they just seem to be duplicates 
> and it's a good idea to ditch LimitBytes (since LimitedStream implements 
> a readline() too) and move it into some common place. http.utils seems a 
> good fit for it.

Moving LimitedStream to somewhere like http.utils sounds like a good plan to 
me.

> Thoughts?

I think HttpRequest definatly needs to ensure that it's safe to treat it as 
any other file-like object, now that it exposes the .read()/.readlines() 
methods, and at the moment it sounds like the behaviour of 
.read(BUFFER_SIZE) past the end of the input is not well defined.

1. It might be reasonable to defer the creation of HttpRequest._stream, but 
to _always_ wrap it in LimitedStream if it is created.  (IE 
HttpRequest._stream is a property that's only initialised when it's 
accessed)  That'd presumably be a performance gain for any code path that 
_doesn't_ access .POST/.raw_post_data/.read, and a performance hit for 
anything that _does_.  In 99% of cases you'll be doing a .read() operation 
without specifying any length at all so the performance hit will be the 
initial creation of the LimitedStream object, but you won't actually have 
subsequent function calls that incur the extra layer of wrapping.

2. Ensure that the .read()/.readline() interfaces will always expose a 
LimitedStream, but avoid creating a limited stream for .POST and 
.raw_post_data by reading directly from the underlying stream and enforcing 
the CONTENT_LENGTH behavior explicitly in those cases.

3. Alter the LimitedStream wrapping behaviour to be more cautious.  The 
current behaviour is to wrap it in LimitedStream if it's known to be 
necessary.It could instead wrap it in LimitedStream _unless_ it's known 
to be _uneccessary_.

Do any of those sound like reasonable options?

I guess the other possibility is:

4. Everything is fine as it is, and in real world cases WSGI servers are 
passing a limited stream on already.

It basically sounds like that's the case, but it's not obvious how much we 
can rely on that.

It might be a case that I need to file a seperate bug for this:

HttpRequest.read(BUFFER_SIZE) behaviour is unclear.

and place a note on the other two that their resolution is dependant on 
this, does that make sense to y'all?

Really appreciate your input on this,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: HttpRequest.read() issues

[Tom Christie]

Actually it occurs to me that (1) shouldn't be a performance hit for 
accessing .POST either, because whilst you're now creating a LimitedStream, 
you're able to drop MultiPartParser's use of LimitBytes.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: HttpRequest.read() issues

[Tom Christie]

It occurs to me that (1) isn't a hit for accessing multipart POST requests, 
since we're wrapping the underlying stream in LimitedStream, but beng able 
to drop MultiPartParser's use of LimitBytes.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: HttpRequest.read() issues

[Tom Christie]

> Where is the proof that using a limited stream is a performance issue? These 
sorts of things are never going to be the bottleneck and sounds a bit like 
premature optimisation to think that wrapping it with a length limiting 
stream is going to be an issue.

There isn't any, so good point. :)

Even so, presumably we wouldn't want WSGIRequest to create a LimitedStream 
object for _every_ single incoming HTTP request, regardless of if it 
actually has a payload or not?

In that case deferring the LimitedStream creation, but always ensuring that 
HttpRequest._stream is a LimitedStream rather than the underlying stream 
would be a sensible thing to do.  It's fairly simple logic, and 
it's guaranteed to be a safe thing to do.

I'd be happy to code up a patch for this, if we reckon that's a sensible 
approach.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: HttpRequest.read() issues

[Tom Christie]

Okay, since there's clearly an underlying issue here I've created a seperate 
ticket for this and marked the other two tickets as duplicates of it.

So...

#15785 - HttpRequest.read(NUM_BYTES) can read beyond the end of wsgi.input 
stream.  (Violation of WSGI spec & under-defined behaviour) [1]

There is a patch (with tests) attached. [2]

* Changes WSGIRequest._stream to be a property that is (always) instantiated 
as a LimitedStream when first accessed.
* Removes some redundant code in HttpRequest and MultiPartParser.
* Fixes some minor bugs in tests/regressiontests/requests/tests.py
* Adds two tests for MultiPartParser to check graceful behaviour on 
truncated or empty multipart requests.
* Adds a test for TestClient request.read(LARGE_BUFFER) behaviour.

This fixes (#15762) without having to alter any test client code, and 
includes the fixes for (#15763)

This looks like a decent way to do things to me - it's less code than 
before, it's safer, and it's easy to understand.

I'd appreciate any eyeballs/kicking the tires etc...

Whaddya reckon?

  t.

[1] http://code.djangoproject.com/ticket/15785
[2] 
http://code.djangoproject.com/attachment/ticket/15785/limited_stream.diff

NB. There's also some behaviour in MultiPartParser that could be adapted 
slightly to support streaming requests with unknown length.
Obv that's mostly entirely pointless right now since WSGI won't support them 
anyway (yet?...)

Right, as far as I understand it PEP doesn't address chunked requests, 
only chunked responses, so I guess this point is moot?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: HttpRequest.read() issues

[Tom Christie]

> Last night I actually did test it :-). You're right the difference in
> performance is less than a statistical deviation between different
> uploads over network.

Nice work.  And point well made Graham!

> Creating a single wrapper object is a negligible hit but the code bloat 
from making
> it lazy is very real. So I'm -1 on laziness.
> Apart from the laziness issue your patch looks fine to me. Thanks!

Great stuff.
New patch added to the ticket, just to keep everything easy.

http://code.djangoproject.com/attachment/ticket/15785/limited_stream_lazy.diff

Coolio, thanks for input peeps...

  t.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Customizable serialization. (Finally?)

[Tom Christie]

Hi all,

  Over the coding sprints at djangocon.eu I started working on a proposal 
for customizable serialization. [1]
I've talked over the API and implementation with Russ, and I think we're 
broadly happy with it all,
so I'd like to open it up to wider discussion.

  What I have right now looks something like this:

1) Serialization is a two stage process - firstly serializing into native 
datatypes, then rendering into json/xml/whatever.
2) Serialization behavior is specified by inheriting from a Serializer class 
and setting various attributes such as 'fields', 'depth' etc... on an inner 
'Meta' class.
3) Serialization of a given field on a model can be overridden by specifying 
a method on the Serialization class.
4) Behavior can also be refined by overriding various public methods on 
the Serializer class, if required.
5) Multiple Serialization classes can be composed together, to provide for 
more complex behavior.

Here's an example of the proposed API, that replicates the existing json 
serialization behavior, without any of the implicit assumptions about 
serialization structure that are currently made:

class PKOnlySerializer(Serializer):
"""
Serializes models into their pk representation.
"""
def serialize_model(self, instance):
return instance.pk


class FieldsSerializer(Serializer):
"""
Serializes only the fields on a model.
"""
class Meta:
exclude = ('id', 'pk')
related_serializer = PKOnlySerializer


class DjangoSerializer(Serializer):
"""
Serializes into the existing django style.
"""
class Meta:
fields = ('pk', 'model', 'fields')

def model(self, instance):
return unicode(instance._meta)

def fields(self, instance):
return FieldsSerializer().serialize(instance)

Which can be used like this:

data = DjangoSerializer().serialize(queryset)
ret = json.dumps(data, cls=DateTimeAwareJSONEncoder, indent=4, 
sort_keys=False)

The code as it stands is available on github [2].
You'll want to look at impl.py [3] for the implementation and views.py [4] 
for example API usage.

The serializer currently supports depth-limited and recursion-limited 
output, and I'm right now I'm looking to work on finalizing the xml related 
issues,
adding some slightly more concise ways of being able to describe nested 
models [Russ: I think this'll be closer to what you were looking for], and 
then
trying to refactor Django to use the new serialization code, and see what 
test breakage I come up against.

Would def welcome any constructive criticism of the API or implementation.

Also if anyone wants to help in contributing in order to get this (long 
outstanding) feature into Django, that would be very much appreciated.
(I suspect there might be quite a lot to do once I look at merging into 
Django!)

It's not totally impossible that we might be able to get this feature in for 
1.4, which'd be pretty cool,
and there's also a bunch of serialization related tickets that we might be 
able to close off in the process. [5]

Feedback?

  t.

[1] 
https://code.djangoproject.com/wiki/SummerOfCode2011#Customizableserialization
[2] https://github.com/tomchristie/scratch
[3] https://github.com/tomchristie/scratch/blob/master/myapp/impl.py
[4] https://github.com/tomchristie/scratch/blob/master/myapp/views.py
[5] 
https://code.djangoproject.com/query?status=assigned&status=new&status=reopened&component=Core+(Serialization)&order=priority

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/ZIVyqCS9JZ0J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: WSGIRequest._stream.read() blocking

[Tom Christie]

> Isn't this the same bug as  
and ?

Yes.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/pKfIoVY78G0J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Customizable serialization. (Finally?)

[Tom Christie]

> A suggestion though, is to be able to declare serializers in a
> similar fashion as models or forms.

Actually that's a fair point, yup - the proposal could be made a little 
closer to the existing Forms and Models APIs.
Something along these lines...

class ModelPKField(SerializerField):
def get_value(self, obj, key):
return unicode(obj.pk)

class ModelNameField(SerializerField):
def get_value(self, obj, key):
return unicode(obj._meta)

class ModelFieldSetField(SerializerField):
def get_value(self, obj, key):
return FieldsSerializer().serialize(obj)

class RelatedPKField(SerializerField):
def get_value(self, obj, key):
return unicode(getattr(obj, key).pk)


class FieldsSerializer(Serializer):
"""
Serializes only the fields on a model.
"""
class Meta:
exclude = ('id', 'pk')
default_related_field = RelatedPKField()


class DjangoSerializer(Serializer):
"""
Serializes into the existing django style.
"""
pk = ModelPKField()
model = ModelNameField()
fields = ModelFieldSetField()


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/wm9YOnddZ3EJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Customizable serialization. (Finally?)

[Tom Christie]

Yup.
The current implementation (see the github repo) allows properties and 
methods that only take a 'self' argument to be serialized by adding the name 
to 'fields' or 'include'.
If you need anything more involved then you'd write a custom method on the 
Serializer class. (or a custom SerializerField if the final API uses that 
style instead.)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/vrH_lLTRbdEJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Customizable serialization. (Finally?)

[Tom Christie]

Cool, thanks, all good points.

Slight haitus working on this right now whilst I try to get my head around 
the right way to try to bring it more into line with the Forms API, but 
hoping to pick it up again once I've marshaled my thoughts.

Obviously I'll keep the list updated if I do make more headway, and if 
anyone's thinking about contributing to this / sprinting at DjangoCon US or 
whatever, then please do give me a shout.

And, slightly OT, but...

> Didn't you just redefine the 'in' operator?

Not quite:

>>> a={}
>>> b={}
>>> collection=[a]
>>> b in collection
True
>>> any([b is elem for elem in collection])
False



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/q1m8XCXnKXAJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re-opening 5617?

[Tom Christie]

Hi All,

  I recently bumped into the same issue as reported in 
#5617.  
(And duplicate #6377 .)
IE: Django's default 500 view uses a Context, not a RequestContext, which 
means any 500 templates which uses MEDIA_URL or suchlike won't work.

I thought I'd re-punt the suggestion given in one of the comments - attempt 
to render using the RequestContext, and only fall back to the Context if 
that fails.  It seems like a reasonable approach, but it looks like it's 
something that didn't ever get properly considered, because no-one pushed 
the issue to the list, as far as I can tell.

Where do we stand on this?  I'd have thought that a large amount of installs 
are having to define their own 500 view function, which does nothing more 
than the default view, other than to use a RequestContext so that the 
template can render correctly.  Would there be any value in re-opening the 
ticket given the suggested fallback approach?

Cheers,

  Tom.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/hBlPW8dRGI0J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Re-opening 5617?

[Tom Christie]

Heya,

  Thanks for the feedback.  I quite like the explicit 'STATIC_URL' only 
approach, although I think a lot of users would still run into a problem 
there, because 'request' isn't also added in explicitly to the Context...

  For context, my particular use case is a simple '500.html' template, that 
extends a 'base.html' template.  I don't use any other context in the base 
template other than 'request' and 'STATIC_URL'.  In the case of a 500 error, 
I'd see the template render correctly, except that it'd look like the user 
isn't logged in.  Coming across that as a dev that'd confuse the hell out of 
me the first time I came across it unless I already understood the 500 
Context behavior, and it's not ideal from the end-user perspective either.

  I'd imagine that plenty of other setups would have a similar setup, so you 
could argue that returning this:

Context({'STATIC_URL': settings.STATIC_URL, 'request': request})

would be an okay thing to do in the default 500 handler.

  Personally I think that'd probably be absolutely fine (and the most 
sensible default 500 view), although the obvious counter argument is that 
that's getting into the realms of special-case, rather than default-case. 
 (That's not my opinion, but it'd be a valid argument.)

  What do you reckon?

Cheers,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/Ac1EeU-n6xAJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Re-opening 5617?

[Tom Christie]

Hey Jannis,

> https://docs.djangoproject.com/en/dev/releases/1.4/#static-template-tag

That's rather nice.

> Adding the request is a non-starter, IMO; the "request" context
> processor isn't even in the default list of TEMPLATE_CONTEXT_PROCESSORS,
> so this would mean adding something to the 500 page context that isn't
> even added by RequestContext by default. If you want the request object
> in your 500 page, you're well into custom handler500 territory.

Yeah, that's me being stupid!  I should really have been talking about 
having 'user' available, given that 
'django.contrib.auth.context_processors.auth' _is_ enabled in 
TEMPLATE_CONTEXT_PROCESSORS.

So you _could_ argue that returning 'Context({'user': request.user})' would 
be a more useful default than the current empty context.

However I think I'm convinced by Carls point that attempting to do things 
with 'user' inside a 500 view might not be best practice in any case.

Thanks for the input both.  I've linked from the bugs to here.  Should be 
nice and clear for anyone hitting them again in the future.

I'm considering making the doc's on the 500 
viewmore
 obviously explicit about the use of Context, not RequestContext, as 
it's a bit of a gotcha, and reasonably easy to miss.  IE, putting the text 
"...is 
rendered with an empty Context..." into a Note block to make it more 
obvious.  I'll submit a separate bug and patch for that when I've got a 
moment.

Cheers,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/j2p7-vE7ZM0J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django 1.4 roadmap

[Tom Christie]

>
>  > Since we never had a list of features that we were committing to adding

> for 1.4, I don't see any reason why we can't release an alpha as soon as
> > the release blockers are dealt with.

 

Agreed. There will 
> always be extra features that we can add (app
> refactor, anyone?) but I think we're well past the point where an
> alpha is called for.


Great stuff people, super exciting!

I don't know how quickly the alpha might be do-able, but from my point of 
view it'd be good if there's a little bit of notice to give devs a chance 
to close off any new feature tickets they might be working on prior to the 
feature-freeze deadline.

Got at least a couple in mind (eg. defiantly 
17193, 
and perhaps 2879 ) that I'd 
like to see get to RFC if possible, but it'd be useful to know how much or 
little headroom I'd have before 1.4.

Might make sense to fix a planned alpha date?

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/vXVQZi1aJSEJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

HttpRequest.read(), HttpRequest.body, HttpRequest.raw_post_data

[Tom Christie]

I see that as of 1.4 `HttpRequest.raw_post_data` is being marked as 
"pending deprecation", and `HttpRequest.body` has been introduced.
This still leaves us with two ways to get at the request body - `.body` and 
`.read()`

If we're going to make the change of marking `raw_post_data` as deprecated,
wouldn't it make sense to only have a single interface to getting the 
request body,
rather than two different interfaces which behave in subtly different ways?

For example, we could also mark '.read()' as pending deprecation, and just 
have 'body' be a stream-like object.

I guess this'll fall into the "we can't do this now that we're in 1.4 beta" 
category,
but wanted to mention it because it won't really be viable to change the 
semantics of `.body` once 1.4 hits release,
and getting this right the first time around would be really nice.

Any thoughts?

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/RTb6HJPMFbgJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: HttpRequest.read(), HttpRequest.body, HttpRequest.raw_post_data

[Tom Christie]

> a design decision was made long ago that HttpRequest objects should 
provide a file-like interface (thus also .readline(), .readlines(), and 
.xreadlines())

Wouldn't having .read() .readline(), .readlines(), and .xreadlines() all on 
`request.body` provide a slightly cleaner interface, whilst still staying 
in line with that design decision?

I might not normally be fussed about pushing for a core change like this, 
but it seems like if we're introducing `.body`, keeping things nice and 
tidy would be worthwhile.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/OadNl5ChOP8J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Making sure Web APIs built with Django don't have CSRF vulnerabilities.

[Tom Christie]

Hi all,

  This is a follow on to an issue that's been discussed on django-security. 
 It's been suggested that it should be raised in this forum instead, so...

Most of the time when building Web APIs with Django the right thing to do 
is to wrap API views in @csrf_exempt.  Generally Web APIs shouldn't have 
CSRF, since API clients authenticate their requests explicitly, and don't 
need (or have) a CSRF token available to them.

That's the right thing to do except for the case where you're building a 
Web API that is consumed by an AJAX client of the same site.  In that case 
the AJAX call happens within the context of the current session, and the 
authentication is implicit in the request.  In the AJAX-client case, 
developers should be CSRF protecting their views, and setting the '
X-CSRFToken ' 
header in their AJAX requests.

The problem comes when developers start using AJAX clients with API 
frameworks that CSRF exempt their views by default, and don't realize that 
they ought to be using CSRF protected views.  I can see at least a 
coupleof
 
examples
 of 
developers using AJAX clients in this way, and it seems to me that there's 
probably a bunch of Django sites out there at the moment that are exposing 
their Web APIs in a way that is vulnerable to CSRF attacks.

I think the ways we could approach this come down to one of these options:

1. It's a problem and something we should address in Django core.
2. It's a problem and something we should address in API frameworks.
3. It's a problem, and the best way to address it is to make sure 
developers understand that session authenticated APIs ought to have CSRF 
protection.
4. It's not a problem - if you're using session authenticated APIs and 
doing it wrong, tough luck.

It arguably isn't an issue with Django core (which does give you all the 
tools you need to be explicit about if and when CSRF protection runs.) 
or arguably in the API frameworks either (For example neither piston nor 
tastypie provide session based authentication by default), but does seem 
like a reasonably big and non-obvious pitfall.

One idea to mitigating this in Django core that I've considered would be 
introducing a '@csrf_defered' decorator, that would act like 
'@csrf_exempt', but wrap request.session in a lazily evaluated 
'@csrf_protect'.  That'd give developers an easy way to generically do 
"Don't CSRF protect this view unless/until it accesses session 
information.".  API frameworks could then use '@csrf_defered' instead of 
'@csrf_exempt', and know that they'll get the correct behavior whether the 
API uses session authentication or not.

I'm fairly agnostic on what approach we end up taking, but I wanted to at 
least raise the issue and open it to discussion.

Regards,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/s46Tlqxh-9sJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Making sure Web APIs built with Django don't have CSRF vulnerabilities.

[Tom Christie]

I don't know how much of an issue it really is (or not), but I haven't 
really seen it being done right.
Of all the examples I've found of devs implementing session authentication 
on top of piston and tastypie, (See 
here,
 
hereand
 
here)
 
none have re-enabled CSRF protection. (As far as I can tell.)

Having said that, I really just wanted to:

1. Get the core devs opinion on if would ought to actively do anything 
about this in core, since it's pretty easy to shoot yourself in the foot.
2. Raise the issue, so that there's at least more awareness that devs 
really should be making sure that any same-site AJAX driven APIs *do* get 
CSRF protection. [*]

I fall into the camp of "you should understand what you're doing when
> you turn CSRF protection off". If the framework authors want to
> support session based authentication, I believe they're capable of
> doing it correctly. Until then, if users want to hack session based
> auth onto the frameworks, they should be careful and understand what
> they're doing.


I'm happy with that.

Thanks for the feedback,

  Tom

[*] Only relevant to APIs that support POST/PUT or DELETE operations of 
course.  Read-only APIs will be just fine.

On Saturday, 17 March 2012 03:24:04 UTC, Paul McMillan wrote:
>
> >> One idea to mitigating this in Django core that I've considered would be
> >> introducing a '@csrf_defered' decorator
>
> > Practically speaking, I think this might be ok and would cover the
> > majority of real cases. But at the very least it means that this
> > decorator should live in contrib.sessions, not in the core CSRF code.
>
> I would be opposed to this code in any shipped part of Django. It
> certainly could be built as a third party module (if we don't have the
> hooks necessary to do this, we can discuss them). My main objection is
> that CSRF is not a topic which should be deferred, or maybe on, or
> anything except absolutely positively explicitly on or off.
> Introducing a deferred format encourages developers to ignore it,
> until it causes problems, at which point they will do exactly the same
> thing as they do now, and turn it off. It's easy enough to screw up
> already - adding a "maybe on" is going to bite developers even more
> than the current format, since it will be even easier to write code
> that works most of the time, for most users, but not all of the time,
> for all users.
>
> I fall into the camp of "you should understand what you're doing when
> you turn CSRF protection off". If the framework authors want to
> support session based authentication, I believe they're capable of
> doing it correctly. Until then, if users want to hack session based
> auth onto the frameworks, they should be careful and understand what
> they're doing.
>
> For readers who have not inspected it yet, Django's CSRF
> implementation is quite instructive:
>
> https://code.djangoproject.com/browser/django/trunk/django/middleware/csrf.py
>
> -Paul
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/Dq4aUlP24nAJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: [GSOC 2012] Customizable serialization

[Tom Christie]

Hi Piotr,

  I'd really like to see something along these lines making it into Django.
I worked on this during the 2011 DjangoCon.eu sprints, which I posted about 
a while 
back
.
That work made it into Django REST framework's 
serializer,
 
but never back into Django,
mostly due to never being fully satisfied with a corresponding 
deserialization API.
(REST framework uses forms for deserialization, but that gets awkward for 
deserializing nested models.)
I've been meaning to follow up on my post for a while now.

As regards your proposal

In general...

  I'm really pleased to see that you've taken on board the previous 
comments and opted for a 2-stage approach - I think it's absolutely the 
right thing to do.
It breaks the problem down into two very well defined tasks.

1. Convert native datatypes to/from data streams.  (eg. Tastypie's 
serializers,
 
REST framework's 
renderers&
 
parsers
)
2. Convert complex objects to/from native datatypes. (eg. Tastypie's 
hydrate/dehydate, REST frameworks 
serializer
, form based 
deserialization
)

In your proposal you've understandably addressed model serialization in 
detail, but I (personally) think it'd be worthwhile breaking things down a 
little bit more.
For example, you might consider the following:

1. Define a base API for part (1).  Is it handled by a single class with 
instance methods for converting in each direction, or by 
two separate classes?
What about formats that are one direction only? (eg. You might want to 
parse HTTP form data, but you're unlikely to want to render it.)
Are formats that are not fully reversible an issue? (eg. Some formats 
[eg csv?] might not have native representations of all the python 
datatypes, and only have string representations.)
Exactly what subset of types do we consider 'native'  - should it be 
strictly python native datatypes, or would we use the set of types covered 
by 'django.utils.encoding.is_protected_type'?
2. Define a base API for the components of part (2) that doesn't include 
the implementation or related specifically to Django models/querysets.
Are the serialization and deserialization handled by the same class? 
 What about serializations that aren't reversible?  [eg only include a 
subset of information]
3. Consider given your general API what interface you'd need for a 
serializer that worked with arbitrary python objects.
4. Given the API for an arbitrary object serializer what else do you need 
to provide in the API to deal with querysets/model instances specifically? 
5. Are object fields handled with a subclass of the Serializer class, or do 
they need a different API?  If they're a subclass what extra information do 
they need?
6. When looking at deserialization is it worth considering mirroring any 
aspects of the form APIs?  How do you treat deserialization errors?
7. Is the deserializer creating saved or unsaved model instances?  How do 
you handle saving the instances, and how do you deal with deserializing 
data where some parts of the object might be implicit?  (Eg deserializing 
data from an API request, where the pk of the model is given in the URL of 
the request?)

If you break it right down like that I think it'd help make sure you get 
the fundamentals right.
I'd expect to see some base classes without implementation, and 
serialization of Django objects tackled purely as a subset of the general 
case.

Some things I was slightly confused by in your proposal as it stands...

* JSONSerializer subclassing Serializer. (and XMLSerializer subclassing 
JSONSerializer.)
  I'd have thought that if you're using a two phase approach you'd keep the 
complex<->native and native<->data stream APIs totally decoupled.
  JSON serialization doesn't itself have anything to do with how you choose 
to structure the serialization of a complex object, so why should it 
subclass the implementation of that part?
* "dehydrate__value__", "hydrate__value__" - what's with the double 
underscore naming?
* I don't get the '@attribute' decorator.  I think it'd be simpler to just 
have 'fields', 'include', 'exclude'.  If fields is None then use the 
'default set of fields' + 'include' - 'exclude'.  If fields is not None, 
use that and ignore include/exclude.
* I wouldn't consider specia

More on Customizable Serialization

[Tom Christie]

Seeing another proposal for Customizable Serialization for the GSoC this 
year
prompted me to dust off the bits of work I've done along similar lines.
I'd really like to see this get properly addressed in core and I thought it
was about time I helped to make it happen.

I've made a fairly comprehensive start, and pushed the results of what I 
have
to date as the `django-serializers` project.  It's available on PyPI, and 
the
source is here:
http://github.com/tomchristie/django-serializers

There are some docs up on the GitHub site, but in brief it gives you:

* A Declarative Serializer/Field API that mirrors the Form/Field API.
* A Serializer class which can deal with arbitrary python objects.
* A ModelSerializer class which can handle model instances and query sets.
* A DumpDataSerializer class which mimics the existing dumpdata behaviour.
* Supports flat or nested ouput, and deals with recursive structures.
* Handles FK, M2M, OneToOne, and reverse relationships, model fields, and 
non-database attributes/properties.
* Serialization structure decoupled from output encoding concerns.
* Currently supports JSON, YAML, and XML.
* Decent test suite of the above.

It's not yet at the point where it could be a proposal for core - Right now 
the
API isn't backwards compatible with the existing serializers, the dump
data serializer is still missing some tweaks such as dealing with natural 
keys,
and there's some work to do on the XML output.  It is a pretty comprehensive
start though, and I'm really happy with the shape of the API.

Given that Piotr's GSoC proposal has now been accepted, I'm wondering what 
the
right way forward is?  I'd like to continue to push forward with this, but 
I'm
also aware that it might be a bit of an issue if there's already an ongoing
GSoC project along the same lines?

Having taken a good look through the GSoC proposal, it looks good, and there
seems to be a fair bit of overlap, so hopefully he'll find what I've done
useful, and I'm sure I'll have plenty of comments on his project as it
progresses.

I'd consider suggesting a collaborative approach, but the rules of the GSoC
wouldn't allow that right?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/RtFUTCF1iPUJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Customizable Serialization check-in

[Tom Christie]

Hey Piotr,

  Thanks for the quick response.

> However sharing ideas and discuss how the API should look and work it 
will be very desirable.

That'd be great, yup.  I've got a couple of comments and questions about 
bits of the API, but I'll wait until you've had a chance to post your 
proposal to the list before starting that discussion. 

> I quickly skimmed the proposal and I noticed speed/performance wasn't 
mentioned. I believe performance is important in serialization and 
especially in deserialization.

Right.  Also worth considering is making sure the API can deal with 
streaming large querysets,
rather than loading all the data into memory at once.
(See also https://code.djangoproject.com/ticket/5423)

- Tom.

On Friday, 27 April 2012 10:11:56 UTC+1, Piotr Grabowski wrote:
>
> W dniu 27.04.2012 10:36, Anssi K��ri�inen pisze: 
> > On Apr 27, 11:14 am, Piotr Grabowski  wrote: 
> >> Hi! 
> >> 
> >> I'm Piotr Grabowski, student from University of Wroclaw, Poland 
> >> In this Google Summer of Code I will  deal with problem of customizable 
> >> serialization in Django. 
> >> 
> >> You can find my proposal here https://gist.github.com/2319638 
> > I quickly skimmed the proposal and I noticed speed/performance wasn't 
> > mentioned. I believe performance is important in serialization and 
> > especially in deserialization. It is not the number one priority item, 
> > but it might be worth it to write a couple of benchmarks (preferably 
> > to djangobench [1]) and check that there are no big regressions 
> > introduced by your work. If somebody already has good real-life 
> > testcases available, please share them... 
> > 
> >   - Anssi 
> > 
> > [1] https://github.com/jacobian/djangobench/ 
> > 
> I didn't think about performance a lot. There will be regressions. 
> Now serialization is very simple: Iterate over fields, transform it into 
> string (or somethink serializable), serialize it with json|yaml|xml. 
> In my approach it is: transform (Model) object to Serializer object, 
> each field from original object is  FieldSerializer object, next  (maybe 
> recursively) get native python type object from each field, serialize it 
> with json|yaml|xml. 
> I can do some optimalizations in this process but it's clear it will 
> take longer to serialize (and deserialize) object then now. It can be 
> problem with time taken by tests if there is a lot of fixtures. 
> I will try to write good, fast code but I will be very glad if someone 
> give me tips about performance bottlenecks in it. 
>
> -- 
> Piotr Grabowski 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/K9cslx5Fa_sJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: More on Customizable Serialization

[Tom Christie]

> If you've got the time and enthusiasm, I'd certainly appreciate you 
hanging around in a "co-mentor"-like capacity.

Sure, I'd be more than happy to help out wherever possible.
Piotr, if you need to get in touch with me off-list for any reason please 
feel free to do so - t...@tomchristie.com

  - Tom

On Saturday, 28 April 2012 04:05:17 UTC+1, Russell Keith-Magee wrote:
>
> Hi Tom, 
>
> On Friday, 27 April 2012 at 12:44 PM, Tom Christie wrote: 
> > Seeing another proposal for Customizable Serialization for the GSoC this 
> year 
> > prompted me to dust off the bits of work I've done along similar lines. 
> > I'd really like to see this get properly addressed in core and I thought 
> it 
> > was about time I helped to make it happen. 
> > 
> > I've made a fairly comprehensive start, and pushed the results of what I 
> have 
> > to date as the `django-serializers` project. It's available on PyPI, and 
> the 
> > source is here: 
> > http://github.com/tomchristie/django-serializers 
> > 
> > There are some docs up on the GitHub site, but in brief it gives you: 
> > 
> > * A Declarative Serializer/Field API that mirrors the Form/Field API. 
> > * A Serializer class which can deal with arbitrary python objects. 
> > * A ModelSerializer class which can handle model instances and query 
> sets. 
> > * A DumpDataSerializer class which mimics the existing dumpdata 
> behaviour. 
> > * Supports flat or nested ouput, and deals with recursive structures. 
> > * Handles FK, M2M, OneToOne, and reverse relationships, model fields, 
> and non-database attributes/properties. 
> > * Serialization structure decoupled from output encoding concerns. 
> > * Currently supports JSON, YAML, and XML. 
> > * Decent test suite of the above. 
> > 
> > It's not yet at the point where it could be a proposal for core - Right 
> now the 
> > API isn't backwards compatible with the existing serializers, the dump 
> > data serializer is still missing some tweaks such as dealing with 
> natural keys, 
> > and there's some work to do on the XML output. It is a pretty 
> comprehensive 
> > start though, and I'm really happy with the shape of the API. 
>
> Thanks for letting us know about the prior art -- I know we discussed 
> serialisation at DjangoCon last year, and I'm kicking myself that I didn't 
> provide more feedback at the time. Hopefully having this as a GSoC project 
> will be enough to kick me into action and bring this project to completion. 
> > 
> > Given that Piotr's GSoC proposal has now been accepted, I'm wondering 
> what the 
> > right way forward is? I'd like to continue to push forward with this, 
> but I'm 
> > also aware that it might be a bit of an issue if there's already an 
> ongoing 
> > GSoC project along the same lines? 
> > 
> > Having taken a good look through the GSoC proposal, it looks good, and 
> there 
> > seems to be a fair bit of overlap, so hopefully he'll find what I've 
> done 
> > useful, and I'm sure I'll have plenty of comments on his project as it 
> > progresses. 
> > 
> > I'd consider suggesting a collaborative approach, but the rules of the 
> GSoC 
> > wouldn't allow that right? 
>
> Unfortunately, the GSoC rules don't allow for collaboration - the work 
> that is submitted needs to be that of the student alone. However, they do 
> allow for others to contribute by providing code reviews, design feedback, 
> and so on. Given that we're building a user-facing API, there's also plenty 
> of room to provide assistance by testing -- i.e., hunting down obscure 
> serialisation use cases, and making sure that the API we've got can cover 
> them. 
>
> If you've got the time and enthusiasm, I'd certainly appreciate you 
> hanging around in a "co-mentor"-like capacity. You've clearly spent a lot 
> of time thinking about on this problem, and I'm sure your input would be 
> extremely valuable. You're also in a slightly more helpful timezone for 
> Piotr, so if he needs some feedback when I'm not available, it would be 
> nice to have someone he can call on that is familiar with the problem and 
> his progress. 
>
> Yours, 
> Russ Magee %-) 
>
>   
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/kiJQ2dvvhlUJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Customizable Serialization check-in

[Tom Christie]

Hey Piotr,

Here's a few comments...

You have 'fields' and 'exclude' option, but it feels like it's missing an 
'include' option - How would you represent serializing all the fields on a 
model instance (without replicating them), and additionally including one 
other field?  I see that you could do that by explicitly adding a Field 
declaration, but 'include' would seem like an obvious addition to the other 
two options.  

I'd second Russell's comment about aliases.  Defining a label on the field 
would seem more tidy.

Likewise the comment about 'preserve_field_order'  I've still got this in 
for 'django-serializers' at the moment, but I think it's something that 
should become private.  (At an implementation level it's still needed, in 
order to make sure you can exactly preserve the field ordering for json and 
yaml dumpdata, which is unsorted (determined by pythons dict key ordered). 

Being able to nest serializers inside other serializers makes sense, but I 
don't understand why you need to be able to nest fields inside fields. 
 Shouldn't serializers be used to represent complex outputs and fields be 
used to represent flat outputs?

When defining custom fields it'd be good if there was a way of overriding 
the serialization that's independent of how the field is retrieved from the 
model.  For example, with model relation fields, you'd like to be able to 
subclass between representing as a natural key, representing as a url, 
representing as a string name etc... without having to replicate all the 
logic that handles the differences between relationship, multiple 
relationships, and reverse relationships.

The "class_name" option for deserialization is making too many assumptions. 
 The class that's being deserialized may not be present in the data - for 
example if you're building an API, the class that's being deserialized 
might depend on the URL that the data is being sent too. eg 
"http://example.com/api/my-model/12";

In your dump data serializer, how do you distinguish that the 'fields' 
field is the entire object being serialized rather than the 'fields' 
attribute of the object being serialized?  Also, the existing dumpdata 
serialization only serializes local fields on the model - if you're using 
multi-table inheritance only the child's fields will be serialized, so 
you'll need some way of handling that.

Your PKFlatField implementation will need to be a bit more complex in order 
to handle eg many to many relationships.  Also, you'll want to make sure 
you're accessing the pk's from the model without causing another database 
lookup.

Is there a particular reason you've chosen to drop 'depth' from the API? 
 Wouldn't it sometimes be useful to specify the depth you want to serialize 
to?

There's two approaches you can take to declaring the 'xml' format for 
dumpdata, given that it doesn't map nicely to the json and yaml formats. 
 One is to define a custom serializer (as you've done), the other is to 
keep the serializer the same and define a custom renderer (or encoder, or 
whatever you want to call the second stage).  Of the two, I think that the 
second is probably a simpler cleaner approach.
When you come to writing a dumpdata serializer, you'll find that there's 
quite a few corner cases that you'll need to deal with in order to maintain 
full byte-for-byte backwards compatibility, including how natural keys are 
serialized, how many to many relationships are encoded, how None is handled 
for different types, down to making sure you preserve the correct field 
ordering across each of json/yaml/xml.  I *think* that getting the details 
of all of those will end up being awkward to express using your current 
approach.
The second approach would be to a dict-like format, that can easily be 
encoded into json or yaml, but that can also include metadata specific to 
particular encodings such as xml (or perhaps, say, html).  You'd have a 
generic xml renderer, that handles encoding into fields and attributes in a 
fairly obvious way, and a dumpdata-specific renderer, that handles the odd 
edge cases that the dumpdata xml format requires.  The dumpdata-specific 
renderer would use the same intermediate data that's used for json and yaml.

I hope all of that makes sense, let me know if I've not explained myself 
very well anywhere.

Regards,

  Tom

On Friday, 4 May 2012 21:08:14 UTC+1, Piotr Grabowski wrote:
>
> Hi, 
>
> During this week I have a lot of work so I didn't manage to present my 
> revised proposal in Monday like i said. Sorry. I have it now: 
> https://gist.github.com/2597306 
>
> Next week I hope there will be some discussion about my proposal. I will 
> also think how it should be done under the hood. There should be some 
> internal API. I should also resolve one Django ticket. I think about 
> this https://code.djangoproject.com/ticket/9279 There will be good for 
> test cases in my future solution. 
>
> I should write my proposal on this group? In github I have nice 

Customizable serialization now passing Django's test suite.

[Tom Christie]

I've been working away on 
django-serializerslately, 
and I've now got a
customizable serialization API that's backwards compatible with the existing
serializers, and is passing Django's serializer tests.

Exactly where I take this will depend on how Piotr Grabowski's GSoC project
progresses, but I'd certainly appreciate any input regarding the API design,
or if there would be any obvious blockers that'd prevent something along 
these
lines making it's way in as a replacement to the existing serializers.

This work intentionally doesn't (yet) address customizable deserialization,
and as such it currently only supports the existing loaddata 
deserialization.


Running the existing tests with django-serializers
==

* Clone Django

* Install django-serializers

* Replace the existing dumpdata serializers with django-serializer's 
versions

In `django/core/serializers/__init__.py`, replace these strings:

"django.core.serializers.xml_serializer",
"django.core.serializers.python",
"django.core.serializers.json",
"django.core.serializers.pyyaml"

With these:

"serializers.dumpdata_xml",
"serializers.dumpdata_python",
"serializers.dumpdata_json",
"serializers.dumpdata_yaml"

* Run the serializer tests from Django's 'tests' directory

./runtests.py --settings=test_sqlite serializers serializers_regress


Example use-case


I've written a small app 
`django-auto-api` 
that uses `django-serializers`
to generate a read-only hyperlinked API for all installed models with 
`html`,
`json`, `yaml`, `xml` and `csv` outputs.

It demonstrates using custom relational fields between models to use
hyperlinking relationships rather than primary key relationships, and shows
how the customizable serialization can be used to build Web APIs with
very little code.


Summary
===

As mentioned I'd appreciate any thoughts on what else it might take to 
bridge
the gap of getting this to the point where it could be adopted into core.

I think it should also serve as a decent point of reference for Piotr's 
GSoC project, in
particular `django-auto-api` is a good concrete use-case that I'd like any
customizable serialization proposal to be able to handle.


 - Tom

[1] django-serializers: https://github.com/tomchristie/django-serializers
[2] django-auto-api: https://github.com/tomchristie/django-auto-api

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/TI2XBLY4FK8J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Customizable Serialization check-in

[Tom Christie]

> if I put list in input I want list in output, not generator

I wouldn't worry about that.  The input and output should be *comparable*, 
but it doesn't mean they should be *identical*.
A couple of cases for example:

*) You should be able to pass both lists and generator expressions to a 
given serializer, but they'll end up with the same representation - there's 
no way to distinguish between the two cases and deserialize accordingly. 
*) Assuming you're going to maintain backwards compatibility, model 
instances will be deserialized into 
django.core.serializer.DeserializedObject instances, rather than 
deserializing directly back into complete model instances.  These match up 
with the original serialized instances, but they are not identical objects. 

> deserialized_value function with empty content

Are you asking about how to be able to differentiate between a field that 
deserializes to `None`, and a field that doesn't deserialize a value at 
all?  I'd suggest that the deserialization hook for a field needs to take 
eg. the dictionary that the value should be deserialized into, then it can 
determine which key to deserialize the field into, or simply 'pass' if it 
doesn't deserialize a value.

> I changed python datatype format returned from serializer.serialize 
method.  Now it is tuple (native, attributes)

I'm not very keen on either this, or on the way that attributes are 
represented as fields.
To me this looks like taking the particular requirements of serializing to 
xml, and baking them deep into the API, rather than treating them as a 
special case, and dealing with them in a more decoupled and extensible way.

For example, I'd rather see an optional method `attributes` on the `Field` 
class that returns a dictionary of attributes.  You'd then make sure that 
when you serialize into the native python datatypes prior to rendering, you 
also have some way of passing through the original Field instances to the 
renderer in order to provide any additional metadata that might be required 
in rendering the basic structure.

Wiring up things this way around lets you support other formats that have 
extra information attached to the basic structure of the data.  As an 
example use-case - In addition to json, yaml and xml, a developer might 
also want to be able to serialize to say, a tabular HTML output.  In order 
to do this they might need to be able attach template_name or widget 
information to a field, that'd only be used if rendering to HTML.

It might be that it's a bit late in the day for API changes like that, but 
hopefully it at least makes clear why I think that treating XML attributes 
as anything other than a special case isn't quite the right thing to do.  - 
Just my personal opinion of course :)

Regards,

  Tom


On Tuesday, 19 June 2012 21:48:37 UTC+1, Piotr Grabowski wrote:
>
>  Hi!
>
> This week I wrote simple serialization and deserialization for json format 
> so it's possible now to encode objects from and to json:
>
>
> import django.core.serializers as s
>
> class Foo(object):
> ��� def __init__(self):
> ������� self.bar = [Bar(), Bar(), Bar()]
> ������� self.x = "X"
>
> class Bar(object):
> ��� def __init__(self):
> ������� self.six = 6
>
> class MyField2(s.Field):
> ��� def deserialized_value(self, obj, instance,� field_name):
> ������� pass
>
> class MyField(s.Field):
> ��� x = MyField2(label="my_attribute", attribute=True)
>
> ��� def serialized_value(self, obj, field_name):
> ������� return getattr(obj, field_name, "No field like this")
>
> ��� def deserialized_value(self, obj, instance,� field_name):
> ������� pass
>
> class BarSerializer(s.ObjectSerializer):
> ��� class Meta:
> ������� class_name = Bar
>
> class FooSerializer(s.ObjectSerializer):
> ��� my_field=MyField(label="MYFIELD")
> ��� bar = BarSerializer()
> ��� class Meta:
> ������� class_name = Foo
>
>
> foos = [Foo(), Foo(), Foo()]
> ser = s.serialize('json', foos, serializer=FooSerializer, indent=4)
> new_foos = s.deserialize('json', ser, deserializer=FooSerializer)
>
>
> There are cases that I don't like:
>
>- deserialized_value function with empty content - what to do with 
>fields that we don't want to deserialize. Should be better way to handle 
>this, 
>- I put list foos but return generator new_foos, also bar in Foo 
>object is generator, not list like in input. Generators are better for 
>performance but if I put list in input I want list in output, not 
>generator. I don't know what to do with this. 
>
>
> Next week I will handle rest of issues that I mentioned in my last week 
> check-in and refactor json format (de)serialization - usage of streams and 
> proper parameters handling (like indent, etc.)
>  
> --
> Piotr Grabowski
>  
>
>
> 

-- 
You received this message because you are subscribe

Re: Customizable Serialization check-in

[Tom Christie]

> default deserialized_value don't returns anything. It sets the field 
value.

Cool, that's exactly what I meant.

> but declaring function only to say "do nothing" isn't good solution for 
me.

 Declaring a method to simply 'pass' seems fine to me if you want to 
override it to do nothing.

> It is the way I am doing deserialization - pass instance to subfields

Seems fine.  It's worth keeping in mind that there's two ways around of 
doing this.

1. Create an empty instance first, then populate it with the field values 
in turn.
2. Populate a dictionary with the field values first, and then create an 
instance using those values.

The current deserialization does something closer to the second.
I don't know if there's any issues with doing things the other way around, 
but you'll want to consider which makes more sense.

> Where I returned (native, attributes) I will return (native, metainfo). 
It's only matter of renaming but metainfo will be more than attributes.

Again, there's two main ways around I can think of for populating metadata 
such as xml attributes.

1. Return the metadata upfront to the renderer.
2. Include some way for the renderer to get whatever metadata it needs at 
the point it's needed.

This is one point where what I'm doing in django-serializers differs from 
your work, in that rather than return extra metadata upfront, the 
serializers return a dictionary-like object (that e.g. can be directly 
serialized to json or yaml), that also includes a way of returning the 
fields for each key (so that e.g. the xml renderer can call 
field.attributes() when it's rendering each field.)

Again, you might decide that (1) makes more sense, but it's worth 
considering.

As ever, if there's any of this you'd like to talk over off-list, feel free 
to drop me a mail - t...@tomchristie.com

Regards,

  Tom

On Wednesday, 20 June 2012 16:28:51 UTC+1, Piotr Grabowski wrote:
>
>  W dniu 20.06.2012 13:50, Tom Christie pisze:
>  
>
> > deserialized_value function with empty content
>
> Are you asking about how to be able to differentiate between a field that 
> deserializes to `None`, and a field that doesn't deserialize a value at 
> all? 
>
> No :) I had this problem before and I managed to resolve it - default 
> deserialized_value don't returns anything. It sets the field value.
> def deserialized_value(self, obj, instance, field_name):
> setattr(instance, field_name, obj)
>
> It is the way I am doing deserialization - pass instance to subfields, 
> retrieve it from them (should be same instance, but in specific cases eg. 
> immutable instance, I can imagine that another instance of same class is 
> returned)  and return it.
>
> If I don't declare deserialized_value function then function from base 
> class is taken. It's expected behavior. So how to say "This field shouldn't 
> be deserialized".  Now I declare:
> def deserialized_value(self, obj, instance, field_name):
> pass
> For true, I can do anything in this function excepting set some value to 
> instance, but declaring function only to say "do nothing"  isn't good 
> solution for me.
>
>  
> > I changed python datatype format returned from serializer.serialize 
> method.  Now it is tuple (native, attributes)
>
>  I'm not very keen on either this, or on the way that attributes are 
> represented as fields.
> To me this looks like taking the particular requirements of serializing to 
> xml, and baking them deep into the API, rather than treating them as a 
> special case, and dealing with them in a more decoupled and extensible way.
>
>  For example, I'd rather see an optional method `attributes` on the 
> `Field` class that returns a dictionary of attributes.  You'd then make 
> sure that when you serialize into the native python datatypes prior to 
> rendering, you also have some way of passing through the original Field 
> instances to the renderer in order to provide any additional metadata that 
> might be required in rendering the basic structure.
>
>  Wiring up things this way around lets you support other formats that 
> have extra information attached to the basic structure of the data.  As an 
> example use-case - In addition to json, yaml and xml, a developer might 
> also want to be able to serialize to say, a tabular HTML output.  In order 
> to do this they might need to be able attach template_name or widget 
> information to a field, that'd only be used if rendering to HTML.
>
>  It might be that it's a bit late in the day for API changes like that, 
> but hopefully it at least makes clear why I think that treating XML 
> attributes as anything ot

Re: Customizable Serialization check-in

[Tom Christie]

Thanks Piotr,

  It's been interesting and helpful watching your progress on this project.
I wouldn't worry too much about not quite meeting all the goals you'd hoped 
for - it's a deceptively difficult task.
In particular it's really difficult trying to maintain full backwards 
comparability with the existing fixture serialization implementation,
whilst also totally redesigning the API to support the requirements of a 
more flexible serialization system.
Like you say, I think the overall direction of your project is right, and 
personally I've found it useful for my own work watching how you've tackled 
various parts of it.

All the best,

  Tom

On Thursday, 23 August 2012 01:14:26 UTC+1, Piotr Grabowski wrote:
>
> Hi, 
>
> Google Sumer of Code is almost ended. I was working on customizable 
> serialization. This project was a lot harder than I expected, and sadly 
> in my opinion I failed to do it right. I want to apologize for that and 
> especially for my poor communication with this group and my mentor. I 
> want to improve it after midterm evaluation but it was only worse. 
>
> I don't think my project is all wrong but there is a lot things that are 
> different from how I planned. How it looks like (I wrote more in 
> documentation) 
> There is Serializer class that is made of two classes: NativeSerializer 
> and FormatSerializer. 
> NativeSerializer is for serialization and deserialization python objects 
> from/to native python datatypes 
> FormatSerializer is for serialization and deserialization python native 
> datatypes to/from some format (xml, json, yaml) 
>
> I want NativeSerializer to be fully independent from FormatSerializer 
> (and vice versa) but this isn't possible. Either NativeSerializer must 
> return some additional data or FormatSerializer must give 
> NativeSerializer some context. For exemple in xml all python native 
> datatypes must be serialized to string before serializing to xml. Some 
> custom model fields can have more sophisticated way to serialize to 
> sting than unicode() so `field.value_to_string` must be called and 
> `field` are only accessible in NativeSerializer object. So either 
> NativeSerializer will return also `field` or FormatSerializer will 
> inform NativeSerializer that it handles only text data. 
>
> Backward compatible dumpdata is almost working. Only few tests are not 
> passed, but I am not sure why. 
>
> Nested serialization of fk and m2m related fields which was main 
> functionality of this project is working but not well tested. There are 
> some issues especially with xml. I must write new xml format because old 
> wont work with nested serialization. 
>
> I didn't do any performance tests. Running full test suite take 40 
> seconds more with my serialization (about 1500s at all) if I remember 
> correctly. 
>
> I will try to complete this project so it will be at least bug free and 
> usable. If someone was interested in using nested serialization there is 
> other great project: https://github.com/tomchristie/django-serializers 
>
> Code: https://github.com/grapo/django/tree/soc2012-serialization 
> Documentation: https://gist.github.com/3085250 
>
> -- 
> Piotr Grabowski 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/a2gBdTn5C6EJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Moving forward with Serialization.

[Tom Christie]

Hi all,

  During Piotr's GSOC addressing customizable serialization I've been 
continued to work on my own implementation.
I've now got what feels to me to be a natural conclusion of the design, 
which is the 'forms' branch of the 
'django-serializers'project.

There's still a few small bits and pieces that need filling in, but it's 
pretty much there,
and I believe the project essentially meets the requirements of a more 
customizable serialization API.

I won't go into the full documentation here (refer to the project itself if 
you're interested), but in summary it gives you:

* A declarative serialization API with Serializer/ModelSerializer classes 
that mirror the existing Form/ModelForm API.
* A backwards compatible FixtureSerializer class.
* Support for nested relationships, pk relationships, natural key 
relationships, and custom relationships such as hyperlinking.
* Validation for deserialization of objects, similar to Form validation.
* Explicitly decouples the structural concerns of serialization from the 
encoding/decoding concerns.
* Passing Django's existing fixture serialization tests.
* Fairly trivial to port into Django while keeping backwards compatibility 
with existing serialization API.

What I particularly like about the approach is how it mirrors the existing 
Forms API, and I think it'd go a long way to making Django a more useable 
framework for authors of Web APIs.  As an example, here's what it looks 
like when using django-serializers to write API 
views
.

At this point what I'm looking for is some feedback:

* Would it be worth me trying to push towards getting this into core.
* Are there any fundamental issues with the API, or use cases it clearly 
doesn't address.
* Any suggested next steps, or things that would need to be 
addressed/clarified before it could considered it for core.

It's probably also worth noting that I'm now planning on using the project 
inside my personal project, django-rest-framework, in case that's relevant 
to the discussion.

Thoughts?

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/3ypJSF7nN8oJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Moving forward with Serialization.

[Tom Christie]

> Do you know Colander?

I do now.  Thanks, that's very similar, and looks well done.

> I personally think that Forms are already the place that should handle 
(de)serialisation. They already serialise to HTML: why should they not be 
able to serialise to other stream types?

Conceptually I agree.  As it happens django-serializers is perfectly 
capable of rendering into HTML forms, I just haven't yet gotten around to 
writing a form renderer, since it was out-of-scope of the fixture 
serialization functionality.

Pragmatically, I'm not convinced it'd work very well.  The existing Forms 
implementation is tightly coupled to form-data input and HTML output, and I 
think trying to address that without breaking backwards compatibility would 
be rather difficult.  It's maybe easy enough to do for flat 
representations, and pk relationships, but extending it to deal with 
nested representations, being able to use a Form as a field on another 
Form, and representing custom relationships would all take some serious 
hacking.  My personal opinion is that whatever benefits you'd gain in 
DRYness, you'd lose in code complexity.  Having said that, if someone was 
able to hack together a Forms-based fixture serialization/deserialization 
implementation that passes the Django test suite, and didn't look too 
kludgy, I'd be perfectly willing to revise my opinion. 

There's also some subtle differences between serializer fields, and 
Django's existing form fields.  Because form fields only handle form input, 
incoming fields can never be null, only blank or not blank.   With other 
representations such as JSON, that's not the case, so for serializer 
fields, the blank=True/False null=True/False style is appropriate, whereas 
for form fields the required=True/False style is appropriate.

I'm also wary of getting bogged down in high level 'wouldn't it be nice 
if...' conversations.  With just a little bit of work, the 
django-serializers implementation could be turned into a pull request 
that'd replace the existing fixture serialization with something much more 
useful and flexible.  What I'm really looking for is some feedback on if 
it'd be worth my time.

Regards,

  Tom


On Wednesday, 29 August 2012 06:27:35 UTC+1, schinckel wrote:
>
> Hi Tom,
>
> I've knocked around ideas of my own (internally, and on #django) related 
> to serialisation: it's something I've had lots to think about, since until 
> recently, most of my django work was in JSON apis.
>
> I personally think that Forms are already the place that should handle 
> (de)serialisation. They already serialise to HTML: why should they not be 
> able to serialise to other stream types?
>
> This is the approach I've started to use for my API generation code. They 
> already have a declarative nature, and then you get all of the form 
> validation on incoming data: a big improvement over how I've done it in the 
> past.
>
> (I've done some work on a form-based API generation framework: 
> https://bitbucket.org/schinckel/django-repose. Whilst this is in use, 
> it's still not really feature complete).
>
> Matt.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/Lc0fSSAdz4IJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Moving forward with Serialization.

[Tom Christie]

Thanks Piotr,

> But when serializing also Field is saved with data so it's not so clean.

I'm not sure whether it's an issue or not.  As with Django's form fields, 
the fields are copied off of base_fields, so saving state 
isn't necessarily a problem.  I'd agree that it's possibly not the cleanest 
part of the API, but it might be good enough.

> For each format there is Serializer class which is made 
from NativeSerializer ... and FormatSerializer

That would work.
I was planning on taking a slightly different approach in order to merge 
the work into Django.
In my opinion the `SERIALIZATION_MODULES` setting wouldn't make much sense 
once serialization and parsing/rendering are decoupled.  Furthermore it 
places some very specific, slightly odd, and 
undocumented constraints on the interface that serialization modules must 
provide.  Eg.  They must include a class named 'Serializer' and a class 
named 'Deserializer'.

My preference would be to put that setting on the road to deprecation, 
replacing it with something more appropriate, eg:

SERIALIZER_CLASS='django.core.serializers.FixtureSerializer'
RENDERER_CLASSES=(...)
PARSER_CLASSES=(...)

It'd be possible to do that in a way that'd be consistent with Django's 
deprecation policy, if the initial release continued to check 
if SERIALIZATION_MODULES is set, and use that in preference to the new 
style settings.

The `serialize`, `deserialize` and `get_serializer` functions continue to 
exist and would be light backward-compatible wrappers around the core of 
the new-style serialization.
 
Having said that I could probably live with either approach.

I guess it would probably make sense for me to pull django-serializers into 
a fork of django at some point, to better demonstrate what I'm suggesting.

> * IMO there is bug related to xml. All model fields must be transform to 
> text before xml serialization. In current django serialization framework 
> field's method value_to_string is responsible for this.

Thanks.  I've not looked into that yet.   It sounds feasible, though I 
think it'd have to only be affecting custom model fields in xml, and should 
be easy enough to fix.  If it's easy for you to provide an example case 
where django-serializer's output differs from the existing fixtures it'd be 
much appreciated.

Regards,

  Tom


On Saturday, 1 September 2012 12:39:32 UTC+1, Piotr Grabowski wrote:
>
> W dniu 31.08.2012 10:25, Tom Christie pisze: 
> > > I personally think that Forms are already the place that should 
> > handle (de)serialisation. They already serialise to HTML: why should 
> > they not be able to serialise to other stream types? 
> > 
> > Conceptually I agree.  As it happens django-serializers is perfectly 
> > capable of rendering into HTML forms, I just haven't yet gotten around 
> > to writing a form renderer, since it was out-of-scope of the fixture 
> > serialization functionality. 
> > 
> > Pragmatically, I'm not convinced it'd work very well.  The existing 
> > Forms implementation is tightly coupled to form-data input and HTML 
> > output, and I think trying to address that without breaking 
> > backwards compatibility would be rather difficult.  It's maybe easy 
> > enough to do for flat representations, and pk relationships, but 
> > extending it to deal with nested representations, being able to use a 
> > Form as a field on another Form, and representing custom relationships 
> > would all take some serious hacking.  My personal opinion is that 
> > whatever benefits you'd gain in DRYness, you'd lose in code 
> > complexity.  Having said that, if someone was able to hack together a 
> > Forms-based fixture serialization/deserialization implementation that 
> > passes the Django test suite, and didn't look too kludgy, I'd be 
> > perfectly willing to revise my opinion. 
>
> I am not quite sure but I think Forms should be build based on some 
> serialization API not opposite. Forms are more precise way of models 
> serialization - they are models serialized to html (specific format) 
> with some validation (specific actions) when deserializing. 
>
>
> I like Tom's django-serialziers but there are some things that I want to 
> mention: 
>
> * Process of serialization is split to two parts - transformation to 
> python native datatype (serializer) and next to specific text format 
> (renderer). But when serializing also Field is saved with data so it's 
> not so clean. I also have an issues with this but I resolve it in 
> different way (not say better :) 
>
> * In master branch Serializer is closely tied to Renderer so if there is 
> different Re

Re: Improve tutorial to use class based views

[Tom Christie]

Hi Jonas,

The tutorial *does* cover class based generic views, but only towards the 
end:

https://docs.djangoproject.com/en/dev/intro/tutorial04/#use-generic-views-less-code-is-better

Starting with function based views, and then moving on to class based views 
by the end feels like the right approach, as it keeps things simple and 
obvious at the start, and gradually moves on to more complex abstractions.

Regards,

  Tom


On Thursday, 13 September 2012 12:14:03 UTC+1, jonas wrote:
>
> Hello, 
>
> Like I, probably most people start with djagno by following the 
> tutorial on the djangoproject.com website. 
> This tutorial makes only use of function based views. 
> But today we have this wonderful things called class based views. 
>
> Is this really the best way to introduce newcomers to django ? 
>
> Are there any plans to improve the docs to use class based views ? 
>
> Regards, 
>
> Jonas. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/8U2lko-oIq4J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django performance vs others

[Tom Christie]

Please.  As Jacob has already made clear, this thread isn't helping move 
anything forward.
Can we please respect his request and move on.

Django's community is absolutely interested in addressing practical steps 
towards improving performance and would no doubt welcome specific work 
towards profiling the request-response cycle or otherwise improving the 
codebase.

The intention may be well-meaning, but links to 3rd party benchmarks 
without context, comment, or thoughts on actionable tasks are clearly not 
adding anything to the discussion, and I'm quite sure the core devs have 
better things to do with their time than thread moderation.

  - Tom

On Tuesday, 16 October 2012 11:36:59 UTC+1, Moonlight wrote:
>
> another one:
>
> http://mindref.blogspot.com/2012/10/python-web-reverse-urls-benchmark.html
>
>
> On Thursday, October 4, 2012 10:50:35 AM UTC+3, Moonlight wrote:
>>
>> I found the following benchmarks recently:
>> 1. http://mindref.blogspot.com/2012/09/python-fastest-web-framework.html
>> 2. http://mindref.blogspot.com/2012/07/python-fastest-template.html
>>
>> It is interesting to see the performance boost using pypy.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/x-2TLNJUutgJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Google groups - possible issues with deleted posts.

[Tom Christie]

Be aware that there may be issues with google groups at the moment...

There are two messages in this 
thread
 that 
are marked as deleted, apparently not 
intentionally
.
There are also three messages in this 
thread,
 
all around the same timeframe which are marked as deleted, which seems a 
little suspect.
I've also seen one message on a group I manage marked as 
deleted,
 
again around the same timeframe, and without any action by the sender of 
the message or by myself.

It is of course not impossible that there's been a spate of moderator 
and/or user error, but it does seem like google groups is probably being a 
bit wonky.

Probably worth keeping an eye out for similar behavior.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/t295xrxODN8J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposal: Django URL Admin (django-urls)

[Tom Christie]

Since this isn't really appropriate for django-dev, could we move any 
further discussion on this over to the django-users group, please?

Many thanks!

  Tom

On Sunday, 16 December 2012 18:53:43 UTC, Zach Borboa wrote:
>
> > 1) Being able to create redirects (which seems to already be on the 
> > todo-list) 
>
> Creating temporary 302 redirects are currently possibly right now. The 
> TODO is for being able to specify both 301 and 302 redirects through 
> the admin. 
>
>
> > 2) Being able to specify extra kwargs to pass to a view so that it would 
> be 
> > possible to change the functionality of a view without adding a new 
> model to 
> > store the different options (or have to change the urlconfig and push 
> new 
> > code every time). 
>
> You can still specify additional urls in urls.py as the framework 
> allows you to do normally; django-urls simply appends to that list of 
> urls. There is also a TODO for adding the ability to edit views 
> through the Django admin using the same editor GitHub uses. These 
> should cover the use cases you describe. 
>
>
> On Sat, Dec 15, 2012 at 11:52 PM, Sam Solomon > 
> wrote: 
> > Yeah, I agree, as it is, I can't see any reason why I would use it, but 
> I 
> > could see it being useful with some modifications such as: 
> > 
> > 1) Being able to create redirects (which seems to already be on the 
> > todo-list) 
> > 2) Being able to specify extra kwargs to pass to a view so that it would 
> be 
> > possible to change the functionality of a view without adding a new 
> model to 
> > store the different options (or have to change the urlconfig and push 
> new 
> > code every time). 
> > 
> > Until those things are implemented (which allow for things that could 
> > probably be implemented in more straightforward/non-dev-user friendly 
> ways), 
> > it seems like a bad idea to store infrastructure in the database (I can 
> only 
> > see it causing problems when you have developers working from a 
> different 
> > urlconf than the production server is using). 
> > 
> > 
> > On Friday, December 14, 2012 11:31:34 PM UTC-6, Amirouche B. wrote: 
> >> 
> >> 
> >> 
> >> On Friday, December 7, 2012 9:07:32 PM UTC+1, Zach Borboa wrote: 
> >>> 
> >>> Does something like this exist already? If not, it should. 
> >> 
> >> 
> >> How this can be useful ? You still need to write the view in Python 
> then 
> >> why not write the urls in Python too, like it's currently the way to 
> go. 
> >> 
> >> If something in this spirit might be useful is something where the view 
> >> could be generated which would be something like databrowser or admin. 
> >> 
> >> Could you elaborate ? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/G1CLGO-hWpQJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django 1.5 release plans

[Tom Christie]

Hi Yann,

There's [a thread on django-users][1] that should answer your request.

>From Russ "It's difficult to give an exact date for the release of Django 
1.5. We've put out 2 beta releases, which means there are no more features 
to be added; and the list of release blocking bugs is down to single 
figures"

There are currently [2 open release blockers][2], so I'd expect there will 
be a new ETA sometime after those are resolved.

[1]: 
https://groups.google.com/forum/?fromgroups=#!topic/django-users/NyYGM1-db4s
[2]: 
https://code.djangoproject.com/query?status=assigned&status=new&status=reopened&severity=Release+blocker

On Wednesday, 2 January 2013 15:37:45 UTC, Yann Beaud wrote:
>
> Hi team,
>
> Thanks for amazing job!
>
> But an update about the release date would be really great :)
>
> Thanks
>
> Yann Beaud
>
> Le mardi 11 septembre 2012 16:22:20 UTC+2, Jacob Kaplan-Moss a écrit :
>>
>> Hi folks -- 
>>
>> I wanted to fill everyone in on our plans for the Django 1.5 release. 
>> The highlights are: 
>>
>> * Feature freeze October 1st, final out before Christmas. 
>>
>> * One marquee feature of Django 1.5 is experimental Python 3 support. 
>> This is where we need your help the most: we need to be sure that our 
>> support for Python 3 hasn't destabilized Django on Python 2. We need 
>> lots of testing here! 
>>
>> * Most features of 1.5 have already landed, but we're also hoping to 
>> land the new pluggable User model work, add support for PostGIS 2.0, 
>> start the process of deprecating django.contrib.localflavor, and a few 
>> other small things. 
>>
>> * This'll be our first "master never closes" release: work, including 
>> new features, can continue to land on master while we ship the 
>> release. 
>>
>> Please read on for details. 
>>
>> Timeline 
>>  
>>
>> Oct 1: Feature freeze, Django 1.5 alpha. 
>> Nov 1: Django 1.5 beta. 
>> Nov 26: Django 1.5 RC 1 
>> Dec 10: Django 1.5 RC 2 
>> Dec 17: Django 1.5 RC 3, if needed 
>> Dec 24 (or earlier): Django 1.5 final 
>>
>> (All dates are "week of" - we'll do the releases that week, though not 
>> neccisarily that exact day.) 
>>
>> Notice the longer-than-usual timeline from beta to final. We're doing 
>> this to provide some extra time stablizing the release after landing 
>> the Python 3 work. Please see below for details and how you can help. 
>>
>> Python 3 support 
>>  
>>
>> Django 1.5 includes experimental support for Python 3 (it's already 
>> landed on master). We're taking a "shared source" approach: Django's 
>> code is written in a way that runs on both Python 2 and Python 3 
>> (without needing 2to3's translation). This means that we've touched 
>> nearly the entire codebase, and so the surface area for possible bugs 
>> is huge. 
>>
>> WE REALLY NEED YOUR HELP testing out Django 1.5 *on Python 2*. Please 
>> grab master, or one of the upcoming alpha/beta/RC releases, and test 
>> it against your apps and sites. We need you to help us catch 
>> regressions. 
>>
>> We're not yet recommending that people target Python 3 for deployment, 
>> so our main focus here is ensuring that we're still rock-solid on 
>> Python 2. If you *want* to give Python 3 a whirl things should be 
>> pretty solid, but we *especially* need real-world reports of success 
>> or failure on Python 2. 
>>
>> Features in 1.5 
>> --- 
>>
>> Besides the stuff that's already landed (see 
>> https://docs.djangoproject.com/en/dev/releases/1.5/), there are a few 
>> other features we're hoping to land: 
>>
>> * The "pluggable User model" work (Russell Keith-Magee). 
>> * Some early low-level schema alteration plumbing work (Andrew Godwin). 
>> * Moving django.contrib.localflavor out into individual external 
>> packages (Adrian Holovaty). 
>> * Support for PostGIS 2.0 (Justin Bronn). 
>> * Python 3 support in GeoDjango (Aymeric Augustin). 
>> * App-loading (Preston Holmes) is "on the bubble" - there's some 
>> debate among the core team over whether its ready, but it's close. 
>>
>> Of course, as with our previous releases, the *real* list of what'll 
>> go in 1.5 is "whatever's done by October 1st". If you want to help 
>> with any of the above areas, contact the person doing the bulk of the 
>> work (listed above) and ask to help. And if you have other features 
>> you'd like to land, get 'em done! 
>>
>> Master never closes 
>> --- 
>>
>> This'll mark our first release where "master never closes". 
>>
>> To recap: in previous releases, once we hit feature freeze we froze 
>> the development trunk, forcing all feature work out to branches. In 
>> practice, this meant months-long periods where new features couldn't 
>> be merged, and led to some stuff withering on the vine. 
>>
>> That's not going to happen this time. Instead, when we release 1.5 
>> alpha we'll make a 1.5 release branch right at that point. Work will 
>> continue on master -- features, bugfixes, whatever -- and the 
>> aplicable bugfixes will be cherry-

Re: Ticket #17093 -- quarantine global state in django.template

[Tom Christie]

Hi Christopher,

> OK, let mi introduce the idea for Google Some Of Code of this year. The 
idea is stolen from Pyramid [3]. The main new concept is a renderer. A 
renderer may be current TemplateEngine as well as JSON serializer or mako 
renderer.

I can't really comment on if it'd be appropriate for GSoC or not, but it'd 
probably be worth you taking a look at Django REST 
framework's 
renderers, which do pretty much exactly what you described.

Simple example of rendering to JSON:

class MyView(APIView):
renderer_classes = (JSONRenderer,)

def get(self, request, *args, **kwargs):
return Response({'desc':'my response to AJAX request'})

Rendering to HTML, using template & template context.

class MyView(APIView):
renderer_classes = (HTMLTemplateRenderer,)

def get(self, request, *args, **kwargs):
return Response({'desc':'my response to AJAX request'}, 
template_name='example.html')

There's a bit more to it than that, but I thought I'd mention it since it's 
so similar to the API you outlined.


On Thursday, 21 February 2013 15:16:09 UTC, Christopher Medrela wrote:
>
> Hello. I would like to report progress of my work (not too much new 
> things, rather cleaning and polishing), discuss some issues and propose an 
> idea for next Google Summer Of Code.
>
> Progres from last time:
>
>- Solved problem with TemplateResponse -- it receives optional keyword 
>argument called `engine` (if omitted, then it calls get_default_engine). 
>During pickling the engine is lost. During repickling get_default_engine() 
>is called.
>- Dropped idea of invalidating caches: base.invalid_var_format_string 
>and context._standard_context_processors. These settings are used in about 
>a dozen places in tests and it's not really necessary now.
>- Renamed _Template back to Template. The Template class accepts 
>optional keyword argument called `engine`. If omitted, then it calls 
>get_default_engine().
>
>
> I've also rebased commits so the branch forks from updated master. I've 
> reordered and squashed commits and rewritten description of commits so all 
> work ended in less then twenty well-described commits.
>
> I tried to make all tests green before next weekend (Django Sprint is 
> coming!), but I failed. There are some failing tests (exactly 450 xD); they 
> are heisenbugs -- they appear only when you run all tests; if you run only 
> one test suit, then they disappear. I guess that the problem is that the 
> default_engine should be invalidated between some tests. I also bisected 
> commits and I found out that the commit introducing failures is [1]. I will 
> try to fix these problems before the sprint on Saturday, because the 
> current state is almost ready to review (and to merge, I hope) and I would 
> like to have something that can be reviewed at the weekend.
>
> I will sum up what I've done. I've not introduced new features. I've 
> gathered all pieces of global state into one (the default_engine). I've 
> introduced TemplateEngine class and I've rewritten most tests so they 
> create their own TemplateEngine instances instead of using the global one. 
> I tried not to introduce backward incompatibilities.
>
> I've rewritten history so in order not to break existing links I won't use 
> ticket_17093 branch any more. Instead, see ticket_17093v2 branch [1].
>
> [1] 
> https://github.com/chrismedrela/django/commit/ed578d64fff8c8eb58898548d5ef1c0815c25f24
> [2] https://github.com/chrismedrela/django/tree/ticket_17093v2
>
> > I agree that it doesn't mean less global state. But there is good global
>
> > state and bad global state. For example, we could make DjangoApplication
>
> > class to avoid global state but there is no need. Global settings is OK
>
> > since we don't need the ability to create more than one django
>
> > application instance. Even tests doesn't require more than one instance
>
> > at the same time, so we can just reinitialize the application when there
>
> > is need for that. That's done by change_settings decorator and it works
>
> > well. And there is bad global state, like current template system.
>
> ...
>
> > I didn't know about Jinja2 and that it's more suitable for use outside
>
> > Django. So it seems like the only goal is refactoring -- we need to
>
> > quarantize global state, but not dependencies on settings. We also don't
>
> > need to split the template system into django-dependent and
>
> > django-independent parts.
>
>
>> I'm not sure the distinction between "good global state" and "bad global
>
> state" is so clear, or even exists. A major part of the motivation for
>
> #17093 was that it could be a step along the path towards a
>
> global-state-free Django (using something like a DjangoApplication or
>
> ApplicationConfig object), which would allow running multiple Django
>
> sites in a single process (which is indeed something that some p

Re: first() and last(), earliest() and latest()

[Tom Christie]

> +1 for the first syntax.  The others are duplicating functionality 
> that is already present via more aptly named methods.  The first 
> syntax is also more consistent with other ORMs. 

Seconded.
Seems much more obvious, simple and explicit than the other options to me.

On Thursday, 28 February 2013 00:37:53 UTC, Ian wrote:
>
> On Wed, Feb 27, 2013 at 3:34 PM, Wim Feijen > 
> wrote: 
> > Which style do you prefer? 
> > 
> > .filter(last_name__startswith='b').order_by('last_name').first()# 
> clear 
> > and long 
> > .first(last_name__startswith='b').order_by('last_name')# first 
> method 
> > has filter syntax. 
> > .filter(last_name__startswith='b').first('last_name')   # first method 
> has 
> > order_by syntax. 
>
> +1 for the first syntax.  The others are duplicating functionality 
> that is already present via more aptly named methods.  The first 
> syntax is also more consistent with other ORMs. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: URL dispatcher fallthrough?

[Tom Christie]

For what it's worth I'd be against this proposal as it stands.

* I haven't seen anything anything that convinces me that a single wrapping 
view isn't a reasonable alternative in the examples cited.
* A `UrlNotMatched` exception sounds like the potential source of 
incredibly non-obvious bugs and surprising behavior.
* Allow a single HTTP call to end up calling into multiple views just seems 
like it would a fundamentally bad design decision.

I really don't see the trade-off of allowing this type of behavior to be 
worth the cost of breaking the current mental model of what a view is and 
does.

Just my personal opinion of course :)

  Tom

On Tuesday, 26 March 2013 17:49:07 UTC, Val Neekman wrote:
>
> +1 here! 
>
> It might also be a great feature to run the 404 through before sending 
> that email out. 
> A replacement and/or complement to IGNORABLE_404_URLS 
>
> Val 
>
>
> On Tue, Mar 26, 2013 at 1:25 PM, Loic Bistuer 
> > 
> wrote: 
> > +1 for me. 
> > 
> > Having a catchall view under a single URL pattern is a tightly coupled 
> > system, more so than allowing independent views to "test the water". 
> > 
> > Django core relies on middleware hacks because the URL dispatcher is 
> missing 
> > this very feature. Having this in core would allow a cleaner 
> implementation 
> > of the fallback mechanisms in contrib.flatpages and contrib.redirects. 
> > 
> > -- 
> > Loic 
> > 
> > On Mar 19, 2013, at 11:18 PM, Adrian Holovaty 
> > > 
> wrote: 
> > 
> > I'd rather not add this to the framework, as it's already possible 
> > with a "wrapper" view (as others have suggested). And on a more 
> > theoretical level, it introduces more coupling between URL patterns 
> > and views. 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "Django developers" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to django-develop...@googlegroups.com . 
> > To post to this group, send email to 
> > django-d...@googlegroups.com. 
>
> > Visit this group at 
> http://groups.google.com/group/django-developers?hl=en. 
> > For more options, visit https://groups.google.com/groups/opt_out. 
> > 
> > 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: URL dispatcher fallthrough?

[Tom Christie]

>
> What is the reasonable alternative? The only way I can think how a 
> wrapping view will be able to handle this case, is to write your own 
> router...
>
 

...In each case you'll have lots of ifs, try..excepts, for each model, and 
> then to appropriate view. For the second and third router_views you'd have 
> to have the same thing repeated, but this time it is nested.


One pattern you could use to avoid ugly nesting would be to write a helper 
function that attempts a queryset lookup against multiple models, for 
example...

obj = get_instance_or_404(Foo, Bar, Baz, pk=pk)
if isinstance(obj, Foo):
...
elif isinstance(obj, Bar):
...
elif isinstance(obj, Baz):
...

Something like that would probably be fine for the simpler cases.

For more complicated cases you can implement pretty much the same API 
you're looking for but do it in view code.  Something like this...

class Switcher(View):
"""
Delegate the handling of the request to one of a list of delegates.

Any delegate may raise an IgnoreRequest exception to pass handling
over to the next delegate in the list.
"""
delegates = []

def dispatch(self, request, *args, **kwargs):
for delegate in self.delegates:
try
return delegate(request, *args, **kwargs)
except IgnoreRequest:
pass
raise Http404

urlpatterns = patterns('',
(r'/([^/]+)/$', Switcher.as_view(delegates=[...])),
(r'/([^/]+)/([^/]+)/$', Switcher.as_view(delegates=[...])),
(r'/([^/]+)/([^/]+)/([^/]+)/$', Switcher.as_view(delegates=[...]))
)

There's also the question of view_middleware not being applied for the 
> appropriate view unless the user specifically looks into how middleware 
> gets called and manages handling view middleware themselves in the 
> different router_views.


The view middleware question seems like another really good argument 
against the proposed change...

The view middleware can't be called just once with the 'correct' view, 
because there's not way of determining up front if a view will raise a 
`UrlNotMatched` without calling into it.

So, what happens when `UrlNotMatched` is raised - do all the middleware 
`process_view` methods get called again?

Cheers,

  Tom :)


On Wednesday, 27 March 2013 23:27:44 UTC, meric wrote:
>
> Tom, you're right about the second and third points though. If the user 
> perform any operation with side effect that requires writing to the 
> database before the view checks whether the keyword arguments are 
> appropriate and decide to raise DoesNotResolve, it can definitely be a 
> source of non-obvious bugs and surprising behaviour. For example, a counter 
> increment in the view to count how many times the view has been visited, 
> placed before checking for whether to raise DoesNotResolve. Multiple such 
> views get executed, multiple view counters incremented for one HTTP 
> connection.
>
> I can only think of adding a stern warning to the documentation something 
> along the lines of "Must not perform any operation requiring a database 
> write or any other operation with side effects before the check for 
> DoesNotResolve is made.".
>
> Eric
>
> On Thursday, March 28, 2013 3:28:10 AM UTC+11, Tom Christie wrote:
>>
>> * A `UrlNotMatched` exception sounds like the potential source of 
>> incredibly non-obvious bugs and surprising behavior.
>> * Allow a single HTTP call to end up calling into multiple views just 
>> seems like it would a fundamentally bad design decision.
>>
>> I really don't see the trade-off of allowing this type of behavior to be 
>> worth the cost of breaking the current mental model of what a view is and 
>> does.
>>
>> Just my personal opinion of course :)
>>
>>   Tom
>>
>>>  
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Request method in urls.py

[Tom Christie]

This proposal is actually *very* similar to the 'URL 
dispatcher fall-though' 
threadthat
 came up recently.

I don't think it'd take much adjustment to take Jacob's django-multiurl 
project  and adapt it to deal 
with method-based dispatching rather than fall-through based dispatching.

You should be able to end up with an API that looks something like this:

from methodurl import methodurl

urlpatterns = patterns('',
methodurl(
url('/objects/$', app.views.create_object, methods=['post', 
'put'], name='create_object'),
url('/objects/$', app.views.list_objects, methods=['get'], 
name='list_objects'),
)
)

Personally, I'm not particular convinced on the merits of method based 
routing, but that's not really the issue.
The important point is that this is something that you should be able to do 
without needing to modify anything in core Django.

Regards,

  Tom

On Saturday, 13 April 2013 22:48:44 UTC+1, Brantley Harris wrote:
>
> There's a line in the django URL Dispatcher documentation that is pretty 
> weird:
>
> The URLconf doesn’t look at the request method. In other words, all 
>> request methods – POST, GET, HEAD, etc. – will be routed to the same 
>> function for the same URL.
>
>
> Well, why?  Most modern web frameworks allow you to specify the method in 
> the routing configuration, Django seems to be an exception in this respect.
>
> It would be extremely easy to implement, and would lead to vastly better 
> code across new projects, including a simpler way of writing rest style 
> interfaces:
>
> url('^objects/$', 'views.create_object', methods=['post', 'put'], 
> name='create_object'),
> url('^objects/$', 'views.get_objects', name='list_objects'),
>
>
> This has come up many times before and been swatted down for various 
> reasons.  One is that it could be implemented with a one-off dispatcher, as 
> in:
>
> url('^objects/$', create_or_list(list=get_objects, create=create_object), 
> name='create_or_list_objects')
>
>
> But this is overly complex for what should be a simple configuration, 
> forces one to create the same name for the url, and worse, creates a level 
> of indirection breaking the abstraction up; or in other words you're trying 
> to do route configuration, why not do it in the place you're already doing 
> route configuration?
>
> The other argument is that you can do this with Class Based Views.  I 
> don't believe this is a good argument as one would have to utilize Class 
> Based Views to get this basic functionality.  In fact CBV's, only really 
> solve two common issues, one is the boilerplate inherit in forms, and the 
> other method routing.  But the proposed solution to method routing is 
> simpler and better.
>
> I will gladly implement the code required for this, but don't wish to do 
> so if it's going to be quashed, which is why I bring it up here.
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Proposal for simplifying part of the GCBV API slightly.

[Tom Christie]

I'd be interested to know what you folks think of this proposal.

SingleObjectMixin currently exposes these three attributes and one method 
all dealing with queryset filter arguments...

* pk_url_kwarg = 'pk'
* slug_field = 'slug'
* slug_url_kwarg = 'slug'
* get_slug_field()

I was wondering if it would be worth considering simplifying the API 
somewhat, by moving those three attributes, and that method, into a 
PendingDeprecation state, and replacing their usage with a single attribute 
instead, that is used both as the URL kwarg, and as the queryset filter.

* lookup_field = 'pk'

That'd (eventually) lead to a simpler get_object implementation internally, 
and (immediately) present a slightly nicer, simpler API.

It'd be marginally different in that slug based lookups would require an 
explicit `lookup_field = 'slug'` to be added to the view,
and also in that it wouldn't handle the case of `slug_field` being set to a 
different value from `slug_url_kwarg`.

Personally I don't see the later being an issue as:

1. It's not clear to me why you'd ever *need* to use a URL kwarg that's not 
the same as the filter arg.
2. If it's really something you need to do, then overriding get_object is 
still really simple, as well as being nice and explicit...

get_object(self, queryset):
if queryset is None:
queryset = self.get_queryset()
return get_object_or_404(queryset, ...) # whatever custom lookup 
behavior you need.

To me, the main trade-off seems to be - Is the resulting API enough of an 
improvement to be worth the change?

Any opinions?

  Tom

As an aside, is the discussion group considered the correct place for 
API-changing proposals like this, or should I just raise them straight to 
the ticket tracker?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal for simplifying part of the GCBV API slightly.

[Tom Christie]

Hi Calvin,

I can think of a few reasons.
> - you've changed models or fields internally through migrations, but need 
> to continue supporting old URLs.
> - you don't like the internal name to be exposed
> - you have different models but want to expose a consistent URL pattern.


Those attributes control the URL kwargs that are used in the regex, we're 
not talking about URL query parameters or anything else that's visible to 
the end-user.  Internal names aren't being exposed anywhere, and there'd be 
no issue with updating field names and continuing to support the URLs that 
reference them.

Having said that, you're probably correct that I've overstated point (1) - 
There might be some circumstances where the developer would prefer to use 
'slug' as the regex kwarg, but look up against a differently named model 
field.  I'd still think that there's a very strong argument to be made that 
we could consider that a corner case that requires overriding `get_object`, 
in exchange for having a simpler API for the standard case.

There would of course be other alternatives such as using both 
`lookup_field` and `lookup_kwarg` with the later defaulting to the same as 
the former if unset, but I'm not wild about something like that given that 
the intention of this proposal is to try to slightly slim down an aspect of 
the API.


On Monday, 22 April 2013 12:37:32 UTC+1, Calvin Spealman wrote:
>
>
> On Apr 22, 2013 7:15 AM, "Tom Christie" > 
> wrote:
> >
> > I'd be interested to know what you folks think of this proposal.
> >
> > SingleObjectMixin currently exposes these three attributes and one 
> method all dealing with queryset filter arguments...
> >
> > * pk_url_kwarg = 'pk'
> > * slug_field = 'slug'
> > * slug_url_kwarg = 'slug'
> > * get_slug_field()
> >
> > I was wondering if it would be worth considering simplifying the API 
> somewhat, by moving those three attributes, and that method, into a 
> PendingDeprecation state, and replacing their usage with a single attribute 
> instead, that is used both as the URL kwarg, and as the queryset filter.
> >
> > * lookup_field = 'pk'
> >
> > That'd (eventually) lead to a simpler get_object implementation 
> internally, and (immediately) present a slightly nicer, simpler API.
> >
> > It'd be marginally different in that slug based lookups would require an 
> explicit `lookup_field = 'slug'` to be added to the view,
> > and also in that it wouldn't handle the case of `slug_field` being set 
> to a different value from `slug_url_kwarg`.
> >
> > Personally I don't see the later being an issue as:
> >
> > 1. It's not clear to me why you'd ever *need* to use a URL kwarg that's 
> not the same as the filter arg.
>
> I can think of a few reasons.
> - you've changed models or fields internally through migrations, but need 
> to continue supporting old URLs.
> - you don't like the internal name to be exposed
> - you have different models but want to expose a consistent URL pattern. 
>
> > 2. If it's really something you need to do, then overriding get_object 
> is still really simple, as well as being nice and explicit...
> >
> > get_object(self, queryset):
> > if queryset is None:
> > queryset = self.get_queryset()
> > return get_object_or_404(queryset, ...) # whatever custom lookup 
> behavior you need.
> >
> > To me, the main trade-off seems to be - Is the resulting API enough of 
> an improvement to be worth the change?
> >
> > Any opinions?
> >
> >   Tom
> >
> > As an aside, is the discussion group considered the correct place for 
> API-changing proposals like this, or should I just raise them straight to 
> the ticket tracker?
> >
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Django developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-develop...@googlegroups.com .
> > To post to this group, send email to 
> > django-d...@googlegroups.com
> .
> > Visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
> > For more options, visit https://groups.google.com/groups/opt_out.
> >  
> >  
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal for simplifying part of the GCBV API slightly.

[Tom Christie]

I've created ticket #20432 <https://code.djangoproject.com/ticket/20342> for 
this proposal.
I don't know if it needs to go to 'design decision needed', but if it gets 
approved I'll plan to work on it at the DjangoCon sprints, aiming for the 
1.6 alpha.

  Tom

On Monday, 22 April 2013 14:40:09 UTC+1, Andrew Ingram wrote:
>
> The original use case for pk_url_kwarg and slug_url_kwargs was something 
> like this:
>
> /(?P[\w-]*)/ - DetailView
> /(?P[\w-]*)/another-resource/ - Scoped ListView
> /(?P[\w-]*)/another-resource/(?P[\w-]*) - Scoped 
> DetailView
>
> In this case, the Scoped ListView and Scoped DetailView would both inherit 
> a mixin that overrides get_queryset(), and the Scope DetailView would just 
> set "slug_url_kwargs = 'another_slug'"
>
> I've used some variant of this pattern on almost every project I've been 
> involved with that utilises class-based views, including some frameworks, 
> so I don't think this edge case is quite as edgy as it might seem at first.
>
> However, I do agree that your simplification is an improvement. I've done 
> a lot with CBVs since I first suggested the URL kwarg overrides, and I 
> think it's better to have less configurable fields and focus instead on 
> having good entry points into customising the classes through method 
> overrides.
>
>
> Andy
>
>
>
> On 22 April 2013 13:33, Tom Christie >wrote:
>
>> Hi Calvin,
>>
>> I can think of a few reasons.
>>> - you've changed models or fields internally through migrations, but 
>>> need to continue supporting old URLs.
>>> - you don't like the internal name to be exposed
>>> - you have different models but want to expose a consistent URL pattern.
>>
>>
>> Those attributes control the URL kwargs that are used in the regex, we're 
>> not talking about URL query parameters or anything else that's visible to 
>> the end-user.  Internal names aren't being exposed anywhere, and there'd be 
>> no issue with updating field names and continuing to support the URLs that 
>> reference them.
>>
>> Having said that, you're probably correct that I've overstated point (1) 
>> - There might be some circumstances where the developer would prefer to use 
>> 'slug' as the regex kwarg, but look up against a differently named model 
>> field.  I'd still think that there's a very strong argument to be made that 
>> we could consider that a corner case that requires overriding `get_object`, 
>> in exchange for having a simpler API for the standard case.
>>
>> There would of course be other alternatives such as using both 
>> `lookup_field` and `lookup_kwarg` with the later defaulting to the same as 
>> the former if unset, but I'm not wild about something like that given that 
>> the intention of this proposal is to try to slightly slim down an aspect of 
>> the API.
>>
>>
>> On Monday, 22 April 2013 12:37:32 UTC+1, Calvin Spealman wrote:
>>>
>>>
>>> On Apr 22, 2013 7:15 AM, "Tom Christie"  wrote:
>>> >
>>> > I'd be interested to know what you folks think of this proposal.
>>> >
>>> > SingleObjectMixin currently exposes these three attributes and one 
>>> method all dealing with queryset filter arguments...
>>> >
>>> > * pk_url_kwarg = 'pk'
>>> > * slug_field = 'slug'
>>> > * slug_url_kwarg = 'slug'
>>> > * get_slug_field()
>>> >
>>> > I was wondering if it would be worth considering simplifying the API 
>>> somewhat, by moving those three attributes, and that method, into a 
>>> PendingDeprecation state, and replacing their usage with a single attribute 
>>> instead, that is used both as the URL kwarg, and as the queryset filter.
>>> >
>>> > * lookup_field = 'pk'
>>> >
>>> > That'd (eventually) lead to a simpler get_object implementation 
>>> internally, and (immediately) present a slightly nicer, simpler API.
>>> >
>>> > It'd be marginally different in that slug based lookups would require 
>>> an explicit `lookup_field = 'slug'` to be added to the view,
>>> > and also in that it wouldn't handle the case of `slug_field` being set 
>>> to a different value from `slug_url_kwarg`.
>>> >
>>> > Personally I don't see the later being an issue as:
>>> >
>>> > 1. It's not clear to 

Re: test discovery

[Tom Christie]

Hi Carl,

  Excellent, excellent stuff - many thanks to both yourself and Preston.

Couple of questions:

* Will it be possible to globally configure the default file/path pattern?
  Jannis's django-discover-runner includes support for a 
`TEST_DISCOVER_PATTERN` - is there anything similar, or just the command 
line `--patterns` option?  The use case I'm looking for is a single `tests` 
directory containing all the test modules, and I'm wondering if that'd be 
easy to support?
* Is this work broadly compatible with `django-discover-runner`?  I'd 
really like to make use of it in third party packages, but I'm wondering if 
I'd be able to do so in a way that lets me also provide compatibility with 
older versions of Django.

Thanks again,

  Tom

On Wednesday, 8 May 2013 23:04:29 UTC+1, Carl Meyer wrote:
>
> Hi Anssi, 
>
> On 05/08/2013 03:31 PM, Anssi K��ri�inen wrote: 
> > It would be really nice to be able to use same syntax for running a 
> > single Django's testcase at least for a while. I expect that there 
> > will be problems if inspecting how a given test case behaves in older 
> > versions compared to HEAD. Also, bisecting over the test case renaming 
> > boundary will be really ugly. I don't believe end users will have too 
> > much problems with this, but bug hunting and development in Django 
> > itself will be a bit more tedious to do. 
>
> I don't think this is enough of a problem to warrant more code to try to 
> work around it. 
>
> If you're bisecting, at worst this means having two commands in your 
> shell history instead of one and hopping up to the right one for each 
> bisect step. I think this is a minor irritation that will fade away soon 
> enough. 
>
> Most of our test apps are small/fast enough that you could just run the 
> whole app when bisecting instead of an individual TestCase, making this 
> a non-issue (since Django's test apps are top-level on sys.path, the 
> command to run a single app in Django's test suite is the same with the 
> new runner as with the old: "python runtests.py app_name"). 
>
> Carl 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal for simplifying part of the GCBV API slightly.

[Tom Christie]

Right now that ticket's just got a cautious +0 against it.
I still think this would be a small step in the right direction of 
presenting a slightly simpler GCBV API, but unless there's any further 
support I'm inclined to give the ticket a few more days and then close it.

On Thursday, 2 May 2013 08:58:56 UTC+1, Tom Christie wrote:
>
> I've created ticket #20432 <https://code.djangoproject.com/ticket/20342> for 
> this proposal.
> I don't know if it needs to go to 'design decision needed', but if it gets 
> approved I'll plan to work on it at the DjangoCon sprints, aiming for the 
> 1.6 alpha.
>
>   Tom
>
> On Monday, 22 April 2013 14:40:09 UTC+1, Andrew Ingram wrote:
>>
>> The original use case for pk_url_kwarg and slug_url_kwargs was something 
>> like this:
>>
>> /(?P[\w-]*)/ - DetailView
>> /(?P[\w-]*)/another-resource/ - Scoped ListView
>> /(?P[\w-]*)/another-resource/(?P[\w-]*) - Scoped 
>> DetailView
>>
>> In this case, the Scoped ListView and Scoped DetailView would both 
>> inherit a mixin that overrides get_queryset(), and the Scope DetailView 
>> would just set "slug_url_kwargs = 'another_slug'"
>>
>> I've used some variant of this pattern on almost every project I've been 
>> involved with that utilises class-based views, including some frameworks, 
>> so I don't think this edge case is quite as edgy as it might seem at first.
>>
>> However, I do agree that your simplification is an improvement. I've done 
>> a lot with CBVs since I first suggested the URL kwarg overrides, and I 
>> think it's better to have less configurable fields and focus instead on 
>> having good entry points into customising the classes through method 
>> overrides.
>>
>>
>> Andy
>>
>>
>>
>> On 22 April 2013 13:33, Tom Christie  wrote:
>>
>>> Hi Calvin,
>>>
>>> I can think of a few reasons.
>>>> - you've changed models or fields internally through migrations, but 
>>>> need to continue supporting old URLs.
>>>> - you don't like the internal name to be exposed
>>>> - you have different models but want to expose a consistent URL pattern.
>>>
>>>
>>> Those attributes control the URL kwargs that are used in the regex, 
>>> we're not talking about URL query parameters or anything else that's 
>>> visible to the end-user.  Internal names aren't being exposed anywhere, and 
>>> there'd be no issue with updating field names and continuing to support the 
>>> URLs that reference them.
>>>
>>> Having said that, you're probably correct that I've overstated point (1) 
>>> - There might be some circumstances where the developer would prefer to use 
>>> 'slug' as the regex kwarg, but look up against a differently named model 
>>> field.  I'd still think that there's a very strong argument to be made that 
>>> we could consider that a corner case that requires overriding `get_object`, 
>>> in exchange for having a simpler API for the standard case.
>>>
>>> There would of course be other alternatives such as using both 
>>> `lookup_field` and `lookup_kwarg` with the later defaulting to the same as 
>>> the former if unset, but I'm not wild about something like that given that 
>>> the intention of this proposal is to try to slightly slim down an aspect of 
>>> the API.
>>>
>>>
>>> On Monday, 22 April 2013 12:37:32 UTC+1, Calvin Spealman wrote:
>>>>
>>>>
>>>> On Apr 22, 2013 7:15 AM, "Tom Christie"  wrote:
>>>> >
>>>> > I'd be interested to know what you folks think of this proposal.
>>>> >
>>>> > SingleObjectMixin currently exposes these three attributes and one 
>>>> method all dealing with queryset filter arguments...
>>>> >
>>>> > * pk_url_kwarg = 'pk'
>>>> > * slug_field = 'slug'
>>>> > * slug_url_kwarg = 'slug'
>>>> > * get_slug_field()
>>>> >
>>>> > I was wondering if it would be worth considering simplifying the API 
>>>> somewhat, by moving those three attributes, and that method, into a 
>>>> PendingDeprecation state, and replacing their usage with a single 
>>>> attribute 
>>>> instead, that is used both as the URL kwarg, and as the queryset filter.
>>>> >
>>>> &

Re: Proposal: Support for HTTP PATCH method

[Tom Christie]

I'm basically in agreement with Russ here.

I'd consider the first three points as being non-controversial.

With regards to ProcessFormView I think implementing a put() method, but 
not a patch() method falls squarely into the 'good-enough' category of 
behavior.  In any case, although there are existing conventions, the exact 
semantics of PATCH requests are a little vague[*], and the generic view's 
handling of PUT/PATCH is of fairly limited use, given that they only 
support form-encoded data, and not, say JSON data. Given that, I don't 
think we need to get too hung up on correctness in that particular area.

Cheers,

  Tom

[*]: As an example Ember's `json-api` style uses RFC 6902, which is more 
involved than most conventional implementations of PATCH: 
http://tools.ietf.org/html/rfc6902


On Tuesday, 21 May 2013 13:38:15 UTC+1, Russell Keith-Magee wrote:
>
> Hi Krzysztof
>
> On Mon, May 20, 2013 at 7:48 PM, Krzysztof Jurewicz <
> krzysztof...@gmail.com > wrote:
>
>> There is a RFC describing HTTP method named PATCH:
>>
>> http://tools.ietf.org/html/**rfc5789 
>>
>> Quoting that RFC:
>> “  The difference between the PUT and PATCH requests is reflected in the
>>way the server processes the enclosed entity to modify the resource
>>identified by the Request-URI.  In a PUT request, the enclosed entity
>>is considered to be a modified version of the resource stored on the
>>origin server, and the client is requesting that the stored version
>>be replaced.  With PATCH, however, the enclosed entity contains a set
>>of instructions describing how a resource currently residing on the
>>origin server should be modified to produce a new version.”
>>
>> Django currently has (seemingly shallow) support for PUT method. Based on 
>> that, I’ve grepped for places in code where it is implemented to see if 
>> support for PATCH could be easily added on a basis of symmetry. Those 
>> places are:
>>
>> In test client:
>> There is a `put` method, but `patch` is also implemented:
>> https://github.com/django/**django/commit/**
>> 293f7a21147ad94c92c7d5b3f33cba**b2f87b001b
>>
>> In View class:
>> https://github.com/django/**django/blob/**a93672622829e0d4a2ff3240456d4d*
>> *73b9d46476/django/views/**generic/base.py#L33
>> `put` is listed in http_method_names
>>
>> In RedirectView:
>> `put` is a fallback to .get():
>> https://github.com/django/**django/blob/**a93672622829e0d4a2ff3240456d4d*
>> *73b9d46476/django/views/**generic/base.py#L207
>>
>> In ProcessFormView:
>> https://github.com/django/**django/blob/**a93672622829e0d4a2ff3240456d4d*
>> *73b9d46476/django/views/**generic/edit.py#L164
>> PUT is implemented as a fallback to POST and it seems to assume that 
>> request data is form encoded. While it is not generally true for PUT 
>> method, it looks like a reasonable assumption when processing a form. 
>> However since PATCH request doesn’t have to contain full form data, I think 
>> the best approach may be to not implement `patch` method here (unless we 
>> want to deal with editing only some of the form’s fields – for example this 
>> is the case in Rails, where PATCH will be the primary method for updates in 
>> 4.0 version: http://weblog.rubyonrails.org/**
>> 2012/2/25/edge-rails-patch-is-**the-new-primary-http-method-**
>> for-updates/).
>>  The same stays true to `get_form_kwargs` method in FormMixin:
>> https://github.com/django/**django/blob/**a93672622829e0d4a2ff3240456d4d*
>> *73b9d46476/django/views/**generic/edit.py#L44
>> In general, Django’s approach to whether PUT should be treated as a 
>> request containing form data seems to be inconsistent because we generate 
>> forms in the views, but we refuse to generate PUT QueryDict on a request 
>> object: https://groups.google.com/d/**msg/django-developers/**
>> dxI4qVzrBY4/m_9IiNk_p7UJ.
>>  So maybe it’s worth to consider removing PUT support in form processing 
>> just on the basis of consistency.
>>
>> In conclusion, it seems pretty easy to add basic PATCH support similar to 
>> PUT. So if it gets enough +1s, I’m willing to make a patch and hope it gets 
>> included in 1.6 alpha. Or maybe it is (from a technical point of view) so 
>> small a feature that it doesn‘t need to be merged before al

Re: Proposal: better support for generating rest apis in django core

[Tom Christie]

Hi Karol,

  Thanks for bringing that up.  I'll start by saying that I would love to 
see richer Web API support in core Django - in my opinion there are 
absolutely some fundamentals that could be addressed.   However, I 
personally think that the benefit that we'd get probably isn't (yet?) worth 
the very large amount of work that would be required.  I'll get onto that 
in a moment, but I'm first going to go through some of the core components 
that could be addressed...

Request parsing is an obvious one.  Right now it's actually quite awkward 
to deal with parsing form data in anything other than POST requests. 
 There's subtleties to get right for supporting both url-encoded and 
multipart forms, handling charset encodings, handling corrupt data, and 
dealing with the various possible states of the underlying stream.  That 
gets more complicated again if you want to support both form data and json 
or other content types.  That sort of thing has to be tackled each time by 
every new API framework, and it'd be a good candidate for better support in 
core.  REST framework's approach is to introduce a `request.DATA` that 
supports parsing arbitrary content types depending on which parser are 
registered (either globally or per-view).  Something similar in Django 
would be valuable, because it's a minimal amount of API, but addresses a 
fundamental issue.

Serialization is another fundamental that's currently lacking.  This is a 
deceptively difficult area, particularly the deserialization and validation 
aspects.  Django's forms aren't sufficient for many of the use-cases you 
need to support with APIs.  A mature, complete serialization framework 
needs to be able to support nested objects, various types of representation 
for relationships, support for both ORM and non-ORM data, serializing and 
deserializing from composites of models (eg both User data and UserProfile 
data in a single representation), and various other bits & pieces.  Ideally 
a serialization framework should be flexible enough that it's equally 
capable of rendering a model instance into an HTML form representation, as 
it is of rendering it into JSON.  Furthermore, if someone wanted to get a 
serialization framework into Django it'd make sense if it could also be 
used to replace the existing fixture loading/saving.  Getting a really 
decent serialization framework into Django would obviously be a valuable 
thing to do, as it'd give us all a common, well-supported way of doing for 
Web APIs what forms currently allow us to do in Web apps.  However, it'd 
evidently require a very large amount of work to successfully land in 
Django.

Content negotiated responses are another possible candidate.  Content 
negotiation is a core component of Web APIs, but we've currently no support 
in Django for dealing with it.  There's not any great value in each new API 
framework solving content negotiation again, because there's a reasonably 
consistent set of rules about how it should be handled.  Being able to 
return a response and have it render into various formats depending on the 
client request is something that could be dealt with in core Django once, 
and then reused by any API framework.

Another area worth mentioning is authentication - in particular CSRF 
protection.  The right way to deal with CSRF protection in Web APIs is 
non-obvious.  Session authentication is valid for AJAX clients, and should 
require CSRF protection.  Other authentication types such as OAuth are also 
valid but should not require CSRF protection.  Handling that in a correct, 
secure way is fiddly, and something that I'm personally a little 
uncomfortable with API frameworks having to individually deal with.

I think it's feasible that we could address some of these core ares, and 
still leave plenty of space for API frameworks on top of Django to 
experiment and thrive, while improving the baseline support for folks using 
just Django.

Back to the point, though.  The amount of work that'd be required to 
comprehensively address this in core would be pretty huge.  We're in decent 
situation at the moment, with at least a couple of great, really 
well-supported API frameworks as third-party packages.  Having those 
packages independent to Django is disadvantageous in that it fragments the 
already limited resources of our community, but it's also advantageous in 
that they can iterate more quickly than core Django, as well as being able 
to take opinionated design decisions that suit their domain.

Django's ecosystem is pretty much it's biggest strength.  In my opinion the 
cheapest, most beneficial option would be to look at ways to better promote 
third party packages from within the documentation.  Obviously there's some 
discussion to be had about how to do that in a fair and open way, while 
still allowing an element of curation and ensuring that packages meet a 
required quality.  Calling out quality packages in some way from the docs 
would help new users

Confusion around generic views queryset cloning.

[Tom Christie]

I've noted that the generic view implementations use the private `_clone` 
method when returning the queryset 
attribute
.

Presumably the need for that is something to do with the potential for 
querysets to be cached or otherwise incorrectly stateful if this cloning 
isn't performed.  What's confusing me is that it's not at all obvious 
*exactly* what the implications of making (or failing to make) this call 
are.

Furthermore if it *is* something that's strictly required in this 
circumstance then do we need to be documenting whatever behavior is being 
triggered, so that developers writing their own class based views don't 
make the (mistake?) of simply returning/using a queryset attribute without 
having first cloned it?

For example, is the following incorrect?

class SomeBaseGenericView(View):
queryset = None

def get_queryset(self):
"""
Simply return `.queryset` by default. Subclasses may override 
this behavior.
"""
return self.queryset

If so, under what set of conditions can it fail, and is it possible to unit 
test for the failing behavior?

I've dug into the source, but the `_clone` method isn't documented, nor can 
I find anything terribly helpful related to queryset cloning after googling 
around for a while.

Many thanks,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Confusion around generic views queryset cloning.

[Tom Christie]

My apologies, this was intended to be posted to the Django users group.

On Thursday, 11 July 2013 21:54:25 UTC+1, Tom Christie wrote:
>
> I've noted that the generic view implementations use the private `_clone` 
> method when returning the queryset 
> attribute<https://github.com/django/django/blob/master/django/views/generic/detail.py#L72>
> .
>
> Presumably the need for that is something to do with the potential for 
> querysets to be cached or otherwise incorrectly stateful if this cloning 
> isn't performed.  What's confusing me is that it's not at all obvious 
> *exactly* what the implications of making (or failing to make) this call 
> are.
>
> Furthermore if it *is* something that's strictly required in this 
> circumstance then do we need to be documenting whatever behavior is being 
> triggered, so that developers writing their own class based views don't 
> make the (mistake?) of simply returning/using a queryset attribute without 
> having first cloned it?
>
> For example, is the following incorrect?
>
> class SomeBaseGenericView(View):
> queryset = None
>
> def get_queryset(self):
> """
> Simply return `.queryset` by default. Subclasses may override 
> this behavior.
> """
> return self.queryset
>
> If so, under what set of conditions can it fail, and is it possible to 
> unit test for the failing behavior?
>
> I've dug into the source, but the `_clone` method isn't documented, nor 
> can I find anything terribly helpful related to queryset cloning after 
> googling around for a while.
>
> Many thanks,
>
>   Tom
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Confusion around generic views queryset cloning.

[Tom Christie]

Thanks both, the responses are very much appreciated.

> It's the same issue as using mutable default arguments or attributes to a 
class. If someone is writing their own generic CBVs without understanding I 
don't see why the queryset case is any more problematic than having a class 
with an empty list defined on it and appending things to it from instances.

True, it's exactly the same issue, although it's more of a gotcha, in that 
appending to a list is explicitly a stateful operation, whereas it's more 
of an implicit side effect when iterating over a queryset.

Something as simple as the following is broken, although it might not be 
obvious on first sight.

class MyView(View):
queryset = Users.objects.all()
template_name = 'foobar'

def get(self, request):
return render(request, self.template_name, {'items': 
self.queryset})

I don't know that there's a huge amount worth doing about that.  We *could* 
add some documentation warning against modifying class attributes on CBV, 
but I'm not sure if it'd just be adding noise and be more confusing that 
it's worth for many users.

> What we could do is add an explanatory comment to the code as to why we 
are doing that and/or change it to use `all()` instead and/or promote 
`clone()` to be a real part of the API.

Generally, ensuring that the GCBVs are implemented using public API seems 
sensible. Anything else is indicative that there's something missing from 
the API that is in fact necessary to end-users. Having said that I really 
don't have strong views on the right approach here.

* `.clone` as public API seems okay, but I'd always prefer to avoid 
introducing new API if it can be avoided.
* Using `.all` also seems okay, but it feels a bit odd that 
`MyModel.objects.all()` would then have end up having a second `.all()` 
operation chained to it during `get_queryset()`.  I realize that's 
equivalent to what's happening now, but it looks odd and might be confusing.
* Continue using `._clone`, but adding a comment in the implementation 
might be sufficient.
* Maybe things are okay as they are right now.

Cheers,

  Tom

On Friday, 12 July 2013 09:43:10 UTC+1, Marc Tamlyn wrote:
>
> I don't think this should need documenting explicitly. It's the same issue 
> as using mutable default arguments or attributes to a class. If someone is 
> writing their own generic CBVs without understanding I don't see why the 
> queryset case is any more problematic than having a class with an empty 
> list defined on it and appending things to it from instances. I think it's 
> quite well documented that the a queryset caches its results.
>
> Also, in many cases in the CBVs this clone is not strictly necessary as 
> the qs is being further filtered (or called with `get` or `count` or some 
> such) in such a way that the original qs is not being used directly.
>
> What we could do is add an explanatory comment to the code as to why we 
> are doing that and/or change it to use `all()` instead and/or promote 
> `clone()` to be a real part of the API.
>
> Hope that makes some sense.
>
>
> On 11 July 2013 22:26, Carl Meyer > wrote:
>
>> Hi Tom,
>>
>> (You said in a follow-up this was intended for django-users, but
>> personally I think it's on-topic for -developers, so I'm replying here.)
>>
>> On 07/11/2013 02:54 PM, Tom Christie wrote:
>> > I've noted that the generic view implementations use the private
>> > `_clone` method when returning the queryset attribute
>> > <
>> https://github.com/django/django/blob/master/django/views/generic/detail.py#L72
>> >.
>> >
>> > Presumably the need for that is something to do with the potential for
>> > querysets to be cached or otherwise incorrectly stateful if this cloning
>> > isn't performed.  What's confusing me is that it's not at all obvious
>> > *exactly* what the implications of making (or failing to make) this call
>> > are.
>> >
>> > Furthermore if it *is* something that's strictly required in this
>> > circumstance then do we need to be documenting whatever behavior is
>> > being triggered, so that developers writing their own class based views
>> > don't make the (mistake?) of simply returning/using a queryset attribute
>> > without having first cloned it?
>> >
>> > For example, is the following incorrect?
>> >
>> > class SomeBaseGenericView(View):
>> > queryset = None
>> >
>> > def get_queryset(self):
>> > """
>> > Simply return `.query

Re: Support POST of application/json content type

[Tom Christie]

Hi Stefan,

Sure, I'd be interested in seeing us improve how we deal with JSON requests 
and responses.

My preference would be to introduce a request parsing and response 
rendering API that allows us to support not just JSON, but any media type 
that has a parser installed for it.  (I've commented on some of this 
before, 
here,
 
although I think I'm warming towards the idea that it's probably about time 
we started addressing at least some of this in core.)

Unsurprisingly I'd suggest the same general approach that is used in REST 
framework - A lazy `request.DATA` attribute (or similar) that when 
accessed, inspects the media type on the request, and parses the request 
stream with an appropriate parser if possible.  The installed parsers can 
be configured globally, or on a per-request basis.  The existing multipart 
and form-encoded parsing behaviour would no longer be a special case baked 
directly into the request object, but instead be the default installed 
parsers.

Taking this approach makes it trivial to write views that can handle both 
JSON and form data, and providing a proper parser API makes it easy for 
developers to package up and share their own parser implementation, such as 
YAML, XML and MsgPack.  (And, looking forwards, JSON-based media types such 
as hypermedia types.)

In REST framework this behaviour is (by necessity) implemented in a Request 
object that wraps the underlying HttpRequest, but the same basic 
implementation can be applied to implementing it directly in the Request 
object, and would be somewhat easier.

I'm interested to see Marc suggesting middleware specifically for handling 
JSON requests.  That'd work, and would be a simple approach.  My 
reservations with that would be:

* We'd not be making it any easier for users to deal with request parsing 
generally.
* There's no way to write views that deal with request data in an agonistic 
way, and dealing with differing media types would require switching based 
on the media type in the view itself.  For example, the generic views would 
still only support form data.  As another example, if you wanted to add, 
say, MsgPack support to your application, you'd need to re-write all your 
views.

>From my point of view this is already a solved problem, and I'd really like 
to see a generic approach to handling request data, and a corresponding 
approach to rendering responses into an appropriate media type.

 All the best,

  Tom

On Tuesday, 3 September 2013 06:30:04 UTC+1, Stefan Berder wrote:
>
> Hi,
> I looked around the list and couldn't find any mention of this subject.
>
> In `django.http.request.HttpRequest._load_post_and_files()` there is 
> explicit mention of two content type ('multipart/form-data' and 
> 'application/x-www-form-urlencoded'), any other content type will get empty 
> values for self._post.
>
> Given that a lot of user form interaction is now happening through 
> 'XMLHttpRequest', I think that the 'application/json' content type should 
> be supported. A lot of javascript libraries will use json as the default 
> format:
> * angularjs: http://docs.angularjs.org/api/ng.$http, see "Setting HTTP 
> Headers"
> * emberjs: 
> https://github.com/emberjs/data/blob/master/packages/ember-data/lib/adapters/rest_adapter.js#L974
> * backbone: http://backbonejs.org/#Sync
> * jquery: http://api.jquery.com/jQuery.ajax/ (the only one using 
> 'application/x-www-form-urlencoded' by default)
>
> I'm trying primarily to create a discussion on the subject and am ready to 
> provide the code for it as I had to write it. This would help avoid hacks 
> to handle the request object in my view. 
>
> I know there are some apps to handle API construction (django-tastypie, 
> django-rest, django-piston and certainly others) they use either request 
> wrappers or request handling in their views.
>
> Stefan
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Support POST of application/json content type

[Tom Christie]

> Creating a request.DATA attribute would break compatibility with old code 
and seem to me less intuitive.

The implementation would ensure `request.POST` would still be populated for 
form data.

There wouldn't have to be any backwards incompatible changes, and usage of 
`request.DATA` (or whatever better name there might be, perhaps simply 
`request.data`) would be entirely optional for using with generic request 
parsing, instead of form data only.

> Where should request.FILE go in that case

request.FILES would be populated by form data as normal, and would be empty 
on JSON or other submissions.  In order to support multipart parsing the 
request parsing API would need to provide for parsers that support file 
upload, in a similar way that REST framework currently does.

> Would request.POST just be a call to request.DATA?

That's an open question, but I'd probably expect it to only return data if 
the request contains multipart or url-encoded form data.  (Ie. the 
behaviour wouldn't change.)

Cheers,

  Tom

On Wednesday, 4 September 2013 12:33:00 UTC+1, Stefan Berder wrote:
>
> Tom,
> I agree that the middleware solution is not the best and in my mind 
> creates fragmentation of the same type of request handling.
>
> A generic content-type framework would make the request parsing more 
> flexible, stronger to change in the future and open the door to any type.
>
> I'm curious about the choice of request.DATA though, when doing a POST 
> request, I expect the data to be in the POST dictionary and nowhere else. 
> Creating a request.DATA attribute would break compatibility with old code 
> and seem to me less intuitive. Where should request.FILE go in that case? 
> Would request.POST just be a call to request.DATA?
>
> Stefan
>
> On Wednesday, 4 September 2013 18:13:12 UTC+8, Tom Christie wrote:
>>
>> Hi Stefan,
>>
>> Sure, I'd be interested in seeing us improve how we deal with JSON 
>> requests and responses.
>>
>> My preference would be to introduce a request parsing and response 
>> rendering API that allows us to support not just JSON, but any media type 
>> that has a parser installed for it.  (I've commented on some of this 
>> before, 
>> here<https://groups.google.com/forum/#!searchin/django-developers/tomchristie%7Csort:date/django-developers/Qr0EorpgYKk/6qyCrVqZwmMJ>,
>>  
>> although I think I'm warming towards the idea that it's probably about time 
>> we started addressing at least some of this in core.)
>>
>> Unsurprisingly I'd suggest the same general approach that is used in REST 
>> framework - A lazy `request.DATA` attribute (or similar) that when 
>> accessed, inspects the media type on the request, and parses the request 
>> stream with an appropriate parser if possible.  The installed parsers can 
>> be configured globally, or on a per-request basis.  The existing multipart 
>> and form-encoded parsing behaviour would no longer be a special case baked 
>> directly into the request object, but instead be the default installed 
>> parsers.
>>
>> Taking this approach makes it trivial to write views that can handle both 
>> JSON and form data, and providing a proper parser API makes it easy for 
>> developers to package up and share their own parser implementation, such as 
>> YAML, XML and MsgPack.  (And, looking forwards, JSON-based media types such 
>> as hypermedia types.)
>>
>> In REST framework this behaviour is (by necessity) implemented in a 
>> Request object that wraps the underlying HttpRequest, but the same basic 
>> implementation can be applied to implementing it directly in the Request 
>> object, and would be somewhat easier.
>>
>> I'm interested to see Marc suggesting middleware specifically for 
>> handling JSON requests.  That'd work, and would be a simple approach.  My 
>> reservations with that would be:
>>
>> * We'd not be making it any easier for users to deal with request parsing 
>> generally.
>> * There's no way to write views that deal with request data in an 
>> agonistic way, and dealing with differing media types would require 
>> switching based on the media type in the view itself.  For example, the 
>> generic views would still only support form data.  As another example, if 
>> you wanted to add, say, MsgPack support to your application, you'd need to 
>> re-write all your views.
>>
>> From my point of view this is already a solved problem, and I'd really 
>> like to see a generic approach to handling request data, and a 
>> corresponding approach to rendering responses into an appropriate media 
>

Re: Support POST of application/json content type

[Tom Christie]

In REST framework, we expose the accepted renderer as an 
`request.accepted_renderer` property.
This allows you to switch based on the media type when needed by either 
doing something like:

if request.accepted_renderer.format == 'yaml':
# do something specific

or like this:

if request.accepted_renderer.media_type == 'application/json':
# do something specific

That allows you to do media type specific stuff if needed, but generally 
you'll just access `request.DATA`, and not need to be concerned about 
exactly what media type was decoded for you.  (Of course the API specifics 
could be open to bike shedding.)  In response to Jonathan's particular 
query, REST framework supports a YAML parser, and there's never any need to 
do specific branching to detect it, because it just returns native python 
datatypes: lists, dicts, etc... exactly the same as the JSON parser does.

And yes, different media types can and do have different capabilities (eg. 
form data decodes into QueryDict objects, rather than regular dicts) but 
you're able to switch on that if specifically needed.

I'm not at all keen on the idea of injecting differing attribute names on 
the request based on the media type that is decoded, partly because it just 
feels like an odd API to me, but more concretely because it unnecessarily 
ties your view logic to your parser classes, and doesn't allow you to 
easily support views that accept multiple content types.  (The combination 
of either form data or json being the most common.) 

> what else do you have in DRF's Request that you would need

The bits of API we'd need to support something like this:

* `request.DATA` / `request.data` for accessing the parsed data.
* `request.parsers` or similar for modifying the parsers prior to parsing 
on a per-view basis.  (Note the parallel to request.upload_handlers)
* `request.accepted_parser` or similar for branching on media type if 
needed.
* A setting that controls the default set of installed parsers.

There are a couple of other bits of behaviour/API, but they're not related 
to request parsing.  I wouldn't mind discussing those as well at some 
point, but I don't want to go off on a tangent just yet.

The other core thing would be the question of if dealing with request 
parsing also means we should necessarily also tackle response rendering 
(and ideally, content negotiation) at the same time.  I don't really mind 
whether that'd be seen as a requirement or not, but it is important that we 
have a consistent design between both parsing and rendering.

Cheers all,

  Tom

On Thursday, 5 September 2013 00:31:43 UTC+1, Curtis Maloney wrote:

To weight in from an author of a different API tool
>
> In django-nap there's a simple "get_request_data" method which, 
> essentially, determines how to parse the request according to the content 
> type.  However, currently each API only supports a single serialiser format 
> [and HTTP Form encoding] so there's little guessing involved.
>
> However, I wouldn't want to see the same on Request, unless there was also 
> a direct way to imply the type you want.  For instance, if I get a request 
> that's in XML, and I ask for request.JSON, I'd like it to either yield 
> empty or raise an error.  Whereas when I didn't case, accessing 
> request.DATA would make a best guess.
>
> So I see two paths... either you use a "Decode it for me according to 
> content-type" interface, or a "Treat is as this type, and fail predictably 
> if it's not" one.
>
> The current GET/POST is, of course, the latter.  And there's no reason we 
> can't have both :)
>
> --
> Curtis
>
>
>
> On 5 September 2013 04:06, Jonathan Slenders 
> 
> > wrote:
>
>> Would that mean that the object returned by request.DATA/POST/whatever 
>> could be a different type, depending on what the user posted?
>>
>> I don't want to see code like:
>>
>> if isinstance(request.DATA, YamlObject): ...
>> elif isinstance(request.DATA, dict): ...
>>
>> although, I'm not sure how any view could handle any random 
>> content-type...
>>
>>
>>
>> Le mercredi 4 septembre 2013 13:57:29 UTC+2, Marc Tamlyn a écrit :
>>>
>>> The thing with request.POST is that it's kinda unintuitive when extended 
>>> to other HTTP methods (e.g. PUT). This is why the old request.raw_post_data 
>>> was renamed request.body.
>>>
>>> request.POST would behave in the expected traditional web way of picking 
>>> up form encoded POST data, which would also be available in request.DATA as 
>>> well, but request.DATA is the "new" way of doing it

Re: Support POST of application/json content type

[Tom Christie]

> why keep data and files separated

Mostly because that's the way it already works, so...

* request.data would essentially provide a strict superset of the 
functionality that request.POST provides.  In general you'd be able to 
replace `request.POST` with `request.data` anywhere and seemlessly start 
supporting JSON or other data without any other changes required.
* Form expect the data and files to be provided separately which would be 
awkward otherwise.

> In the absence of strong objection, I will start working on this base. 

Sure thing.  As it happens, I was also considering taking a crack at this 
in the coming weeks, so please do follow up on this thread linking to your 
repo if you start working on it, so myself and others can track any 
progress.  (And perhaps collaborate.)

Cheers :)

  Tom

On Wednesday, 11 September 2013 04:52:08 UTC+1, Stefan Berder wrote:
>
> On Tue, Sep 10, 2013 at 12:17 PM, S Berder > 
> wrote: 
> > On Tue, Sep 10, 2013 at 9:05 AM, Curtis Maloney 
> > > wrote: 
> >> 
> >> On 9 September 2013 19:50, S Berder > 
> wrote: 
> >>> 
> >>> Gents, 
> >>> to sum it up, arguments made and details of how I see the 
> >>> implementation of a response/request encode/decode framework: 
> >>> 
> >>> * need a pluggable interface so current content-types are supported 
> >>> (`application/x-www-form-urlencoded`, `multipart/form-data`), new 
> >>> types (`application/json`), custom and future types 
> >>> (`application/vnd.foobar+json` anybody? See 
> >>> http://developer.github.com/v3/media/#api-v3-media-type-and-the-future 
> >>> for example, `application/msgpack`, `application/protobuf`, 
> >>> `application/capnproto`, etc). 
> >>> * decoder/encoder map (content-type, decoder) should be smart to 
> >>> handle patterns like `text/*` or `application/*xml*` and match things 
> >>> like `Accept: application/json, text/plain, * / *` 
> >>> * choice of decoder would be made on the Content-Type header, maybe 
> >>> supporting a raw by default so data is just passed in case of unknown 
> >>> content type. 
> >>> * decoder/encoder should be available through `request` and `response` 
> >>> objects. 
> >>> * decoded data structure (python object) would be stored in 
> `request.data` 
> >>> * first step is to support requests, next step is to handle responses 
> >>> with the same pluggable functionality and coherent API. 
> >>> * A sensible default for response Content-type would be `text/html; 
> >>> charset=UTF-8`. It should be made available through a setting entry 
> >>> anyway 
> >>> 
> >> 
> >> You should also have access to the decision made by the data parser as 
> to 
> >> which parser was used, instead of having to infer it yourself from the 
> >> content type header. 
> > 
> > Indeed, that's the 4th point of my list, maybe it's not clear as it is 
> > but this would be supported. 
> > 
> >>> 
> >>> Some questions though: 
> >>> 
> >>> * why keep data and files separated, I see no good reason for this 
> >>> except mimicking PHP's structure. An uploaded file comes from a named 
> >>> input, I hope to find it in request.data (why do a common structure 
> >>> otherwise). I might be missing something but nothing indicates a real 
> >>> need for this in django/http/request.py 
> >> 
> >> 
> >> True, there's some added complexity [small as it is] in forms because 
> File 
> >> fields need to look elsewhere for their values. 
> >> 
> >>> 
> >>> * isn't more or less any data sent to your backend representable as a 
> >>> dict or object with dict access modes? I try to think about 
> >>> occurrences where some data would not have a 'name'. 
> >>> 
> >> 
> >> I frequently send JSON lists of data to my APIs... 
> > Ok, was a bit short sighted on this one, still thinking in terms of 
> > form bound data, it was a long day here in Shanghai. I suppose that 
> > the kind of python object you receive is not so important as you 
> > should do data validation anyway. Your earlier concern about checking 
> > for different content-types doesn't apply to the solution I have in 
> > mind as to whatever data representation you have at the beginning, you 
> > should get a very similar object after decoding. What I mean is if you 
> > send the *same* data through Yaml or JSON, the object in request.data 
> > should be the same or extremely close. I say extremely close because 
> > I'm thinking about xml that is always way more verbose than the others 
> > and *might* add more data to the resulting object. (hint: I don't like 
> > XML, don't need it in what I do and last used it ~8/9 years ago in a 
> > disastrous explosion of SOAP and unix/microsoft interfaces) 
> > 
> > Stefan 
> > -- 
> > http://www.bonz.org/ 
> >  /(bb|[^b]{2})/ 
>
> In the absence of strong objection, I will start working on this base. 
>
> Stefan 
> -- 
> http://www.bonz.org/ 
>  /(bb|[^b]{2})/ 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and

Re: A policy on calling super()

[Tom Christie]

Calling super in base classes (ie anything that inherits from 'object') 
just seems unnecessary and obscure to me.  It's not a pattern I use or have 
seen, and after playing around a bit I can't see any sensible case where 
it'd make a difference.  `View` should always be the last (right-most) 
class in hierarchy, so there shouldn't ever be any parent behaviour that 
needs calling into.

On Saturday, 28 September 2013 13:34:23 UTC+1, Daniele Procida wrote:
>
>  
>
> There's some discussion of a particular class, django.views.base.View, and 
> whether its __init__() should contain a super(View, self).__init__(). 
>
> But there's also a wider question of whether there should be a general 
> rule about this, whether the integrity of the __init__() chain should be 
> maintained, and whether it matters that not all of our classes that are 
> likely to be subclassed do it. 
>
> Any comments? 
>
> Daniele 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: A policy on calling super()

[Tom Christie]

I'm not sure that's really sensible style, tho - classes that are intended 
to be used as mixins should call super, and the base class should be the 
last in the hierarchy.

class FooMixin(object):
 def __init__(self):
super(FooMixin, self).__init__()
self.foo = 3

class MyModel(FooMixin, models.Model):
pass

Am I missing some useful case where it's necessary for a mixin class to be 
used as the base class rather than a super class?

On Sunday, 29 September 2013 18:01:38 UTC+1, Alex_Gaynor wrote:
>
> It matters if you're going to mixin a class whose common ancestor is 
> object, e.g.:
>
> class FooMixin(object):
>  def __init__(self):
> self.foo = 3
>
> class MyModel(models.Model, FooMixin):
> pass
>
> if models.Model.__init__ doesn't call super().__init__, then 
> FooMixin.__init__ won't be invoked.
>
> Alex
>
>
> On Sun, Sep 29, 2013 at 10:00 AM, Tom Christie 
> 
> > wrote:
>
>> Calling super in base classes (ie anything that inherits from 'object') 
>> just seems unnecessary and obscure to me.  It's not a pattern I use or have 
>> seen, and after playing around a bit I can't see any sensible case where 
>> it'd make a difference.  `View` should always be the last (right-most) 
>> class in hierarchy, so there shouldn't ever be any parent behaviour that 
>> needs calling into.
>>
>>
>> On Saturday, 28 September 2013 13:34:23 UTC+1, Daniele Procida wrote:
>>>
>>> <https://code.djangoproject.**com/ticket/2<https://code.djangoproject.com/ticket/2>>
>>>  
>>>
>>>
>>> There's some discussion of a particular class, django.views.base.View, 
>>> and whether its __init__() should contain a super(View, self).__init__(). 
>>>
>>> But there's also a wider question of whether there should be a general 
>>> rule about this, whether the integrity of the __init__() chain should be 
>>> maintained, and whether it matters that not all of our classes that are 
>>> likely to be subclassed do it. 
>>>
>>> Any comments? 
>>>
>>> Daniele 
>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to 
>> django-d...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
>
> -- 
> "I disapprove of what you say, but I will defend to the death your right 
> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: 125F 5C67 DFE9 4084
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Working towards a simpler GCBV implementation?

[Tom Christie]

Hi folks,

I recently released an alternative implementation of Django's existing 
class based views.  The intention was to mirror the *exact* same set of 
functionality that we currently provide, but simplify the implementation 
and API.  It's nothing cleverer or grander than a clean re-write, but the 
end result is *significantly* less complex.  The class hierarchy is 
trivial, the API is much smaller, and the flow control is much more obvious.

I won't go into any more here, as there's plenty of detail in the 
documentation:

http://django-vanilla-views.org

There's also useful context in the related blog post, here:

http://dabapps.com/blog/fixing-djangos-generic-class-based-views/

The difficult thing here really is that there's no obvious approach to 
introducing something like this in a backwards compatible way.

It might be that we could take an incremental approach using the standard 
deprecation process, but given the nature of the GCBVs it'd likely be 
pretty awkward.
There might be some bits of deprecation process we simply wouldn't be able 
to deal with in any sensible way, and we'd certainly end up with a 
seriously gnarly implementation during the deprecation period.

I'd be interested in getting some opinions from folks on the following:

* If a simpler GCBV implementation along the lines of django-vanilla-views 
is something we think we should working towards.
* What approaches we might be able to take to dealing with backwards 
compatibility if we did want to do so.

Thanks for your time.

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fe19ddbc-a4d4-4dc0-a241-df45f6e73c96%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Permission to use some Django material ?

[Tom Christie]

Hi Stan,

I would simply include some text in the license noting that some images are 
taken from the Django source and their usage is subject to it's license, 
with a link to the Django license.  I'm sure that'd be plenty sufficient 
attribution.

All the best,

  Tom

On Thursday, 3 October 2013 10:13:03 UTC+1, Stan wrote:
>
> Thanks Russ,
>
> Do you know if there is a best way to "suitably attribute" images ? Maybe 
> a text file beside REAME.txt / LICENCE.txt ?
>
> Cheers,
>
> Stanislas
>
> On Thursday, October 3, 2013 11:02:04 AM UTC+2, Russell Keith-Magee wrote:
>>
>> Hi Stanislas,
>>
>> Django is licensed under the terms of the BSD license. This means you can 
>> use any of Django's source code -- including images -- for whatever 
>> purpose, commercial or otherwise, as long as whatever you use is suitably 
>> attributed, and you don't make any claims that what you've done is endorsed 
>> by Django or it's contributors.
>>
>> The license text itself is quite easy to read:
>>
>> https://github.com/django/django/blob/master/LICENSE
>>
>> Yours,
>> Russ Magee %-)
>>
>>
>> On Thu, Oct 3, 2013 at 4:28 PM, Stan  wrote:
>>
>>> Hi,
>>>
>>> I am working on a small app providing Ajax search for a 
>>> ModelChoiceFieldLike in addition to an admin-like related lookup (pop up to 
>>> changelist). Is it possible for me to embed the magnifier image from admin 
>>> statics ? (static/admin/img/selector-search.gif)
>>>
>>> Thanks.
>>>
>>> Stanislas.
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/2d566ef5-c4a2-4801-ac42-da77506b7603%40googlegroups.com
>>> .
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ae579780-fd75-4fa5-8fd0-c884183a7c3e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Working towards a simpler GCBV implementation?

[Tom Christie]

> But django-vanilla-views are not as usable because you cannot 
reuse isolated pieces of functionality like you can with mixins.

The reusable components that it gives you are `GenericView` and 
`GenericModelView`.
Generally they're going to be the baseline that you'd want to work with 
anyways.
As one example of reusing these base components I've refactored Andrew 
Ingram's `django-extra-views` package using vanilla views. (*)
The end result is a set of generic views for dealing with formsets that 
came out something like 2/3 as much code as was previously required, and 
again, with a much simpler more obvious implementation throughout.

> from django.new_views import ..., like was done with new_forms?

Okay, so there's a precedent.
Really I'm still trying to sound out where the community stands on this.
If there was a pull request along these lines should I expect it to get 
rejected out of hand as too much of a radical change, or are folks 
sufficiently convinced that this is a good idea?

Thanks,

  Tom

(*): See https://github.com/tomchristie/django-extra-views - Not properly 
doc'ed or packaged yet, but considering doing so and releasing as a 
`django-vanilla-formsets` package.

On Saturday, 5 October 2013 14:45:38 UTC+1, is_null wrote:
>
> I should state that I have no education and that I have the feeling 
> that I understand GCBVs perfectly. 
>
> But django-vanilla-views are not as usable because you cannot reuse 
> isolated pieces of functionality like you can with mixins. 
>
> Maybe this is a documentation problem ? 
>
> Maybe the docs should explicitly recommend to read the source code ? 
>
> Maybe it should recommend some links for users to sharpen their 
> understanding of GCBVs ? 
>
> Hope this helps 
>
> James 
>
> -- 
> http://yourlabs.org 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/89d68213-f15a-48e0-86bb-f5a77fd364d2%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: #18659 -- Deprecating request.REQUEST

[Tom Christie]

+1 on removing it.
It's usage reinforces incorrect assumptions about the meaning and behaviour 
of `request.POST` and `request.GET`.

> It's hardly ever a good design pattern to handle GET and POST identically

I'd phrase is as "It's hardly ever a good design pattern to handle query 
parameters and request bodies identically", but otherwise exactly this, yes.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3e98864c-ba33-48bf-b693-8006e786dea5%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: #18659 -- Deprecating request.REQUEST

[Tom Christie]

> but perhaps we should provide better names for `request.GET` and 
`request.POST` at the same time

Sure, I'd absolutely agree in principle, and for what it's worth REST 
framework provides `.QUERY_PARAMS` and `.DATA` attributes on the request 
which are recommended instead of using `.GET` and `.POST`.  Of course the 
.DATA attribute in that case provides general purpose request parsing, and 
isn't limited to form content types.

The current names are fundamentally incorrect, and I think they help sow 
the seeds for newcomers to misunderstand the basics of HTTP as a result.

Having said that, in practice I don't know if they're something we'd ever 
move away from.

I don't really feel in any position to judge how we weigh up the current 
naming awkwardness versus the pain of introducing such a major API 
difference. 


On Friday, 18 October 2013 14:13:48 UTC+1, James Aylett wrote:
>
> On Wednesday, October 16, 2013 5:48:09 PM UTC+1, Aymeric Augustin wrote:
>  
>
>> While pour point is technically valid as far as request.GET and 
>> request.POST are concerned, in practice they're so commonly used as a 
>> metonymy for HTTP GET and HTTP POST that it's worth having a strong stance 
>> on keeping them separate.
>>
>
> I'm not entirely serious, but perhaps we should provide better names for 
> `request.GET` and `request.POST` at the same time (with compat). One 
> contains some parameters from the request URL (but can be provided on any 
> HTTP verb, not just GET), the other contains data from the request entity, 
> providing it comes in one of two convenient formats that have common usage 
> (and can be provided on various HTTP verbs, not constrained to POST). The 
> current names are misleading if you try to learn HTTP by learning Django 
> (and I'm guessing a lot of people do).
>
> Certainly the ?next / next= case pointed out above by Marc (a pattern I 
> use a fair amount) would read more precisely (in the absence of 
> `request.REQUEST`, which is a clunky, blurry, quite possibly misguiding but 
> technically no worse named convenience).
>
> I'm +1 on deprecating `request.REQUEST`, and maybe +0 on the rest of what 
> I've just said ;-)
>
> J
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a7af947a-8d78-4855-abaa-4625d896e0b6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Support POST of application/json content type

[Tom Christie]

I've created the ticket 21442 <https://code.djangoproject.com/ticket/21442>to 
track 'Configurable request parsing'.

I've also made a first pass at refactoring the current request parsing into 
a generic interface that can support arbitrary parsers,
and pushed the current status to the ticket_21442 
branch<https://github.com/tomchristie/django/tree/ticket_21442>on my GitHub 
account.

It's likely that the API will need a few design decisions along the way, so 
I'd suggest anyone who's interested in contributing to this work makes sure 
to follow the ongoing work in that branch.

Cheers all,

  Tom

On Thursday, 12 September 2013 11:20:07 UTC+1, Tom Christie wrote:
>
> > why keep data and files separated
>
> Mostly because that's the way it already works, so...
>
> * request.data would essentially provide a strict superset of the 
> functionality that request.POST provides.  In general you'd be able to 
> replace `request.POST` with `request.data` anywhere and seemlessly start 
> supporting JSON or other data without any other changes required.
> * Form expect the data and files to be provided separately which would be 
> awkward otherwise.
>
> > In the absence of strong objection, I will start working on this base. 
>
> Sure thing.  As it happens, I was also considering taking a crack at this 
> in the coming weeks, so please do follow up on this thread linking to your 
> repo if you start working on it, so myself and others can track any 
> progress.  (And perhaps collaborate.)
>
> Cheers :)
>
>   Tom
>
> On Wednesday, 11 September 2013 04:52:08 UTC+1, Stefan Berder wrote:
>>
>> On Tue, Sep 10, 2013 at 12:17 PM, S Berder  wrote: 
>> > On Tue, Sep 10, 2013 at 9:05 AM, Curtis Maloney 
>> >  wrote: 
>> >> 
>> >> On 9 September 2013 19:50, S Berder  wrote: 
>> >>> 
>> >>> Gents, 
>> >>> to sum it up, arguments made and details of how I see the 
>> >>> implementation of a response/request encode/decode framework: 
>> >>> 
>> >>> * need a pluggable interface so current content-types are supported 
>> >>> (`application/x-www-form-urlencoded`, `multipart/form-data`), new 
>> >>> types (`application/json`), custom and future types 
>> >>> (`application/vnd.foobar+json` anybody? See 
>> >>> 
>> http://developer.github.com/v3/media/#api-v3-media-type-and-the-future 
>> >>> for example, `application/msgpack`, `application/protobuf`, 
>> >>> `application/capnproto`, etc). 
>> >>> * decoder/encoder map (content-type, decoder) should be smart to 
>> >>> handle patterns like `text/*` or `application/*xml*` and match things 
>> >>> like `Accept: application/json, text/plain, * / *` 
>> >>> * choice of decoder would be made on the Content-Type header, maybe 
>> >>> supporting a raw by default so data is just passed in case of unknown 
>> >>> content type. 
>> >>> * decoder/encoder should be available through `request` and 
>> `response` 
>> >>> objects. 
>> >>> * decoded data structure (python object) would be stored in 
>> `request.data` 
>> >>> * first step is to support requests, next step is to handle responses 
>> >>> with the same pluggable functionality and coherent API. 
>> >>> * A sensible default for response Content-type would be `text/html; 
>> >>> charset=UTF-8`. It should be made available through a setting entry 
>> >>> anyway 
>> >>> 
>> >> 
>> >> You should also have access to the decision made by the data parser as 
>> to 
>> >> which parser was used, instead of having to infer it yourself from the 
>> >> content type header. 
>> > 
>> > Indeed, that's the 4th point of my list, maybe it's not clear as it is 
>> > but this would be supported. 
>> > 
>> >>> 
>> >>> Some questions though: 
>> >>> 
>> >>> * why keep data and files separated, I see no good reason for this 
>> >>> except mimicking PHP's structure. An uploaded file comes from a named 
>> >>> input, I hope to find it in request.data (why do a common structure 
>> >>> otherwise). I might be missing something but nothing indicates a real 
>> >>> need for this in django/http/request.py 
>> >> 
>> >> 
>> >> True, there's some added complexity [small as it is] in forms because 
&g

Re: HTTP PUT request

[Tom Christie]

Hi Mark,

Sadly Malcolm is no longer with 
us
.

There is a thread 
herefor
 dealing with request parsing which - if it makes it into core - would 
deal with customisable handling of content types on POST, PUT and other 
HTTP methods.  If you're interested in following this up, that thread would 
be the place to discuss things.

All the best,

Tom

On Sunday, 1 December 2013 05:38:00 UTC, Mark Brown wrote:
>
> Hey Malcolm, 
>
> Is this still the case?
> This response was five years ago, why would Django not allow access to PUT 
> and DELETE data? 
>
> On Friday, 10 October 2008 11:09:02 UTC+11, Malcolm Tredinnick wrote:
>>
>>
>> On Thu, 2008-10-09 at 14:13 -0700, DaveV wrote:
>> > Ahh - never mind - I misread the first post.
>> > 
>> > Still, it would seem helpful if PUT data was processed in a way that
>> > was more readily accessible, such as a PUT dictionary like the POST or
>> > GET ones.
>>
>> No, because it would be almost always wrong to do so.
>>
>> The point is that request.POST is designed for web-browser POST
>> submission, which means it's going to be data encoded as a form
>> submission (or a mime-multipart if it contains a file upload). Web
>> browsers are very restricted beasts. Normal web services encompass a
>> much broader range of domains and there's no concept of a "common"
>> format for uploads. You have to look at the content-type and act
>> appropriately. It could be an XML document (or some subtype), image
>> data, a word document... anything. The content is described in the HTTP
>> method. It would be incorrect to attempt to force any of those data
>> types into dictionaries and not particularly useful for Django to
>> special case one particular type that will, in practice, actually be
>> pretty uncommon (machine interacting web services tend to use more
>> structured formats for sending data than form-encoded, since they're
>> sending more complex data than simple forms).
>>
>> If you're doing REST-based web service stuff -- as opposed to just
>> interacting with a web browser -- you should ignore request.POST as well
>> for the same reasons unless you have a very well-understood, restricted
>> domain that happens to always send form-encoded data.
>>
>> Apologies for being unclear in my original post, although you seem to
>> have worked out my intention. I was trying to say that POST and PUT (and
>> OPTIONS and DELETE) are treated identically in that all the data is in
>> raw_post_data, not that there was an attribute for each method. The
>> latter isn't appropriate for general cases.
>>
>> Regards,
>> Malcolm
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/928cbe64-f0de-470e-bace-ff53db326227%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [GSOC] Weekly update

[Tom Christie]

 1) get_fields should return a list instead of a tuple
>>> Previously, get_fields would return a tuple. The main reason was 
related to memory and immutability. After discussing with Russell, we 
decided this endpoint should actually return a list as it is a more suited 
data structure.
>> Can you clarify the reasons for this choice? If it’s just the purity of 
using the “proper” data type, I’m not sure it beats the practicality of 
returning an immutable iterable and avoiding the cost of a copy or the risk 
of incorrect manipulation.
> This is a design decision, tuple performs better.

I'm confused - does it return a list or a tuple? Your previous answer 
mentioned that it was a list, but you later state "This is a design 
decision, tuple performs better.".

Given that it ought to be immutable, I can't see any reason for it *not* to 
be a tuple.

Great work on all of this!

All the best,

  Tom

On Sunday, 3 August 2014 15:48:04 UTC+1, Daniel Pyrathon wrote:
>
> Hi Aymeric,
>
> Thanks for writing back
>
> On Sunday, August 3, 2014 4:24:27 PM UTC+2, Aymeric Augustin wrote:
>>
>> On 3 août 2014, at 15:11, Daniel Pyrathon  wrote:
>>
>> *1) get_fields should return a list instead of a tuple*
>> Previously, get_fields would return a tuple. The main reason was related 
>> to memory and immutability. After discussing with Russell, we decided this 
>> endpoint should actually return a list as it is a more suited data 
>> structure.
>>
>>
>> Can you clarify the reasons for this choice? If it’s just the purity of 
>> using the “proper” data type, I’m not sure it beats the practicality of 
>> returning an immutable iterable and avoiding the cost of a copy or the risk 
>> of incorrect manipulation.
>>
>
> This is a design decision, tuple performs better. 
>  
>
>>
>> The bugs that would occur if the cached value is altered inadvertently 
>> could be very hard to track down.
>>
>
> Only a couple of days...
>
> https://github.com/PirosB3/django/commit/2411ff58d032c38a3151c1e54198723a86e6a8af
>  
>
>>
>> *2) Move tree cache out of the apps registry*
>> The main optimisation for the new API (described here 
>> https://code.djangoproject.com/wiki/new_meta_api#Mainoptimizationpoints) 
>> is currently storing information on the Apps registry. After discussing 
>> with Russell we agree this shouldn't be the correct place to keep this.
>>
>>
>> +1
>>
>> A solution is to store related fields information separately on each 
>> model (
>> https://github.com/PirosB3/django/pull/5/files#diff-98e98c694c90e830f918eb5279134ab9R275).
>>  
>> This has been done, and all tests pass.
>> Unfortunately, there are performance implications with this approach: we 
>> are storing a new list on each model regardless of if it has related fields 
>> or not. I will be discussing with Russell about performance tomorrow.
>>
>>
>> Is the extra cost incurred only at start up ? How large is it? If it 
>> isn’t too bad and and run time performance is unchanged, it could be 
>> acceptable.
>>
>
> Yes! only at start-up. Once cache warming has finished, everything should 
> be as fast (maybe faster, given that we are accessing properties on the 
> single Options instance).
>  
>
>>
>> *3) Revisiting field types and options*
>> Last week I opened this: 
>> https://groups.google.com/forum/#!topic/django-developers/s2Lp2lTEjAE in 
>> order to discuss with the community possible new field and option names. 
>> This week I have actually revisited the fields and options for get_field/s 
>> and I have come up with the following: (…)
>>
>>
>> The latest iteration looks clear, consistent and straightforward. I like 
>> it.
>>
>
> Thanks! Please feel free to comment with doubts or suggestions.
>  
>
>>
>> -- 
>> Aymeric.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dad7fb6e-9984-41aa-814e-d373c384b475%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [GSOC] Weekly update

[Tom Christie]

> This has been resolved by using the ImmutableList datastructure

Looks fine. Behaviourally it's the same as a tuple, but with more explicit 
errors if you attempt to modify it, which seems reasonable.

Given that `.get_fields()` and `.fields` return `ImmutableList`, presumably 
`.field_names`, `.concrete_fields` and `.local_concrete_fields` should do 
as well right?

> A tuple is *not* "just an immutable list".

Interestingly, that's a point where we'd differ. In my mental model a 
programming construct is defined *purely* by it's behaviour, with no room 
for any 'conceptual' or 'theroretical' difference - as far as I'm concerned 
an (unnamed) python tuple really *is* just an immutable python list.

I think any further conversation on that is out of the scope of this list, 
but I've heard both positions advocated, and figured it's an interesting 
point to note. :)

(And it does have the occasional impact on how we'd choose to write 
documentation examples etc...)

All the best & keep up the great work,

  Tom


On Wednesday, 6 August 2014 01:33:53 UTC+1, Russell Keith-Magee wrote:
>
>
> On Tue, Aug 5, 2014 at 12:54 AM, Łukasz Rekucki  > wrote:
>
>> On 4 August 2014 16:14, Daniel Pyrathon > 
>> wrote:
>> > Hi All,
>> >
>> > This has been resolved by using the ImmutableList datastructure
>> >
>> > 
>> https://github.com/PirosB3/django/blob/soc2014_meta_refactor_upgrade_flags_get_field_tree/django/db/models/options.py#L629
>> >
>>
>> But why? What's the benefit over using a tuple? ImmutableList is not
>> even a list, because it inherits from tuple.
>>
>
> Communication. 
>
> From a purist theoretical perspective, there shouldn't be any argument - 
> the data we're talking about is a list. Lists have homogeneous elements; 
> Tuples have heterogeneous elements, but have *positional* homogeneity. A 
> "Point" is a tuple, because element 0 of the tuple "means" the x 
> coordinate. A Database row is a tuple - The first element is the primary 
> key (an integer), second is the "name" column (a string), and so on.
>
> A tuple is *not* "just an immutable list".
>
> From a pragmatic perspective, tuples are faster to work with, because they 
> aren't carrying around all the baggage needed to support mutability. And, 
> in this case, there is a risk of an end-user accidentally mutating the 
> result of get_fields(), which, due to cache-related optimizations, means 
> there's a risk of side effects.
>
> So - in this case, at a theoretical level, we are dealing with an list of 
> homogeneous objects (fields) that must be either immutable, or copied every 
> time it is used. Copying is a bit expensive (not *completely* prohibitive, 
> but noticeable), so that means we need to look to immutability. At a 
> theoretical I'm not wild about the fact that ImmutableList subclassing 
> tuple - it should be subclassing *list* - but I'm willing to defer to 
> pragmatism. Given that we're dealing with a relative internal detail that 
> most users won't be exposed to, I'm willing to hold my nose and accept 
> optimised solutions over theoretically pure solutions in the interests of 
> *all* users having better performance.
>
> Yours,
> Russ Magee %-)
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dd0a4b3d-de5f-4107-8351-dfa2e8bee642%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Requiring GitHub login for actions on Trac

[Tom Christie]

Absolutely +1.

Clearly the most pragmatic choice.

On Thursday, 7 August 2014 13:43:45 UTC+1, Josh Smeaton wrote:
>
> I don't think "vendor lock in" is a good enough reason to avoid it. If 
> GitHub were to go away, the move to a new code platform would be the 
> greater problem. Also, nothing will be "lost". The old usernames will still 
> be there, they just won't be properly linked to your github username. I 
> don't think that's really a major concern either.
>
> > Finally, to be honest, I’d rather adjust Django’s tools to enthusiastic 
> > beginners than grumpy freedom extremists who refuse to use GitHub.
>
> +1
>
> On Thursday, 7 August 2014 16:49:00 UTC+10, Christian Schmitt wrote:
>>
>> I'm a little bit concerned about that.
>> First I'm using a different user on Trac than on Github, so everything I 
>> wrote so far will getting lost (not that bad problem for me), but I think 
>> there are many users who are in the same situation.
>>
>> The next thing is vendor lock-in. What will happen if Github don't have 
>> enough money? Then all usernames would need to migrate back or to another 
>> OAuth provider, then everything could be lost a second time.
>> Or that Github gets bad / mad.
>>
>> Currently we already live in a world were everything gets connected. And 
>> that is really awful. One must consider that Github is definitely a target 
>> for intelligence agencies. And I don't mean the NSA only. 
>> Maybe I'm a little bit too paranoid but at the current state of the 
>> internet we shouldn't try to connect everything, just it is easier to login.
>>
>>
>>
>>
>> 2014-08-07 8:46 GMT+02:00 Aymeric Augustin > >:
>>
>>> To be clear, I have a working implementation of GitHub OAuth that I can
>>> activate as soon as we reach a consensus.
>>>
>>>
>>>
>>> On 7 août 2014, at 02:43, Ben Finney  wrote:
>>>
>>> > −1. I am happy to agree to Django's BTS terms of use, not GitHub's.
>>> > Please don't make the former depend on the latter.
>>>
>>> I didn’t know our Trac installation had terms of use. So, are you
>>> volunteering to jump in and delete spam as it comes in? Or do you
>>> have an alternative proposal?
>>>
>>>
>>>
>>> On 7 août 2014, at 02:47, Shai Berger  wrote:
>>>
>>> > Today, it is possible to contribute to the Django project without a
>>> > Github account. I would like this to remain the case.
>>>
>>> This is possible but in a limited capacity. To be honest, I think that
>>> ship sailed when we moved to GitHub. We would have also moved
>>> issues there if GitHub’s tools were usable.
>>>
>>>
>>>
>>> On 7 août 2014, at 02:58, Andre Terra  wrote:
>>>
>>> > Most importantly, how would Django as a project benefit from this
>>> > choice other than reducing minimal spam?
>>>
>>> Did you just ask “how would Django as a project benefit from having
>>> core devs work on committing patches rather than fighting spam”?
>>>
>>> If you don’t already have a djangoproject.com account, you’re likely to
>>> give up on reporting a small bug just because it’s too complicated to
>>> log in. Considering our target demographic, GitHub OAuth would
>>> eliminate this problem.
>>>
>>> Also, if you’re trying to report a bug anonymously, you’re likely to be
>>> unable to pass the CAPTCHA, and also be unable to report it, because
>>> you’re still getting blocked by the CAPTCHA. See complaints:
>>> https://code.djangoproject.com/search?q=captcha&noquickjump=1&ticket=on
>>>
>>> Finally, to be honest, I’d rather adjust Django’s tools to enthusiastic
>>> beginners than grumpy freedom extremists who refuse to use GitHub.
>>>
>>> > A better solution would be to strengthen what it means to have an 
>>> identity
>>> > on djangoproject.com. Rather than restricting user actions to Trac, we
>>> > could motivate users to create something like a Django profile which 
>>> would
>>> > be used for Trac (among may other uses)
>>>
>>> We already have that: https://www.djangoproject.com/~aaugustin/
>>>
>>> > and could later be linked to any OAuth providers, including but not 
>>> limited
>>> > to GitHub.
>>>
>>> We don’t have that.
>>>
>>> > TL;DR Identity on djangoproject.com, Authentication linked to 
>>> multiple OAuth,
>>> > Authorization in Trac.
>>>
>>> Are you volunteering to do this work, and if so, when will it be done?
>>>
>>> > I hope that idea makes sense. I may be just babbling nonsense.
>>>
>>>
>>> I’m sorry, but ideas don’t matter nearly as much as execution here.
>>> We just need working tools — nothing fancy.
>>>
>>>
>>>
>>> On 7 août 2014, at 02:59, Josh Smeaton  wrote:
>>>
>>> > is it easy enough to support github oauth + the current trac auth 
>>> concurrently?
>>> > If a user chooses to go through the harder path, that's fine.
>>>
>>> It may be doable to provide two authentications endpoints, like /login 
>>> and
>>> /login/github. Trac just looks at REMOTE_USER and creates a session that
>>> lasts until you logout. I’ll look into it.
>>>
>>> That solves the “GitHub is evil, I don’t want to touch their bytes with 
>>> a six

Re: The greatest proposal yet: rename this damn group

[Tom Christie]

Options?...

* Django core
* Django core development

For comparison the Rails folks have "Ruby on Rails: Talk 
" and "Ruby on 
Rails: Core "

 - Tom

On Friday, 5 September 2014 08:58:19 UTC+1, Robert Grant wrote:
>
> I am one of the happy few who read the line about what this group is for 
> before I started posting. 
>
> However, that line, and the endless supply of people who think this is for 
> Django developers (see also: Java developers are generally considered to be 
> people* who develop in Java, not who develop Java), might be symptoms of 
> the fact that this group has a funny name for something that is both 
> *developed* and *developed in*.
>
> Can we rename it 
> ?
>  
> :) Some semi-serious suggestions (because I can't think of an obvious 
> alternative) :
>
> Django Masters
> Django Private
> Django Debate
> Django Internals
> Aymeric And Friends
>
>
>
>
>
> * Yes, they're still people.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ea6589a1-5b3d-4f20-8de2-bf50dfcaf941%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: The greatest proposal yet: rename this damn group

[Tom Christie]

>  If we rename the group, all those links break

Changing the group email would result in the group URL changing, so that's 
not an acceptable option.
Changing the group display name wouldn't change any links.

> At the end of the day, it's a name. It doesn't matter which name you pick 
- *someone* is going to interpret it incorrectly.

It is particularly liable to misinterpretation as it stands.

> I don't see any downsides to trying something like "Django developers 
(core development)

Something like that seems like a reasonable option as it would still be 
consistent with the email address, while being less ambiguous.

On Friday, 5 September 2014 18:45:58 UTC+1, Russell Keith-Magee wrote:
>
>
> Hi Robert,
>
> Thanks for the suggestion. 
>
> You're not the first to make the suggestion - the same suggestion has been 
> made many times in the past. You can search the archives if you want to see 
> those discussions.
>
> There are a number of problems with this proposal:
>
> 1) History. There are 10 years of archives and blog links that reference 
> existing discussions. If we rename the group, all those links break. That 
> would be a huge loss to the community.
>
> 2) More importantly, you can't solve a social problem with technology. At 
> the end of the day, it's a name. It doesn't matter which name you pick - 
> *someone* is going to interpret it incorrectly. To pick some of the 
> suggestions that have been made:
>
>  * "Django Masters" and "Django Private" both imply that newcomers aren't 
> welcome to join and make suggestions, which is a social signal we don't 
> want to send.
>  * "Django Core" could be interpreted to mean that it's only for the core 
> team - and, by the by, there *is* a "django-core" list for private core 
> team discussions (there aren't many of these, but it sometimes necessary 
> for security sensitive issues, etc)
>  * "Django debate" doesn't make it clear who is discussing what.
>
> Yes, I agree that the "you should be posting to Django-users" message 
> isn't ideal, and some of the responses that are given don't have the best 
> tone. If your first message to a Django group gets a response that sounds 
> like "go away", that's not a good look for the community.
>
> However, I *guarantee* that renaming the list won't make this problem go 
> away.
>
> Yours,
> Russ Magee %-)
>
> On Fri, Sep 5, 2014 at 12:58 AM, Robert Grant  > wrote:
>
>> I am one of the happy few who read the line about what this group is for 
>> before I started posting. 
>>
>> However, that line, and the endless supply of people who think this is 
>> for Django developers (see also: Java developers are generally considered 
>> to be people* who develop in Java, not who develop Java), might be symptoms 
>> of the fact that this group has a funny name for something that is both 
>> *developed* and *developed in*.
>>
>> Can we rename it 
>> ?
>>  
>> :) Some semi-serious suggestions (because I can't think of an obvious 
>> alternative) :
>>
>> Django Masters
>> Django Private
>> Django Debate
>> Django Internals
>> Aymeric And Friends
>>
>>
>>
>>
>>
>> * Yes, they're still people.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/fd27a285-be01-417f-ab4b-4026d7221239%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b29e5576-d10a-4274-a0fb-27ba4c45c5b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Should reverse() return a Unicode string?

[Tom Christie]

One point of clarity is that we ought to return the same type for each of 
`reverse`, `request.path`, `request.get_full_path`, `request.path_info`, 
and the values in the `request.GET` dictionary. Given that, the answer is 
clearly "it should be a string".

It's also a little unclear to me what type is currently expected from 
values in the `request.META` dictionary. I believe that the `HTTP_*` keys 
currently return byte objects, but it's not documented, or clear to me if 
that's consistently true.

On Thursday, 18 September 2014 22:11:39 UTC+1, Carl Meyer wrote:
>
> Hi Jon, 
>
> On 09/18/2014 02:01 PM, Jon Dufresne wrote: 
> > In my Django application, I'm making a strong attempt to always deal 
> > with Unicode strings at the application layer. Byte strings are only 
> > handled at the protocol layer -- sending data out on the "wire". If my 
> > application tests if an object is a string, I'll use isinstance(obj, 
> > unicode) (Python2). 
> > 
> > One gotcha that I noticed is that reverse() will always return a byte 
> > string. Tracing this through the code, I see this happens during the 
> > call to iri_to_uri(), as this function creates a string consisting 
> > only of ASCII characters, other characters are escaped. 
> > 
> > Now, reverse() is often used to grab a URL and handle it at the 
> > application layer. It is not reserved only for the protocol layer. An 
> > example would be presenting a URL inside a HTML template, (as an href 
> > or as text), mail, or JSON. 
> > 
> > In my opinion, reverse() should return a Unicode string, even if that 
> > string consists only of ASCII characters. It is not until the string 
> > hits the wire that it ought to be forced to bytes. 
> > 
> > To verify this, I have created a unit test that I placed in 
> > "urlpatterns_reverse.tests.URLPatternReverse" to demonstrate this is 
> > at the Django layer. 
> > 
> > def test_reverse_unicode(self): 
> > name, expected, args, kwargs = test_data[0] 
> > self.assertIsInstance( 
> > reverse(name, args=args, kwargs=kwargs), 
> > six.text_type) 
> > 
> > What do you think? If others agree, I can file a bug and create a pull 
> > request to fix this. 
>
> It makes sense to me that `reverse()` should return a text (unicode) 
> string. A URL may be "just bytes" on the network, but within the Django 
> context it is clearly text. 
>
> I'm a bit concerned about the backwards-compatibility implications, 
> particularly for Python 3 projects where `bytes` and `str` don't 
> silently interoperate. It would be really interesting to hear if anyone 
> on this list has a real-world Python 3 Django project handy and could 
> test the impact of this change. 
>
> Carl 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ec6d2143-2fe5-4f5f-8cbc-9e8236609a3f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django 1.7, KeyError: u'manager' error

[Tom Christie]

Hi Taenyon,

This list is for development on Django itself.
I see you've also asked the question on the `django-users` mailing list, 
which is the right mailing list for your question.
You'll want to follow up any further questions on the `django-users` thread 
rather than this one.

All the best,

  Tom

On Friday, 17 October 2014 10:18:04 UTC+1, Taenyon Kim wrote:
>
> I had a problem in using django 1.7.
> Example:
>
> class ModelA(models.Model):
> fielda = models.CharField(max_length=10)
>
> class ModelsB(models.Model):
> modela = models.ForeignKey(ModelA)
> fieldb = models.CharField(max_length=10)
>
> when I query like " modela_instance.modelb_set()', I will get KeyError: 
> u'manager'.
>
>
> 
>
> How can I resolve this problem?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c495532e-54da-42fb-baa1-730c20b55e62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django query in DetailView

[Tom Christie]

Hi Ronaldo,

This list is for the discussion of development *on the Django project 
itself*,
not for discussion of usage questions when developing Django sites.

You want the django-users mailing list, 
here: https://groups.google.com/forum/#!forum/django-users

All the best,

  Tom

On Wednesday, 29 October 2014 16:20:02 UTC, Ronaldo Bahia wrote:
>
> *This question is also in 
> http://stackoverflow.com/questions/26635406/django-query-in-detailview
>
> I have DetailVIew wich returns a list of related objects (m2m throught). 
> It works just fine!
>
> But I need to search for objects'names and it is returning all objects 
> instead of only the related ones.
>
> How can I approach this?
>
> Thanks.
>
> #models.pyclass Candidate(models.Model):
> user = models.OneToOneField(User, primary_key=True)
> birth = models.CharField(max_length=50)
> ...
> class Job(models.Model):
> candidate = models.ManyToManyField('Candidate', through='CandidateToJob')
> title = models.CharField(max_length=500)
> ...
> class CandidateToJob(models.Model):
> job = models.ForeignKey(Job, related_name='applied_to')
> candidate = models.ForeignKey(Candidate, related_name='from_user')
> STATUS_CHOICES = (
>('1', 'Not approved'),
>('2', 'Approved'),
>('3', 'Hired')
> )
> status = models.CharField(max_length=2, choices=STATUS_CHOICES)
>
> My search query
>
>  #views.py
>  def get_query(query_string, search_fields):
> query = None # Query to search for every search term
> terms = normalize_query(query_string)
> for term in terms:
> or_query = None # Query to search for a given term in each field
> for field_name in search_fields:
> q = Q(**{"%s__icontains" % field_name: term})
> if or_query is None:
> or_query = q
> else:
> or_query = or_query | q
> if query is None:
> query = or_query
> else:
> query = query & or_query
> return query
> def searchcandidate(request, *args, **kwargs):
> query_string = ''
> found_entries = None
> if ('q' in request.GET) and request.GET['q'].strip():
> query_string = request.GET['q']
> entry_query = get_query(query_string, ['candidate__user__first_name', 
>'candidate__user__last_name'])
>
> found_entries = 
> CandidateToJob.objects.filter(entry_query).order_by('-candidate')
>
> return render_to_response('dashboard/candidates_results.html',
> { 'query_string': query_string, 'found_entries': found_entries },
> context_instance=RequestContext(request)
> )
>
> The view with the list of objects(candidates)
>
> class Screening(generic.DetailView):
>
> model = Job
> template_name = 'dashboard/screening.html'
>
> def get_context_data(self, **kwargs):
> context = super(Screening, self).get_context_data(**kwargs)
> context['candidate_list'] = 
> self.object.applied_to.all().order_by('candidate')
> return context  
>
> My urls
>
> #urls.py
> url(r'^dashboard/job/(?P\d+)/screening/$', views.Screening.as_view(), 
> name='screening'),
> url(r'^dashboard/job/(?P\d+)/screening/results/$', 
> 'companies.views.searchcandidate', name='searchcandidate'),
>
> And the templates
>
> #html for Screening(DetailView) in which works good method="get" action="{% url 'searchcandidate' job.pk %}">
> 
>  placeholder="Buscar candidatos por nome" />
> 
> 
> ...

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/02785977-14fc-403e-81d4-d784bf497147%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Django Content Negotiation implement

[Tom Christie]

There's no particular timeline for it. The two people most likely to progress 
it are Mark Lavin and myself.

Mark is the author of the DEP and has been busy lately, focusing on releasing 
the "Lightweight Django" book that he's co-authored. He has mentioned that it's 
possible he'll get back to progressing the DEP following on from that.

My priority at the moment is the REST framework kickstarter, perhaps I'd get 
onto the DEP after that, tho as things currently stand I'd prefer if someone 
else was going to be primarily responsible and for me to just help with design 
guidance.

Anyone else is also free to get the ball rolling on it - we've got a very clear 
idea what we need, so it just needs someone to make the commitment to it, in 
which case I'd be very willing to also put some time into guiding it through.

Hope that helps answer your question. Contributions are of course very welcome,

All the best,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c8068f76-7ea0-4a3e-aac3-a9ff58036c7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Content Negotiation implement

[Tom Christie]

> I'll be trying to enhance my knowledge base about django request response 
and other related stuffs necessary if you please show me some light to get 
started or from where actually should I start.

I'd generally think that a decent place to start would be to write up what 
some of the documentation *before* actually implementing any code.
Figure out where would you describe `request.data` in the Django docs, 
figure out what other bits of the DEP would need to be public documented 
API, ignoring the under-the-hood implementation details what does the user 
need to know about them and how they work?

An alternative way to get involved without necessarily diving right into 
the DEP might be to review any outstanding tickets to do with HTTP handling 
and requests and responses. Even if you don't feel able to resolve any of 
those issues having a browse through and trying to understand them would be 
a good way to start getting to grips with the Django codebase.

Hope that helps,

  Tom

On Sunday, 9 November 2014 13:36:53 UTC, Asif Saifuddin wrote:
>
> Thank you both for your kind response.
>
> I have only almost about a year working with django and still learing. My 
> django and python skills are still so limited as far I know. Reading the 
> dep and other related links of discussions and browsing some source code 
> things are bit overwhelming to me right now :p  but I am willing to expand 
> my technical knowledge  by learning and trying to contribute on this part.
>
> I have some limitations in design decisions and understanding the overall 
> implementation process But still I want to start working on this part 
> primarily with you guys helps and suggestions.
>
> I'll be trying to enhance my knowledge base about django request response 
> and other related stuffs necessary if you please show me some light to get 
> started or from where actually should I start.
>
> Kind Regards
>
> Asif 
>
> On Sun, Nov 9, 2014 at 4:31 PM, Asif Saifuddin  > wrote:
>
>> Thank you both for your kind response.
>>
>> I have only almost about a year working with django and still learing. My 
>> django and python skills are still so limited as far I know. Reading the 
>> dep and other related links of discussions and browsing some source code 
>> things are bit overwhelming to me right now :p  but I am willing to expand 
>> my technical knowledge  by learning and trying to contribute on this part.
>>
>> I have some limitations in design decisions and understanding the overall 
>> implementation process But still I want to start working on this part 
>> primarily with you guys helps and suggestions.
>>
>> I'll be trying to enhance my knowledge base about django request response 
>> and other related stuffs necessary if you please show me some light to get 
>> started or from where actually should I start.
>>
>> Kind Regards
>>
>> Asif 
>>
>>
>> On Saturday, November 8, 2014 3:01:28 PM UTC+6, Asif Saifuddin wrote:
>>>
>>> Hi,
>>>  when https://github.com/django/deps/blob/master/drafts/
>>> content-negotiation.rst this dep is to be implemented in django? 1.8?
>>>
>>> Regards
>>>
>>> Asif
>>>
>>  -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/django-developers/g1P7H1PhZaY/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to 
>> django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/436d3c71-2567-462d-a035-9880511124ab%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/befd06ef-b4ff-4958-9e7c-c2420b009349%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Mptt Admin as an inline form

[Tom Christie]

Hi Neeraj,

This group is for discussion of development of Django itself.
You want the 'Django users' group, 
here... https://groups.google.com/forum/#!forum/django-users

All the best,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b61ef35a-8ef8-4aa8-92a3-f390d7a4f27e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Explicit relative imports

[Tom Christie]

Contrary voice here, but I don't dig explicit relative imports, it's just 
adds an extra decision point for no real benefit.

Personally I always recommend full absolute imports, ordered strictly 
alphabetically - there's then never any room for confusion or decision 
making around how the imports should be written, and it's always tidy and 
consistent.

Not looking to necessarily change the Django project's stance on that, but 
that's the style I use throughout my projects and it's trivial to stick to. 

Cheers,

  Tom

On Wednesday, 12 November 2014 21:59:42 UTC, Jannis Leidel wrote:
>
>
> > On 11 Nov 2014, at 22:51, Aymeric Augustin <
> aymeric@polytechnique.org > wrote: 
> > 
> > Hello, 
> > 
> > We’ve started using explicit relative imports in newer parts of the 
> Django source tree. They’re short and readable. That’s good. 
> > 
> > I would like to add guidelines about imports in the coding style guide 
> in order to improve consistency. 
> > 
> > My inclination would be to recommend relative imports within 
> “components” but avoid them between “components”, where a component is: 
> > 
> > - a well-defined sub-framework (django.core.cache, django.db, 
> django.forms, django.template, etc.) 
> > - a contrib app 
> > - an app in the tests/ directory 
> > - etc. 
> > 
> > I would discourage going back into parent modules with relative imports 
> because statements such as `from ...spam import eggs` are hard to parse. 
> > 
> > You can see an example of this style in django.apps which has only three 
> files. 
> > 
> > What do you think? 
>
> Yup, the way to go. 
>
> Jannis

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7f1a5323-1efd-4891-bc02-eab5d67bc8b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Delivering PR 3114 - The _meta refactor

[Tom Christie]

One thing I'm unclear on: Does the new API contain any (public API) way of 
determining if a relationship has an associated through table?

Entirely possible that I'm being stupid and that's outside the scope of the 
API, but it's the one thing I see in Django REST framework's ModelSerializer 
'_meta' inspection code that's not obviously handled here.

Is it something that needs to be covered? (Of course if we're talking about 1.8 
blockers I've no particular problem if the answer needs to be 'maybe but could 
come later')

Cheers,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ab90dc57-fd67-421e-94f4-60829b5af210%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Delivering PR 3114 - The _meta refactor

[Tom Christie]

Thanks Russ, appreciate your time.

I've no great problem with keeping 'through' table API undocumented with 1.8, 
probably doesn't matter too much either way. Just flagging it up, but looks 
like 'no change' there, which is okay.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f52d9809-18dd-4ec8-afcd-1bcef71cc968%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Last call: Meta API (GSoC 2014/PR 3114) is ready for commit

[Tom Christie]

Wonderful - great work Daniel and thanks to everyone else for their hard 
work on review & guidance.
Lovely stuff.

  Tom

On Monday, 5 January 2015 05:12:41 UTC, Russell Keith-Magee wrote:
>
> Hi all,
>
> Following up on my pre-Christmas email - I believe that PR 3114 - the 
> formalisation of _meta - is now ready to land.
>
> https://github.com/django/django/pull/3114
>
> The code has received a detailed review by several members of the core 
> team, including Tim, Carl, Collin, and myself; we've had review notes from 
> many others. The documentation has also been reviewed. The issues raised by 
> those code and documentation reviews have all (to the best of my knowledge) 
> been addressed.
>
> Other than last-minute cleanups (e.g., PEP8 compliance, -Wall checks etc), 
> and the conclusion of one discussion about how to express one particular 
> set of relationships in code, I believe the branch is ready to land.
>
> Unless there are objections, my intention is to land this patch later this 
> week (around Wednesday/Thursday UTC). If anyone has any objections, speak 
> now, etc etc.
>
> Once again, a huge thanks to Daniel for his great work over the summer 
> (and continued work into the winter as we refined the patch for trunk), and 
> to everyone else who has contributed to reviewing his work.
>
> Yours,
> Russ Magee %-)
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/51034deb-f021-48bd-a579-ecc37cf64703%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Settings: lists or tuples?

[Tom Christie]

Hi Andreas,

  I'm completely in agreement with you that *in theory* using tuples would 
be a (very marginal) improvement. I also happen think that the idea that 
tuples are semantically different from simply being  immutable lists is a 
nonsense - regardless of what a particular section of documentation may or 
may not say, language features are defined solely in terms of their 
behavior, they do not have dreams, hopes and grand philosophical outlooks 
on how they should be used.

  However, the simple case that tuples have a far poorer syntax in python 
than lists, and are *very* easy for beginners to get wrong is a good enough 
reason to prefer the usage of lists consistently.

  ("a string")

  ("a tuple",)

Beginners will not be following your "I notate every (globally available) 
constant sequence in the pattern" advice, and no amount of documentation is 
sufficient to prevent it being a very simple and easy error, that might be 
difficult for a beginner to track down. I think that's a bigger and easier 
trap than intentional assignment.

> accidental assignment to the settings object itself could be easily 
prevented with a __setattr__ method on its class

I'd suggest treating that as a separate issue - perhaps if you or someone 
else came up with a pull request that enforced immutability of the settings 
object that'd be considered on it's own merits. (Note that you could 
perfectly well also deal with making translating list and dict objects into 
immutable objects at *that* point of API, even if they're not in the 
settings module.) I haven't given that any great thought, so expect it 
would have it's own set of obstacles to overcome, but I think it's a 
different issue to the topic at hand here, which is really just about 
settling on an acceptable and consistent style.

> maybe a compromise would be to explicitly note in the docs

I'd be against that as unnecessary fluff - doing one thing in the code and 
recommending another in the docs just introduces noise and uncertainty.

The topic's been discussed more that it really deserves, but I understand 
that it can be frustrating if it feels like your reasoned arguments are 
being brickwalled. I wanted to at least contribute and note that I do agree 
with you in theory, even if practically I'd say that lists throughout is 
consistent, clear, and slightly less likely for developers to get wrong.

Cheers,

  Tom


On Tuesday, 20 January 2015 17:52:51 UTC, Andreas Kahnert wrote:
>
> Just for completness: accidential assignment to the settings object itself 
> could be easily prevented with a __setattr__ method on its class, since 
> django yields on various other places about configuration problems it could 
> not be wrong if the programmer gets noted about an illegal assignment. If 
> everything works fine the method will only get called during startup, so 
> there is no runtime overhead. Simplified example:
> def __setattr__(self, key, val):
> if self.configured:
> raise Exception('settings can not be changed after server startup')
> super(LazySettings, self).__setattr__(key, val)
>
> @Carl Meyer: At the first hand you're right, a thing all programmers 
> should know about (if they call themself so), but he assumed there existed 
> some kind of copy-on-read semantic for the settings, because you get 
> something different when imported from django.conf instead directly and 
> because it's a "magical" lazy object.
>
>
> But since you all seem to like lists that much, maybe a compromise would 
> be to explicitly note in the docs that there is a danger in using lists 
> which can be prevented by tuple usage.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/063919e0-8d5e-4bfc-b6dc-429885fcfbc6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Settings: lists or tuples?

[Tom Christie]

> So, I've been working on a patch ... Suggestions?

Normalizing on lists as the default throughout is good. Warning about using 
tuples and putting their usage on a deprecation path seems a little 
unnecessary. I don't think that was discussed in this thread, but I've only 
scanned through, so I could be wrong.

If nothing else let's at least treat those as two *separately reviewable* 
pull requests.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c3d819ae-bbcd-4ab1-b5a4-9ae5fab01fc0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: resurrecting an old friend, whitespace in forms, #6362

[Tom Christie]

Trimming at `request.POST` or at the `Form` layer absolutely isn't 
sensible, and a `normalize` argument is probably just overcomplicating 
things, but I got the impression from that ticket that nobody would have 
any great issue with someone issuing a patch with a `trim_whitespace` flag 
on CharFields and TextFields.

The only thing beside then to resolve would be if we'd be okay with *its 
default being True*, despite the (edge case) backward incompatibility 
that'd cause. That seems to me to be simple and desirable behavior, doesn't 
need any extra settings, and you'd only need to override that argument in 
the bizarre cases where you *don't* want to strip leading and trailing 
whitespace.

Granted that ticket's been around for an *awfully* long time, but it's only 
a case of someone actually acting on it, and seeking a consensus on a 
sensible option, so...

Cheers,

  Tom


On Tuesday, 3 February 2015 17:22:38 UTC, frantisek holop wrote:
>
> good day, 
>
> a recent technical support incident conducted remotely and 
> involving a lot of back and forth of "huh? but i have entered 
> what you sent me" left me my head scratching. 
>
> the reason turned out to be a trailing space in the username of 
> the django admin loginform (thank god for nginx's "$request_body" 
> log_format parameter). 
>
> this of course sent me on an archeological journey into the lands 
> of stackoverflow, blogs, and finally #6362 and this mailing list. 
> it has been some 5 years now since the decision on that 
> controversial ticket. 
>
> i also went through the whole emotianal rollercoaster: 
>
> how come, so many batteries are included, but when it comes to 
> this essential POST best practice, i needed in every single 
> webform i have ever made, and now i have to do it myself? 
> error-prone and not DRY. especially for the admin login form, 
> it is a usability issue. 
>
> vs 
>
> the BDFL is right, silently discarding user input is just wrong. 
> just use a package like happyforms[1], or pick a stackoverflow 
> answer and be done with it. 
>
>
> but wait, then HTML is also wrong, because it silently folds all 
> whitespace into 1 piece of space, we are all used to this.  even 
> if the user entered whitespace is saved, pushing it back onto the 
> web will silently corrupt it (unless taken care of).  i am not 
> saying this requirement does not exist for someone, somewhere, 
> but i have yet to see a site in the wild that needs this (hello, 
> ascii art people).  whitespace in fields was always reserved for 
> government sites :) 
>
>
> it seems to me that there is a vocal group (majority?) that would 
> welcome a simple switch to make whitespace go away _now_, instead 
> of waiting for that perfect solution lurking in the future along 
> the lines of a generic normalize kwarg, or a flag on _every_ 
> {Char,Text}Field on the model or overriding form.fields 
> attributes like required. 
>
> apps that need to preserve the whitespace are the exception, 
> not the rule, and that is why i would prefer not to start every 
> project by overriding BaseForm._clean_fields[2]. 
>
> so i would like to present another idea for a possibe solution, a 
> proposal i have not seen so far: to have a global setting like 
> FORM_STRIP_FIELDS=True or some such and then roughly: 
>
> diff --git a/django/forms/forms.py b/django/forms/forms.py 
> index c9b8cf2..aab737a 100644 
> --- a/django/forms/forms.py 
> +++ b/django/forms/forms.py 
> @@ -8,6 +8,7 @@ from collections import OrderedDict 
>  import copy 
>  import datetime 
>   
> +from django.conf import settings 
>  from django.core.exceptions import ValidationError, NON_FIELD_ERRORS 
>  from django.forms.fields import Field, FileField 
>  from django.forms.utils import flatatt, ErrorDict, ErrorList 
> @@ -355,6 +356,8 @@ class BaseForm(object): 
>  if isinstance(field, FileField): 
>  initial = self.initial.get(name, field.initial) 
>  value = field.clean(value, initial) 
> +elif isinstance(value, basestring) and 
> settings.FORM_STRIP_FIELDS: 
> +value = field.clean(value.strip()) 
>  else: 
>  value = field.clean(value) 
>  self.cleaned_data[name] = value 
>
> i know, it is a big hammer, but for me, it is like the timezone, 
> or csrf, i'd like to just set it, and forget about it. 
>
> -f 
>
> [1] https://pypi.python.org/pypi/happyforms 
> [2] http://chriskief.com/2012/12/21/trim-spaces-in-django-forms/ 
> -- 
> nobody can be exactly like me.  even i have trouble doing so. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.g

Re: resurrecting an old friend, whitespace in forms, #6362

[Tom Christie]

> it will be backwards incompatible for every Django installation out 
there, but also because throwing away data that the user actually entered 
should be an opt-in, not opt-out behavior.

If adequately called out I think there'd be a valid case that the current 
and future issues it'll be causing to applications right now would outweigh 
the risk of compatibility breakages. I can see a couple of cases where I 
might not want stripped whitespace, but only in slightly contrived and edge 
case situations.

Validating and normalizing input at the field level in a way that supports 
the common case by default seems like a good plan to me. I'm not sure I see 
any great difference between a user entering "3.1" in an integer field and 
having 3 returned is so very different from having "hello " return "hello" 
on a char field. And we're not of course throwing anything away at the 
`request.POST` or `Form` layer - it's only once we're validating an 
individual field that the normalization is being applied.

I don't have an great investment in this either way around, but I think 
it's a valid user-focused improvement worth considering *if* someone 
actually puts together a pull request for it.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f62ee326-11a8-447b-a19c-3028b6e7453f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: resurrecting an old friend, whitespace in forms, #6362

[Tom Christie]

> leaving a ticket open is a better signal for inviting discussion and 
patches.

There's been over 22,000 issues raised in Django's history. It's 
understandable and desirable that maintainers close off tickets 
aggressively if they don't think they should be tackled.

I wouldn't get too hung up on the process there - a sound argument and a 
bit of proactive work is all that's needed to convince folks to take the 
time to reassess something like that issue. Personally I think there's a 
good case to be made for this one so from my point of view, so I'd say go 
ahead, make the proposal, or better still a pull request, make the argument 
and *then* see if it is possible to reach a consensus.

There's clearly absolutely no guarantee it'd be accepted (there's two core 
members in this thread making different judgement calls on it, so it's 
*far* from cut & dried) but you can be sure that folks are willing to put 
the time in to listen to anyone willing to put in a bit of legwork.

Cheers,

  Tom :)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/14bdc9b2-3cd5-4ca8-945b-891884da7c5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Postgres specific fields and third party apps.

[Tom Christie]

I'm not sure if there'll be an actionable outcome of this, but wanted to 
raise it for discussion.

We've started supporting some of the PostgreSQL specific fields in Django 
REST framework, and I ran into a roadblock when adding a new testcase.
Unsurprisingly we're not able to test them without also switching the test 
database to PostgreSQL.

As a third party app that's not something I want to have to do - it'd mean 
a bunch of work, an extra dependency, slower tests, and (unacceptablly) 
more complication for our contributors.

The only way I can see to resolve it would be if we had support for those 
fields with the sqlite backend.
Technically I assume that's feasible - use JSON encoding for storing the 
data (probably easy), and do awfully non-optimal in-memory filtering for 
the field-specific lookups (probably very hard).
There'd also be the obstacle requiring someone with the time and motivation 
to make it happen.

Options I can see:

* The status quo - don't support fields these in SQLite. (I assume this is 
by far the most likely.)
* Support the fields in SQLite, but don't support the lookups. (This'd 
actually hit the sweet spot for me - I can still test against these fields, 
and test cases will run just fine, so long as you're not hitting any field 
specific fitlering in querysets.)
* Support the fields and lookups in SQLite. (Sounds highly unlikely.)

Any thoughts?

Cheers!

   Tom


Aside: I'm not interested in the "you should be testing with the same 
database as you use in production" gumpf here. It's a third party app.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f08447af-395c-4fb0-9f01-d2af3b4b8504%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Postgres specific fields and third party apps.

[Tom Christie]

Thanks Marc, that's helpful input.

> I would probably have the motivation to create a PR for DRF showing a 
possible travis/testing setup for what I have proposed.

No, don't worry about that. We actually don't have any need to *hit* the 
database in those tests at all, we just need to make sure that the model 
fields get correctly mapped to the equivalent serializer fields. There will 
be mocking-type approaches to those tests that mean less work than 
switching the CI database.


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/334b3409-ba79-4f6a-bcef-6875c229fd3c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: JsonResponse and list values

[Tom Christie]

I haven't dug out sources on this, but I think that vulnerability had been 
dealt with a long time ago, and isn't an issue on browsers with any real market 
share today.

Would need to double check though - don't remember where I came across it 
previously.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7b051285-8c64-43b4-b582-b8db6c453f60%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Extending the URL Dispatcher (GSoC 2015 proposal)

[Tom Christie]

> E.g., flask uses a simple `` to match an integer and capture it 
in the `id` parameter. Support to check for conflicts would be a lot 
simpler with those patterns. It would definitely be a best-effort feature.

>From my point of view, this by itself would make for a really nicely-scoped 
GSoC project.
Being able to demonstrate an API that allowed the user to switch to a URL 
resolver that used that simpler style would be a really, really nice 
feature,
and also feels like it might actually be a manageable amount of work.
This wouldn't *necessarily* need to allow decorator style routing, instead 
of the current URLConf style, but that might also be a nice addition. 
Personally though I would consider tackling that as an incremental 
improvement.

Things I'd be wary of:

* Anything around "continue resolving" exceptions or object inspection 
during routing, both of which sound like an anti-pattern to me.
* Method based routing. Feel less strongly about this, but still not 
convinced that it's a good style.
* Generic views / model routes / automatic routing. Too broadly defined, 
and with no clear best answer.

Anyways, interesting stuff Marten, look forward to hearing more.

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bc559676-60d9-48e1-bbe0-7abb52397184%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Admin New Look

[Tom Christie]

Amazing work so far. :)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cdfb54ac-0d0b-4367-b904-0b3c32235050%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fellow Report - July 11, 2015

[Tom Christie]

Consistently bloody amazing.
Thanks for doing such a great job.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1f9ff3f3-32cf-4adb-9c9d-9d9f51b295d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: #25227 Add utility method `get_updated_model()` to `ModelForm`

[Tom Christie]

I'm not super convinced that form.instance is widely better than 
form.save(commit=False).
That's even more true if we're not sufficiently convinced by it that we'd 
be deprecating the existing style.

It would:

* Promote making views making in-place changes on the instance -> doesn't 
tend to be a good style.
* Expose the not-quite-an-instance without m2m attributes in what feels 
like a more casual way than the more verbose form.save(commit=False).
* Allow for (mostly) erroneous use of form.instance.save()

Don't feel that strongly about it, and wouldn't be able to make a proper 
judgement call in the absence of a pull request, but it's not terribly 
clear to me what we're gaining here. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a29ef366-4179-4ca9-865d-baac283a5a5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Admin New Look

[Tom Christie]

Could we give users guidance on overriding the admin's application specific 
static files with global static files if we do this?

Not immediately clear to me what order static files are copied in, but 
presumably it's possible to ensure that the jquery static files included in 
the admin app directory are overridden with user specified ones in a 
project-level static files directory, right?

Also, tho I'm broadly in favor with us staying with up to date with the 
currently active version of jQuery, to clarify: is there anything 
*specific* that we'd be gaining from the switch at this point?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/972b31f8-f085-4b42-80fa-ea49abf40721%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: request for API review of streaming responses additions

[Tom Christie]

Potential there to treat these as separately reviewable pull requests.

For example - needing streaming template generation doesn't *necessarily* 
imply needing streaming responses. The node-by-node rendering might mean 
imply that less memory is used during the template rendering, even if the 
complete response content all ends up in memory. (I haven't got a handle 
from the pull request on if that'd be the case or not, and it's possible 
that I've misrepresented it, and there's no benefit other than being able 
to stream the output bytes)

More generally: As presented there's lots of technical change, with very 
little simple end-user presentable "why and when this is a benefit?".

* Template.stream()

What's the data to back this up?
Does this result in lower memory usage during the template generation, or 
is there no difference aside from allowing the output to be streamed? If 
there is an internal benefit then how large do those templates need to be 
before that's significant?

* StreamingTemplateResponse

Again, where's the data to back this up? We're introducing a decision point 
for our users without giving them any basis on which to make that decision. 
At what point would I likely want to use this, and what real-world cases 
for our users are driving this? (For really large CSV responses are 
templates a good idea in any case?)

I also don't really understand from the documentation how the behavior for 
SimpleStreamingTemplateResponse and StreamingTemplateResponse differs - 
"does not handle any rendering attributes/methods (including callbacks)" - 
I understand what the callbacks refer to there, but what does the rest of 
that mean?

* django.views.generic.base.StreamingTemplateView

Unclear to me that the decision point this view introduces to users is 
worth the benefit it provides.
Writing a view that streams a simple template is just about as simple a 
view as is possible to write in Django - would it be better if we simply 
took an informed decision on which of TemplateView / StreamingTemplateView 
is likely to be a better default on behalf of our users?

Also lukewarm on introducing a new GCBV that actually just requires a 
one-line addition to an existing GCBV. Should we prefer to instead document 
the usage of `response_class` more fully, rather than extending the API 
surface area?

Apologies for jumping in with the criticisms - intended positively! 
Understand the intent behind this, but a bit wary of changes that come 
across as purely technical with very little in the way of concrete 
demonstration of benefit. Think it'd be good to see it approached from a 
perspective of "what are we actually recommending our users do here?"

Cheers & thanks to everyone for their hard work!

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8120693f-652f-4767-909b-90a62505f297%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Why leave explicit null=True when blank is True in IntegerField, DateField etc?

[Tom Christie]

That kind of implicit behavior would just leave things more confused.
Writing a few characters less code isn't a benefit if it comes at the 
expense of making the behavior less obvious.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0e56558e-0efd-467f-8b08-75e20c5e6fae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: moving the class-based View base class out of django.views.generic?

[Tom Christie]

Yup, I think that's a good proposal.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/61cbcc8c-08d9-42af-83d5-4a8ee3d79dac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: status of 1.9 release blockers

[Tom Christie]

Given that it addresses such a core component I'd probably rather see it 
deferred to 1.10.

I'd hope that doesn't affect the motivation of the author (it's a fiddly 
bit of work to get right and its good to see it being addressed) but from 
my point of view it'd be better to see it really thoroughly reviewed before 
commit.

As one example it's not clear what the preferred resolution of this point 
 should be. 
Should the `DATA_UPLOAD_MAX_MEMORY_SIZE` guard include accessing 
`request.body` directly or not? (Related to Tim's point re. documentation.)

As I say, its good work, and I'd like to see it reach completion, but its 
not the sort of change we should be rushing.

(I wouldn't oppose a contrary judgement on it tho)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/54acf290-b08c-487b-ad2d-962bd26e8cb9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: status of 1.9 release blockers

[Tom Christie]

To back that up I'll make a formal commitment to helping review & ensure 
completion of the PR if it *does* get deferred to 1.10.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/39828781-b20f-46b7-a49d-7a18554a1b46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.