Re: Moving forward with Serialization.

[Tom Christie]

Thanks Piotr,

> But when serializing also Field is saved with data so it's not so clean.

I'm not sure whether it's an issue or not.  As with Django's form fields, 
the fields are copied off of base_fields, so saving state 
isn't necessarily a problem.  I'd agree that it's possibly not the cleanest 
part of the API, but it might be good enough.

> For each format there is Serializer class which is made 
from NativeSerializer ... and FormatSerializer

That would work.
I was planning on taking a slightly different approach in order to merge 
the work into Django.
In my opinion the `SERIALIZATION_MODULES` setting wouldn't make much sense 
once serialization and parsing/rendering are decoupled.  Furthermore it 
places some very specific, slightly odd, and 
undocumented constraints on the interface that serialization modules must 
provide.  Eg.  They must include a class named 'Serializer' and a class 
named 'Deserializer'.

My preference would be to put that setting on the road to deprecation, 
replacing it with something more appropriate, eg:

SERIALIZER_CLASS='django.core.serializers.FixtureSerializer'
RENDERER_CLASSES=(...)
PARSER_CLASSES=(...)

It'd be possible to do that in a way that'd be consistent with Django's 
deprecation policy, if the initial release continued to check 
if SERIALIZATION_MODULES is set, and use that in preference to the new 
style settings.

The `serialize`, `deserialize` and `get_serializer` functions continue to 
exist and would be light backward-compatible wrappers around the core of 
the new-style serialization.
 
Having said that I could probably live with either approach.

I guess it would probably make sense for me to pull django-serializers into 
a fork of django at some point, to better demonstrate what I'm suggesting.

> * IMO there is bug related to xml. All model fields must be transform to 
> text before xml serialization. In current django serialization framework 
> field's method value_to_string is responsible for this.

Thanks.  I've not looked into that yet.   It sounds feasible, though I 
think it'd have to only be affecting custom model fields in xml, and should 
be easy enough to fix.  If it's easy for you to provide an example case 
where django-serializer's output differs from the existing fixtures it'd be 
much appreciated.

Regards,

  Tom


On Saturday, 1 September 2012 12:39:32 UTC+1, Piotr Grabowski wrote:
>
> W dniu 31.08.2012 10:25, Tom Christie pisze: 
> > > I personally think that Forms are already the place that should 
> > handle (de)serialisation. They already serialise to HTML: why should 
> > they not be able to serialise to other stream types? 
> > 
> > Conceptually I agree.  As it happens django-serializers is perfectly 
> > capable of rendering into HTML forms, I just haven't yet gotten around 
> > to writing a form renderer, since it was out-of-scope of the fixture 
> > serialization functionality. 
> > 
> > Pragmatically, I'm not convinced it'd work very well.  The existing 
> > Forms implementation is tightly coupled to form-data input and HTML 
> > output, and I think trying to address that without breaking 
> > backwards compatibility would be rather difficult.  It's maybe easy 
> > enough to do for flat representations, and pk relationships, but 
> > extending it to deal with nested representations, being able to use a 
> > Form as a field on another Form, and representing custom relationships 
> > would all take some serious hacking.  My personal opinion is that 
> > whatever benefits you'd gain in DRYness, you'd lose in code 
> > complexity.  Having said that, if someone was able to hack together a 
> > Forms-based fixture serialization/deserialization implementation that 
> > passes the Django test suite, and didn't look too kludgy, I'd be 
> > perfectly willing to revise my opinion. 
>
> I am not quite sure but I think Forms should be build based on some 
> serialization API not opposite. Forms are more precise way of models 
> serialization - they are models serialized to html (specific format) 
> with some validation (specific actions) when deserializing. 
>
>
> I like Tom's django-serialziers but there are some things that I want to 
> mention: 
>
> * Process of serialization is split to two parts - transformation to 
> python native datatype (serializer) and next to specific text format 
> (renderer). But when serializing also Field is saved with data so it's 
> not so clean. I also have an issues with this but I resolve it in 
> different way (not say better :) 
>
> * In master branch Serializer is closely tied to Renderer so if there is 
> different Renderer class than new Serializer is needed. In forms branch 
> it is done in __init__ serialize method and this must be rewrite for 
> backward compatibility if django-serializers goes to core. I want to 
> propose my solution [1]: 
> For each format there is Serializer class which is made from 
> NativeSerializer ( from models to python native datatype) and 
> FormatSerializer 

Re: Moving forward with Serialization.

[Piotr Grabowski]

W dniu 31.08.2012 10:25, Tom Christie pisze:
> I personally think that Forms are already the place that should 
handle (de)serialisation. They already serialise to HTML: why should 
they not be able to serialise to other stream types?


Conceptually I agree.  As it happens django-serializers is perfectly 
capable of rendering into HTML forms, I just haven't yet gotten around 
to writing a form renderer, since it was out-of-scope of the fixture 
serialization functionality.


Pragmatically, I'm not convinced it'd work very well.  The existing 
Forms implementation is tightly coupled to form-data input and HTML 
output, and I think trying to address that without breaking 
backwards compatibility would be rather difficult.  It's maybe easy 
enough to do for flat representations, and pk relationships, but 
extending it to deal with nested representations, being able to use a 
Form as a field on another Form, and representing custom relationships 
would all take some serious hacking.  My personal opinion is that 
whatever benefits you'd gain in DRYness, you'd lose in code 
complexity.  Having said that, if someone was able to hack together a 
Forms-based fixture serialization/deserialization implementation that 
passes the Django test suite, and didn't look too kludgy, I'd be 
perfectly willing to revise my opinion.


I am not quite sure but I think Forms should be build based on some 
serialization API not opposite. Forms are more precise way of models 
serialization - they are models serialized to html (specific format) 
with some validation (specific actions) when deserializing.



I like Tom's django-serialziers but there are some things that I want to 
mention:


* Process of serialization is split to two parts - transformation to 
python native datatype (serializer) and next to specific text format 
(renderer). But when serializing also Field is saved with data so it's 
not so clean. I also have an issues with this but I resolve it in 
different way (not say better :)


* In master branch Serializer is closely tied to Renderer so if there is 
different Renderer class than new Serializer is needed. In forms branch 
it is done in __init__ serialize method and this must be rewrite for 
backward compatibility if django-serializers goes to core. I want to 
propose my solution [1]:
For each format there is Serializer class which is made from 
NativeSerializer ( from models to python native datatype) and 
FormatSerializer (Renderer)


class Serializer(object):
# class for native python serialization/deserialization
SerializerClass = NativeSerializer
# class for specific format serialization/deserialization
RendererClass = FormatSerializer

def serialize(self, queryset, **options):

def deserialize(self, stream_or_string, **options):

Deserializer = Serializer

This is fully backward compatible and user can do:
serializers.serialize('registered_format', objects, 
serializer=MyNativeSerializer)


This will make new Serializer class with SerializerClass == 
MyNativeSerializer. In this solution NativeSerializer and 
FormatSerializer are more independent. In my solution each 
NativeSerializer can be render by each FormatSerializer but it's not so 
simple. FormatSerializer provide NativeSerializer with some context so 
you can tell that NativeSerializer knows what format will be serialized. 
It's not exactly format but some metadata about it. I am not proud of 
this :/


* IMO there is bug related to xml. All model fields must be transform to 
text before xml serialization. In current django serialization framework 
field's method value_to_string is responsible for this. In 
django-serializers this method is not always called so it can lead to 
error with custom model field


[1] 
https://github.com/grapo/django/tree/soc2012-serialization/django/core/serializers


--
Piotr Grabowski

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Moving forward with Serialization.

[Tom Christie]

> Do you know Colander?

I do now.  Thanks, that's very similar, and looks well done.

> I personally think that Forms are already the place that should handle 
(de)serialisation. They already serialise to HTML: why should they not be 
able to serialise to other stream types?

Conceptually I agree.  As it happens django-serializers is perfectly 
capable of rendering into HTML forms, I just haven't yet gotten around to 
writing a form renderer, since it was out-of-scope of the fixture 
serialization functionality.

Pragmatically, I'm not convinced it'd work very well.  The existing Forms 
implementation is tightly coupled to form-data input and HTML output, and I 
think trying to address that without breaking backwards compatibility would 
be rather difficult.  It's maybe easy enough to do for flat 
representations, and pk relationships, but extending it to deal with 
nested representations, being able to use a Form as a field on another 
Form, and representing custom relationships would all take some serious 
hacking.  My personal opinion is that whatever benefits you'd gain in 
DRYness, you'd lose in code complexity.  Having said that, if someone was 
able to hack together a Forms-based fixture serialization/deserialization 
implementation that passes the Django test suite, and didn't look too 
kludgy, I'd be perfectly willing to revise my opinion. 

There's also some subtle differences between serializer fields, and 
Django's existing form fields.  Because form fields only handle form input, 
incoming fields can never be null, only blank or not blank.   With other 
representations such as JSON, that's not the case, so for serializer 
fields, the blank=True/False null=True/False style is appropriate, whereas 
for form fields the required=True/False style is appropriate.

I'm also wary of getting bogged down in high level 'wouldn't it be nice 
if...' conversations.  With just a little bit of work, the 
django-serializers implementation could be turned into a pull request 
that'd replace the existing fixture serialization with something much more 
useful and flexible.  What I'm really looking for is some feedback on if 
it'd be worth my time.

Regards,

  Tom


On Wednesday, 29 August 2012 06:27:35 UTC+1, schinckel wrote:
>
> Hi Tom,
>
> I've knocked around ideas of my own (internally, and on #django) related 
> to serialisation: it's something I've had lots to think about, since until 
> recently, most of my django work was in JSON apis.
>
> I personally think that Forms are already the place that should handle 
> (de)serialisation. They already serialise to HTML: why should they not be 
> able to serialise to other stream types?
>
> This is the approach I've started to use for my API generation code. They 
> already have a declarative nature, and then you get all of the form 
> validation on incoming data: a big improvement over how I've done it in the 
> past.
>
> (I've done some work on a form-based API generation framework: 
> https://bitbucket.org/schinckel/django-repose. Whilst this is in use, 
> it's still not really feature complete).
>
> Matt.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/Lc0fSSAdz4IJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Moving forward with Serialization.

[Tom Christie]

Hi all,

  During Piotr's GSOC addressing customizable serialization I've been 
continued to work on my own implementation.
I've now got what feels to me to be a natural conclusion of the design, 
which is the 'forms' branch of the 
'django-serializers'project.

There's still a few small bits and pieces that need filling in, but it's 
pretty much there,
and I believe the project essentially meets the requirements of a more 
customizable serialization API.

I won't go into the full documentation here (refer to the project itself if 
you're interested), but in summary it gives you:

* A declarative serialization API with Serializer/ModelSerializer classes 
that mirror the existing Form/ModelForm API.
* A backwards compatible FixtureSerializer class.
* Support for nested relationships, pk relationships, natural key 
relationships, and custom relationships such as hyperlinking.
* Validation for deserialization of objects, similar to Form validation.
* Explicitly decouples the structural concerns of serialization from the 
encoding/decoding concerns.
* Passing Django's existing fixture serialization tests.
* Fairly trivial to port into Django while keeping backwards compatibility 
with existing serialization API.

What I particularly like about the approach is how it mirrors the existing 
Forms API, and I think it'd go a long way to making Django a more useable 
framework for authors of Web APIs.  As an example, here's what it looks 
like when using django-serializers to write API 
views
.

At this point what I'm looking for is some feedback:

* Would it be worth me trying to push towards getting this into core.
* Are there any fundamental issues with the API, or use cases it clearly 
doesn't address.
* Any suggested next steps, or things that would need to be 
addressed/clarified before it could considered it for core.

It's probably also worth noting that I'm now planning on using the project 
inside my personal project, django-rest-framework, in case that's relevant 
to the discussion.

Thoughts?

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/3ypJSF7nN8oJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.