Re: websockets

[ptone]

On Wednesday, April 17, 2013 2:31:48 AM UTC-7, Aymeric Augustin wrote:
>
> 2013/4/17 Daniel Swarbrick 
>
>> On the pure Django side of things, one of the challenges I encountered 
>> was "IDLE IN TRANSACTION" hanging DB connections in the long-running 
>> WebSocket views. I really ought to research a more elegant solution to 
>> this, but for now I'm just doing all my DB queries early in the view, then 
>> fairly brutally closing the DB connection before entering the long-running 
>> WS poll loop.
>>
>
> The new transaction management introduced in Django 1.6 may help with this 
> problem, but it isn't a good idea not to maintain one database connection 
> per websocket on a website with more than a few users anyway :)
>
> I believe you need a connection pooler and asynchronous database I/O if 
> you want to talk to the database from a websocket handler. I haven't looked 
> at this in detail.
>

A difficulty will be moving beyond simple demos to something that can be 
more widely and immediately useful and allow for moderate scale.

The first real challenges happen in scaling beyond one app server process, 
as you now start dealing with arbitrating state change collisions. These 
are less of an issue in more atomic, transaction based request response 
designs - where you expect state may change between page refreshes.

I've got a bunch more research to do, but I've been making some notes in 
this area - some fun geek concepts that are potentially involved: paxos, 
operational transformation, software transactional memory, vector clocks 
etc.

Relying only on the ORM and stock backends will be challenging to support 
distributed concurrency.

Here is a micro manifesto on my thoughts of Django and realtime:

Many/most sites don't need all pages/views to be realtime backed, so for 
many projects traditional Django usage will continue to work just fine, 
this is Django's heritage and strength. Making use of Django components 
(ORM, templates?) in a realtime context should be facilitated.

Django should continue to focus on making the common cases relatively easy, 
while not getting in the way of the more elaborate or complex needs.

Django shouldn't ship any stock solution initially in core, and if/when it 
does ship something, it should be built on clean and fundamental 
 abstracted layers that allow others to continue to innovate without 
needing to reinvent the truly common low level parts.

In my own digging, I haven't found anything so far at the low level that 
would need to be added to Django itself. The possible exception is the 
predicate stuff, and I do still hope to get that work in after Anssi's 
lookup refactor work is completed. 
 
There is the issue of how much model logic gets moved to Javascript, and 
how integrated vs loosely coupled the JS should be from the Python code. On 
one end would be just standardizing on a wire protocol like Meteor's DDP or 
something similar. At the other end would be building on top of that and 
having something that let you declare a ModelForm like class in Python, 
which when used via a special template tag, would magically become a client 
side representation in the browser and automatically set up all the 
realtime data exchanges to the Django model. Those obviously don't have to 
be mutually exclusive ideas - in fact the latter should be based on 
something like the former.

So we need to continue the period of exploration - I do think Tulip looks 
promising enough to start planning a future on - but there will be many 
design components that won't need to exist at the tulip calling level, or 
would be relatively easy to port.

To paraphrase something Jacob said after the Meteor keynote at DjangoCon `I 
know that in  years, I'll be using something that works like Meteor - I 
hope that something can be Django'.

-Preston

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: A second stab at an implementation of composite fields

[Michal Petrucha]

On Wed, Apr 17, 2013 at 03:49:11AM -0700, Anssi Kääriäinen wrote:
> The basic idea is that there is a new ForeignObject class. A
> ForeignObject basically just takes a related model, and from_fields
> and to_fields which are the names of the fields used for the relation.
> Then, the ORM knows how to create JOINs, subqueries etc based on a
> ForeignObject. The ForeignObject itself is a virtual field, it doesn't
> have any concrete fields in the DB.
> 
> There is currently zero public API support for using the
> ForeignObject. The addition of ForeignObject is there to make the work
> on composite fields and multicolumn foreign keys in particular easier.

Thanks for the overview. For the sake of completeness, we had a
discussion on IRC earlier today, the outcome of which is that the
ForeignObject refactor will simplify the task immensely. We also came
upon a few concerns which I'll describe in a moment.

> I took a look at the code and I think the .Meta fields implementation
> is good. (Minor point is that there is no need for the field.virtual
> flag - it should be enough to check if it has a db column or not).

Agreed. Actually, the way I tried to go about the implementation, was
to replace field.column with field.columns which would be a tuple and
in case of virtual fields this would be a property which automatically
gathers the columns of all of its enclosed concrete fields, which
means a field.virtual would be necessary, but it looks like this
brings an inappropriate amount of complexity with little benefit.

> Do you have any plans for updatable primary keys? Updatable primary
> keys would be useful: the current api where obj.lastname = newval;
> obj.save() will result in save of new object instead of update of the
> lastname is plain ugly. OTOH This problem might be too complex to
> include in GSoC - cascading the updates is going to be hard. One
> possible solution is to add some way to create the foreign keys as ON
> UPDATE CASCADE and let the DB solve the problem.

Now I get to the issues that I still haven't figured out or would like
the opinion of others on. I'll separate them into sections for better
readability.

Updatable primary keys
--

The one you mention here is the way the ORM finds out whether to
perform an INSERT or UPDATE on model.save(). The current way this
works is just fine for models with an AutoField which is not visible
to the user. The problem is that as soon as you use an explicit
primary key and aren't extremely careful, you'll get unexpected
results.

This isn't directly related to composite primary keys, however, those
make it more apparent. I actually raised this issue two years ago
(IIRC) and I think I discussed it with Carl and possibly others; the
outcome was that we'd just put a note into the docs that would warn
people to be careful not to make any part of primary key user-editable
in the admin and generally be cautious when handling them directly or
otherwise bad things would happen.

I still think it would be good to change this behavior. The downsides
are that
 1) it would subtly change behavior for models with explicit primary
keys in a backwards-incompatible way
 2) it would require models to hold something like "the last known
value of the primary key in DB"
 3) it would cause weird behavior in situations like if you have two
separate instances corresponding to the same DB row, modify the PK
in one and save. I don't really know how much of an issue this
would be.
 4) as Anssi said, we would need a cascading mechanism for updates.
With certain backends, this could probably be done by the database
itself, with others, I believe we'd have to do it manually
(SQLite?)...

Again, this is not entirely related to this project, but it would
still be nice to get this done before composite fields are released,
should we agree we want this implemented.

GenericForeignKey
-

Two years ago we gave this some thought and decided to leave this for
a later stage. I think the later stage is here.

It's fairly easy to represent composite values as strings, something
like the following works quite well::

",".join(escape(value) for value in composite_value)

Of course, we can just do this and stick it into the database as the
object_id. The problem is with GenericRelation. This is something that
works on the database level and we can't just ask the database server
to compare a string built this way with a tuple of other values.

Ideally, we'd need to come up with an encoding that would be possible
to reproduce as a SQL expression, preferably with the same result on
all DB backends. (The expression itself can be backend-dependent, the
result should probably not.)

As far as I know, Simone Federici had something that was close to what
we need, but IIRC, it had some issues. Simone, could you please chime
in and provide some more detail on this? Have you succeeded? Was it a
dead end?

I'll admit right away that I'm 

Re: Proposal: Redefine specific {% block %} in an intermediate template

[Emil Stenström]

Carl Meyer skrev 2013-04-18 00:09:

Hi Emil,

On 04/17/2013 04:00 PM, Emil Stenström wrote:

Carl Meyer skrev 2013-04-17 18:37:

Why not instead add a new block to base.html? So you'd change base.html
to have:

{% block outer-content %}
{% block content %}{% endblock content %}
{% endblock outer-content %}

And base_with_warning.html:

{% extends "base.html" %}

{% block outer-content %}
...
{% block content %}{% endblock content %}
{% endblock outer-content %}


Thanks for your reply! This is the same suggestion Jacob suggested one
minute before you. See my reply to him for my explanations of the
problems with this solution.


I think you may need to re-read my suggestion more carefully.

This is not the same as your Alternative 1, and your objection does not
apply. Templates inheriting from either base.html or
base_with_warning.html can both override the "content" block, and don't
need to know or care about the "outer-content" block.


Ah, I see what you mean now. Sorry for being slow, it's late here. It IS 
different from Jacobs suggestion too.


It looks a little bit hacky to double-define the block in base.html. 
There's the chance that some child templates to base.html start 
inheriting from content, and some from outer-content, since both work.


Also this doesn't work if we want to do the same with base_with_warning, 
as we then would have to add another wrapper block around content there.


Despite this problems, I agree that that's a nicer solution than all the 
other alternatives.


/Emil

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: Redefine specific {% block %} in an intermediate template

[Carl Meyer]

Hi Emil,

On 04/17/2013 04:00 PM, Emil Stenström wrote:
> Carl Meyer skrev 2013-04-17 18:37:
>> Why not instead add a new block to base.html? So you'd change base.html
>> to have:
>>
>> {% block outer-content %}
>> {% block content %}{% endblock content %}
>> {% endblock outer-content %}
>>
>> And base_with_warning.html:
>>
>> {% extends "base.html" %}
>>
>> {% block outer-content %}
>> ...
>> {% block content %}{% endblock content %}
>> {% endblock outer-content %}
> 
> Thanks for your reply! This is the same suggestion Jacob suggested one
> minute before you. See my reply to him for my explanations of the
> problems with this solution.

I think you may need to re-read my suggestion more carefully.

This is not the same as your Alternative 1, and your objection does not
apply. Templates inheriting from either base.html or
base_with_warning.html can both override the "content" block, and don't
need to know or care about the "outer-content" block.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: Redefine specific {% block %} in an intermediate template

[Emil Stenström]

Andrew Ingram skrev 2013-04-17 18:08:

I've been wanting this exact feature for years. I've always struggled to
explain the problem, but I've had numerous cases where this would have
made for a vastly simpler template structure.

Big +1 from me.


Since there are a couple of -1:s on this. Would you mind sharing an 
example? Preferably one where the alternatives wouldn't work.


/Emil

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: Redefine specific {% block %} in an intermediate template

[Emil Stenström]

Carl Meyer skrev 2013-04-17 18:37:

Why not instead add a new block to base.html? So you'd change base.html
to have:

{% block outer-content %}
{% block content %}{% endblock content %}
{% endblock outer-content %}

And base_with_warning.html:

{% extends "base.html" %}

{% block outer-content %}
...
{% block content %}{% endblock content %}
{% endblock outer-content %}


Thanks for your reply! This is the same suggestion Jacob suggested one 
minute before you. See my reply to him for my explanations of the 
problems with this solution.


/Emil

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: Redefine specific {% block %} in an intermediate template

[Emil Stenström]

Jacob Kaplan-Moss skrev 2013-04-17 18:36:

On Wed, Apr 17, 2013 at 10:50 AM, Emil Stenström  wrote:

{% extends "base.html" %}
{% block content %}
 Be careful when changing these settings!
 {% block content %}{% endblock %}
{% endblock %}


I find this intensely confusing -- I can't build a mental model of
what's going to be rendered there. Allowing re-definining of existing
blocks nested inside those blocks... ouch. I'm -1 on allowing this; it
just seems incredibly confusing and error prone.


The mental model I have, and that I've seen many beginners have when 
comming from other languages, is that the outmost tag is what you're 
"filling in", and the innermost block is what you're making available 
for child templates.


ASP.Net:
Parent: 
Child:  ... 



JSF:
Parent: 
Child:  ... 

Rails:
Parent: <%= yield :head %>
Child: <% content_for :head do %> ... <% end %>

I'm not saying I think you should differetiate the two, just explaining 
my mental model. The big difference is of course that I'm separating 
what a template consumes and what it makes a block available to child 
templates.


I understand the differences to your mental model, that nested blocks 
are also available to child templates.



The general way I've solved this in the past is pretty straightforward::

 {% block content-wrapper %}
 Be careful when changing these settings!
 {% block content %}{% endblock %}
 {% endblock %}

What's wrong with this?


This is "Alternative 1" in my original message, sorry if this was 
unclear. Say I change base.html to have a content-wrapper block, and put 
your code in base_with_warning. Half my templates inherit from base and 
half from base_with_warning. Now when I decide I want to add a warning 
to a template that previously didn't have one, I change parent template 
AND need to remember to rename the block to content instead of 
content-wrapper. The "rename block"-part is the difference between our 
two variants.


In projects with lots of templates, this quickly becomes a maintainance 
nightmare. People forget the change the block name, and accidentially 
don't get the warning they expect, even though they are explicitly 
inheriting from base_with_warning. It's a mess.


This is the problem I'm trying to solve with my proposal. I agree it's 
slightly backwards, but I couldn't come up with a smarter way.


/Emil

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

RE: Proposal: Redefine specific {% block %} in an intermediate

[Babatunde Akinyanmi]

 template
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

+1 for being confusing

Sent from my Windows Phone
From: Carl Meyer
Sent: 4/17/2013 5:37 PM
To: django-developers@googlegroups.com
Subject: Re: Proposal: Redefine specific {% block %} in an intermediate
template
Hi Emil,

On 04/17/2013 09:50 AM, Emil Stenstr=C3=B6m wrote:
> Proposal:
> Make it possible to use the same template block name twice, if the
> second one is nested within the first one. Inheriting from the template
> fills in the innermost block.

This is an interesting proposal, but I am concerned that the syntax you
propose is confusing on several levels:

1) If I haven't seen this done before, and I'm reading your template and
I see the two nested blocks, it's not at all intuitively obvious that
child templates would override the inner one rather than the outer one.
In fact, my naive assumption would have been the opposite.

2) If the "warning" HTML in your example is lengthy, I may see the outer
block and not immediately notice the inner one, and be confused as to
why the child templates aren't overriding the "content" block I can see
right in front of me.

I also think you missed the best (existing) solution to your example
problem:

> --
> Background (why is this useful?):
> Say you have one base.html template defining a {% block content %}, and
> ten templates inheriting from it and defining the contents of the block.
> Now you decide that five of those templates actually should have a
> warning message at the top, and you decide to make a new
> base_with_warning.html template. You should should now be able to just
> change what block five of the templates inherit from, add the warning to
> your new base template, and be done with it?
>=20
> Not really. Your base_with_warning.html would have to look like this:
>=20
> {% extends "base.html" %}
> {% block content %}
> Be careful when changing these settings!
> {% block content %}{% endblock %}
> {% endblock %}
>=20
> And this doesn't work in Django because duplicate block names are not
> allowed. I think making this possible would make cases like the one I'm
> describing above much easier to handle.
>=20
> --
> Alternatives (this is how you should solve it instead...):
> There are other ways to solve the problem I'm describing above. Let me
> list a few and talk about the problems I see with them:
>=20
> 1) Create a new template block in base_with_warning.html and change the
> five templates to use that block instead.
>=20
> Problem: This puts the burden on developers to remember all kinds of
> content blocks they have in this project, and change them depending on
> what template they happen to be inherit from. The interface to the
> developer working on a new child template is much cleaner if they know
> they can always use the content block, and that the template they
> inherit from with handle the rest.

Why not instead add a new block to base.html? So you'd change base.html
to have:

{% block outer-content %}
{% block content %}{% endblock content %}
{% endblock outer-content %}

And base_with_warning.html:

{% extends "base.html" %}

{% block outer-content %}
...
{% block content %}{% endblock content %}
{% endblock outer-content %}

I think this achieves your goal that none of the inheriting templates
have to change anything besides which template they extend. And it does
it with very little added verbosity in the base templates, a clearer
conceptual layering of blocks, and less potential for confusion when
reading the templates.

> 2) Make a child templates call super, and put the warning div inside the
> content block in base_with_warning.html.
>=20
> Problem: It's very easy to forget calling super when making a new
> template. It quickly becomes repetitive to type block.super in each of
> the templates. Should I call super in the five templates without a
> warning too? Just to be sure it isn't forgotten if someone changes the
> inherited template.
>=20
> 3) Use an include tag in each of the child templates instead of inheritin=
g.
>=20
> Problem: This is just as repetitive as copy-pasting the div to each of
> the subtemplate, it's just the {% include "warning.html" %} is a shorter
> string to copy-paste. As with all copy-paste, it makes it easy miss one
> instance when changing the others. Not maintainable.
>=20
> 4) Use a template variable to decide whether to render the warning or not=
.
>=20
> Problem: This moves the copy-paste to the view logic. Now I need to
> remember to pass the right variable in the view instead, with the same
> problem as with copy-paste above.
>=20
> 5) Write a custom template tag that does what I want.
>=20
> Problem: I really don't know how to do this, and I haven't found someone
> that has written a tag like that after are plenty of googling.

I agree that none of these solutions are good.

Carl

--=20
You received this message because you are 

Re: Proposal: Redefine specific {% block %} in an intermediate template

[Carl Meyer]

Hi Emil,

On 04/17/2013 09:50 AM, Emil Stenström wrote:
> Proposal:
> Make it possible to use the same template block name twice, if the
> second one is nested within the first one. Inheriting from the template
> fills in the innermost block.

This is an interesting proposal, but I am concerned that the syntax you
propose is confusing on several levels:

1) If I haven't seen this done before, and I'm reading your template and
I see the two nested blocks, it's not at all intuitively obvious that
child templates would override the inner one rather than the outer one.
In fact, my naive assumption would have been the opposite.

2) If the "warning" HTML in your example is lengthy, I may see the outer
block and not immediately notice the inner one, and be confused as to
why the child templates aren't overriding the "content" block I can see
right in front of me.

I also think you missed the best (existing) solution to your example
problem:

> --
> Background (why is this useful?):
> Say you have one base.html template defining a {% block content %}, and
> ten templates inheriting from it and defining the contents of the block.
> Now you decide that five of those templates actually should have a
> warning message at the top, and you decide to make a new
> base_with_warning.html template. You should should now be able to just
> change what block five of the templates inherit from, add the warning to
> your new base template, and be done with it?
> 
> Not really. Your base_with_warning.html would have to look like this:
> 
> {% extends "base.html" %}
> {% block content %}
> Be careful when changing these settings!
> {% block content %}{% endblock %}
> {% endblock %}
> 
> And this doesn't work in Django because duplicate block names are not
> allowed. I think making this possible would make cases like the one I'm
> describing above much easier to handle.
> 
> --
> Alternatives (this is how you should solve it instead...):
> There are other ways to solve the problem I'm describing above. Let me
> list a few and talk about the problems I see with them:
> 
> 1) Create a new template block in base_with_warning.html and change the
> five templates to use that block instead.
> 
> Problem: This puts the burden on developers to remember all kinds of
> content blocks they have in this project, and change them depending on
> what template they happen to be inherit from. The interface to the
> developer working on a new child template is much cleaner if they know
> they can always use the content block, and that the template they
> inherit from with handle the rest.

Why not instead add a new block to base.html? So you'd change base.html
to have:

{% block outer-content %}
{% block content %}{% endblock content %}
{% endblock outer-content %}

And base_with_warning.html:

{% extends "base.html" %}

{% block outer-content %}
...
{% block content %}{% endblock content %}
{% endblock outer-content %}

I think this achieves your goal that none of the inheriting templates
have to change anything besides which template they extend. And it does
it with very little added verbosity in the base templates, a clearer
conceptual layering of blocks, and less potential for confusion when
reading the templates.

> 2) Make a child templates call super, and put the warning div inside the
> content block in base_with_warning.html.
> 
> Problem: It's very easy to forget calling super when making a new
> template. It quickly becomes repetitive to type block.super in each of
> the templates. Should I call super in the five templates without a
> warning too? Just to be sure it isn't forgotten if someone changes the
> inherited template.
> 
> 3) Use an include tag in each of the child templates instead of inheriting.
> 
> Problem: This is just as repetitive as copy-pasting the div to each of
> the subtemplate, it's just the {% include "warning.html" %} is a shorter
> string to copy-paste. As with all copy-paste, it makes it easy miss one
> instance when changing the others. Not maintainable.
> 
> 4) Use a template variable to decide whether to render the warning or not.
> 
> Problem: This moves the copy-paste to the view logic. Now I need to
> remember to pass the right variable in the view instead, with the same
> problem as with copy-paste above.
> 
> 5) Write a custom template tag that does what I want.
> 
> Problem: I really don't know how to do this, and I haven't found someone
> that has written a tag like that after are plenty of googling.

I agree that none of these solutions are good.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, 

Re: Proposal: Redefine specific {% block %} in an intermediate template

[Jacob Kaplan-Moss]

On Wed, Apr 17, 2013 at 10:50 AM, Emil Stenström  wrote:
> {% extends "base.html" %}
> {% block content %}
> Be careful when changing these settings!
> {% block content %}{% endblock %}
> {% endblock %}

I find this intensely confusing -- I can't build a mental model of
what's going to be rendered there. Allowing re-definining of existing
blocks nested inside those blocks... ouch. I'm -1 on allowing this; it
just seems incredibly confusing and error prone.

The general way I've solved this in the past is pretty straightforward::

{% block content-wrapper %}
Be careful when changing these settings!
{% block content %}{% endblock %}
{% endblock %}

What's wrong with this?

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: Redefine specific {% block %} in an intermediate template

[Andrew Ingram]

I've been wanting this exact feature for years. I've always struggled to
explain the problem, but I've had numerous cases where this would have made
for a vastly simpler template structure.

Big +1 from me.


On 17 April 2013 16:50, Emil Stenström  wrote:

> Hi,
>
> Proposal:
> Make it possible to use the same template block name twice, if the second
> one is nested within the first one. Inheriting from the template fills in
> the innermost block.
>
> --
> Background (why is this useful?):
> Say you have one base.html template defining a {% block content %}, and
> ten templates inheriting from it and defining the contents of the block.
> Now you decide that five of those templates actually should have a warning
> message at the top, and you decide to make a new base_with_warning.html
> template. You should should now be able to just change what block five of
> the templates inherit from, add the warning to your new base template, and
> be done with it?
>
> Not really. Your base_with_warning.html would have to look like this:
>
> {% extends "base.html" %}
> {% block content %}
> Be careful when changing these settings!
> {% block content %}{% endblock %}
> {% endblock %}
>
> And this doesn't work in Django because duplicate block names are not
> allowed. I think making this possible would make cases like the one I'm
> describing above much easier to handle.
>
> --
> Alternatives (this is how you should solve it instead...):
> There are other ways to solve the problem I'm describing above. Let me
> list a few and talk about the problems I see with them:
>
> 1) Create a new template block in base_with_warning.html and change the
> five templates to use that block instead.
>
> Problem: This puts the burden on developers to remember all kinds of
> content blocks they have in this project, and change them depending on what
> template they happen to be inherit from. The interface to the developer
> working on a new child template is much cleaner if they know they can
> always use the content block, and that the template they inherit from with
> handle the rest.
>
> 2) Make a child templates call super, and put the warning div inside the
> content block in base_with_warning.html.
>
> Problem: It's very easy to forget calling super when making a new
> template. It quickly becomes repetitive to type block.super in each of the
> templates. Should I call super in the five templates without a warning too?
> Just to be sure it isn't forgotten if someone changes the inherited
> template.
>
> 3) Use an include tag in each of the child templates instead of inheriting.
>
> Problem: This is just as repetitive as copy-pasting the div to each of the
> subtemplate, it's just the {% include "warning.html" %} is a shorter string
> to copy-paste. As with all copy-paste, it makes it easy miss one instance
> when changing the others. Not maintainable.
>
> 4) Use a template variable to decide whether to render the warning or not.
>
> Problem: This moves the copy-paste to the view logic. Now I need to
> remember to pass the right variable in the view instead, with the same
> problem as with copy-paste above.
>
> 5) Write a custom template tag that does what I want.
>
> Problem: I really don't know how to do this, and I haven't found someone
> that has written a tag like that after are plenty of googling.
>
> --
> Backwards compatibility (this will not break stuff!):
> Since duplicate block names are not allowed today, this change would be
> 100% backwards compatible. Everything would continue working as it always
> has.
>
> --
> FAQ:
> a) How would you inherit from a template with duplicate blocks with the
> same name? This proposal only deals with the case where the first block is
> defined inside the second block. If this happens the child templates
> content should fill the INNER block, not the outer one. There would be no
> way to override the outer block, since that would be shadowed by the
> redefined inner block.
>
> b) If we allow for multiple blocks with the same name, what should happen
> with two blocks side by side (not nested)? We wouldn't allow that. This
> proposal is only about duplicate blocks when nested inside eachother. All
> other duplication would still yield the same error.
>
> c) I have another suggestion that solves your problem without the problems
> with 1) - 5) above, that doesn't require a change in django? That really
> isn't a question, but fine :) Bring it on! I'm really interested in solving
> the problem described in the background section in a clean way, not a
> particular tech solution.
>
> d) Do other people have this problem? I've found a few:
> http://stackoverflow.com/questions/15448211/django-template-block-overriding/-
> http://stackoverflow.com/questions/8717521/django-multi-level-template-extends-and-nested-blocks-
>  This could also be used to handle nicely breadcrumbs in multiple levels.
> 

Proposal: Redefine specific {% block %} in an intermediate template

[Emil Stenström]

Hi,

Proposal: 
Make it possible to use the same template block name twice, if the second 
one is nested within the first one. Inheriting from the template fills in 
the innermost block.

--
Background (why is this useful?):
Say you have one base.html template defining a {% block content %}, and ten 
templates inheriting from it and defining the contents of the block. Now 
you decide that five of those templates actually should have a warning 
message at the top, and you decide to make a new base_with_warning.html 
template. You should should now be able to just change what block five of 
the templates inherit from, add the warning to your new base template, and 
be done with it?

Not really. Your base_with_warning.html would have to look like this:

{% extends "base.html" %}
{% block content %}
Be careful when changing these settings!
{% block content %}{% endblock %}
{% endblock %}

And this doesn't work in Django because duplicate block names are not 
allowed. I think making this possible would make cases like the one I'm 
describing above much easier to handle.

--
Alternatives (this is how you should solve it instead...):
There are other ways to solve the problem I'm describing above. Let me list 
a few and talk about the problems I see with them:

1) Create a new template block in base_with_warning.html and change the 
five templates to use that block instead. 

Problem: This puts the burden on developers to remember all kinds of 
content blocks they have in this project, and change them depending on what 
template they happen to be inherit from. The interface to the developer 
working on a new child template is much cleaner if they know they can 
always use the content block, and that the template they inherit from with 
handle the rest. 

2) Make a child templates call super, and put the warning div inside the 
content block in base_with_warning.html.

Problem: It's very easy to forget calling super when making a new template. 
It quickly becomes repetitive to type block.super in each of the templates. 
Should I call super in the five templates without a warning too? Just to be 
sure it isn't forgotten if someone changes the inherited template.

3) Use an include tag in each of the child templates instead of inheriting.

Problem: This is just as repetitive as copy-pasting the div to each of the 
subtemplate, it's just the {% include "warning.html" %} is a shorter string 
to copy-paste. As with all copy-paste, it makes it easy miss one instance 
when changing the others. Not maintainable.

4) Use a template variable to decide whether to render the warning or not.

Problem: This moves the copy-paste to the view logic. Now I need to 
remember to pass the right variable in the view instead, with the same 
problem as with copy-paste above.

5) Write a custom template tag that does what I want.

Problem: I really don't know how to do this, and I haven't found someone 
that has written a tag like that after are plenty of googling.

--
Backwards compatibility (this will not break stuff!):
Since duplicate block names are not allowed today, this change would be 
100% backwards compatible. Everything would continue working as it always 
has.

--
FAQ:
a) How would you inherit from a template with duplicate blocks with the 
same name? This proposal only deals with the case where the first block is 
defined inside the second block. If this happens the child templates 
content should fill the INNER block, not the outer one. There would be no 
way to override the outer block, since that would be shadowed by the 
redefined inner block.

b) If we allow for multiple blocks with the same name, what should happen 
with two blocks side by side (not nested)? We wouldn't allow that. This 
proposal is only about duplicate blocks when nested inside eachother. All 
other duplication would still yield the same error.

c) I have another suggestion that solves your problem without the problems 
with 1) - 5) above, that doesn't require a change in django? That really 
isn't a question, but fine :) Bring it on! I'm really interested in solving 
the problem described in the background section in a clean way, not a 
particular tech solution.

d) Do other people have this problem? I've found a few: 
http://stackoverflow.com/questions/15448211/django-template-block-overriding/ 
- 
http://stackoverflow.com/questions/8717521/django-multi-level-template-extends-and-nested-blocks
 
- This could also be used to handle nicely breadcrumbs in multiple levels. 
You can probably think of more use-cases.

e) Is this the same as Ticket #4529: 
https://code.djangoproject.com/ticket/4529 ? No. The code in that ticket 
would still not work.

--
Next steps:
I'm very eager to hear your thoughts on this, and maybe, if you think this 
is a good idea, start to write up a ticket.

--
Thanks for reading this far! I love the work 

Re: [GSoC 2013] Revamping validation functionality proposal

[Jacob Kaplan-Moss]

Hi Christopher -

Thanks for the proposal, this is quite good. I really appreciate the
detail; it's clear you've put a lot of thought into this. In general,
I think this is a strong proposal and one I'd support. However, I do
have some comments/feedback:

1. We've had some discussions about bringing django-secure into core
as part of a more general "checkdeploy" command. The idea being
something you can run shortly before deployment that'd check for all
the stuff that django-secure checks for -- but also more (outdated
dependencies, debug mode, exposed admin, etc). I think this dovetails
nicely with your proposal: it seems that all these "checks"
(validation, deployment, security) could use a single discovery and
running mechanism. I'd love to see you think about modifying your
proposal to include this sort of unification as well as the bringing
of django-secure into core.

2. I think your proposal is a bit too big. I'd general prefer to see a
less ambitious proposal with a high probability of success over a high
risk/high reward. I'd like to see you drop the "django-updates" part
of the proposal, and focus on validation and django-secure. If you end
up with extra time, you can always use it to write more checks.

3. You've done a good job breaking up the first half of the project
into one week chucks, which shows me you've thought carefully about
the tasks and steps involved. However, when you get to the second
half, you're a lot more vague. I'd like to see you think more
carefully about your time during that second half.

4. Pet peeve alert: "documentation" shouldn't be an afterthought. I
HATE seeing "week X: documentation" -- it implies that you're planning
on *not* writing documentation as you go, but instead saving it for
last. You've been around long enough to know that's not how we do
things; documentation happens alongside code. You'd make me much
happier if you updated your proposal to not imply that you'd leave
documentation for later.

Again, I want to stress that overall this is a really solid proposal;
don't take my criticism *too* hard. I think it could be an excellent
one, though, so I hope you'll take my suggestions into account. Good
luck!

Jacob

On Tue, Apr 16, 2013 at 12:09 PM, Christopher Medrela
 wrote:
> I would like to participate in Google Summer of Code this year. I've written
> proposal for "revamping validation functionality" idea. It's available as a
> gist: https://gist.github.com/chrismedrela/82cbda8d2a78a280a129 . Below, I'm
> pasting the proposal. Feel free to criticize it.
>
> Table of content
>
> 1. Abstract 1.1 Drawbacks of the existing framework 1.2 Goals 1.3 Benefits
> 2. The new framework 2.1 Overview 2.2 Advantages
> 3. New features 3.1 django-secure 3.2 django-update
> 4. Schedule and milestones
> 5. Who am I?
>
> 1. Abstract
>
> 1.1 Drawbacks of the existing framework
>
> Django currently has a validation framework, but there are a lot of problems
> with it. First of all, it is monolithic and developers cannot write custom
> validation (see [#16905]) or modify the existing functionality (see
> [#12674]); validation lives in a few functions like
> django.core.management.validation.get_validation_errors [1] or
> django.contrib.admin.validation.validate [2]. The validation functionality
> is not separated from other stuff like printing found errors during
> validating models in get_validation_errors or registering models in admin
> app [3] (see [#8579]); it is sometimes done during first call to an
> important method, i. e. CurrentSiteManager [4] is validated in its
> get_queryset [5] method.
>
> [#12674] https://code.djangoproject.com/ticket/12674
> [#16905] https://code.djangoproject.com/ticket/16905
> [1]
> https://github.com/django/django/blob/master/django/core/management/validation.py#L22
> [2]
> https://github.com/django/django/blob/master/django/contrib/admin/validation.py#L14
> [3]
> https://github.com/django/django/blob/master/django/contrib/admin/sites.py#L52
> [#8579] https://code.djangoproject.com/ticket/8579
> [4]
> https://github.com/django/django/blob/master/django/contrib/sites/managers.py#L5
> [5]
> https://github.com/django/django/blob/master/django/contrib/sites/managers.py#L38
>
> There are few tests of the validation framework and it is not easily
> testable because validation functions return concatenated error messages
> instead of list of errors (see
> django.tests.invalid_models.invalid_models.models [6]). It also lacks some
> features like warnings (see [#19126]). Due to this disadvantages lots of
> apps do not have any validation, i. e. they do not check inter-app
> dependencies.
>
> [6]
> https://github.com/django/django/blob/master/tests/invalid_models/invalid_models/models.py#L366
> [#19126] https://code.djangoproject.com/ticket/19126
>
> 1.2 Goals
>
> This proposal is about revamping current validation framework. First of all,
> we need to write more tests and rewrite existing ones. Then we need an
> consistent 

Re: A second stab at an implementation of composite fields

[Anssi Kääriäinen]

On 12 huhti, 18:34, Michal Petrucha  wrote:
> On Fri, Apr 12, 2013 at 07:35:45AM -0700, Anssi Kääriäinen wrote:
> > On 12 huhti, 16:44, Michal Petrucha  wrote:
> > ForeignKeys have been changed a lot since 2012-11-04. The introduction
> > of ForeignObject (which is base for ForeignKey) means that there is
> > support for multicolumn joins in the ORM now, and that ForeignObject
> > itself is a non-concrete field (not in ._meta.virtual_fields though).
> > ForeignObject currently works only on ORM level, so there is no API
> > for creating multicolumn foreign keys.
>
> > In any case, the GSoC proposal should take the introduction of
> > ForeignObject in account. tests/foreign_object contain some examples
> > related to ForeignObject functionality, exploring how those examples
> > work is a good idea.
>
> This has been on my TODO list for a few days actually, I was going
> through the work done regarding #19385 [1] and noticed this. However,
> I still need to give it a more thorough examination to fully get a
> grasp on this.

The basic idea is that there is a new ForeignObject class. A
ForeignObject basically just takes a related model, and from_fields
and to_fields which are the names of the fields used for the relation.
Then, the ORM knows how to create JOINs, subqueries etc based on a
ForeignObject. The ForeignObject itself is a virtual field, it doesn't
have any concrete fields in the DB.

There is currently zero public API support for using the
ForeignObject. The addition of ForeignObject is there to make the work
on composite fields and multicolumn foreign keys in particular easier.

> > Another big issue to solve is how to store the fields in model._meta.
> > IMO a good idea is to add all fields into some attribute ('all_fields'
> > as name might be good...), then separate the fields for different use
> > cases based on features the fields has. For example concrete_fields is
> > [f for f in self.all_fields if f.column], form fields are [f for f in
> > self.all_fields if f.form_field] (or maybe this should be called
> > logical fields - a multicolumn foreign key is logical, but not
> > concrete field, and the fields used by the foreign key are concrete
> > but not logical fields). The different field lists can be
> > cached_properties so there will be no problems from performance
> > perspective.
>
> As a matter of fact, this is one of the first things I did back in
> 2011. (-: Almost exactly like that -- _meta.fields contains all fields
> (except for M2M), _meta.concrete_fields, _meta.virtual_fields and I
> think I also needed a _meta.local_concrete (for non-inherited fields;
> I couldn't find a better name for it that wouldn't be too verbose).

I took a look at the code and I think the .Meta fields implementation
is good. (Minor point is that there is no need for the field.virtual
flag - it should be enough to check if it has a db column or not).

Do you have any plans for updatable primary keys? Updatable primary
keys would be useful: the current api where obj.lastname = newval;
obj.save() will result in save of new object instead of update of the
lastname is plain ugly. OTOH This problem might be too complex to
include in GSoC - cascading the updates is going to be hard. One
possible solution is to add some way to create the foreign keys as ON
UPDATE CASCADE and let the DB solve the problem.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: first() and last(), earliest() and latest()

[Anssi Kääriäinen]

On 28 helmi, 01:34, Wim Feijen  wrote:
> Hi all,
>
> We struggled to get a proper definition for a first() and last() method vs.
> earliest() and latest() . I'd like to make one proposal. After that, I
> really like your opinion on which syntax you prefer.
>
> First, let me give you a lenghty introduction. We discussed several use
> cases on this mailing 
> list.
> Then, I realized that:
>
> .filter(last_name__startswith='b').order_by('last_name').first()
> is an acceptable compromise for me to use in stead of:
> .first(last_name__startswith='b').order_by('last_name')
>
> Last weekend Aymeric explained to me that earliest can actually accomplish
> the same:
> .filter(last_name__startswith='b').earliest('last_name')
>
> Then, I find "earliest" an inappropriate name, because it does not describe
> functioning well.
>
> Therefore, my proposal is, if we are going to implement the earliest and
> latest method, we should definitely rename them to: first and latest.
>
> After that, there is only one question left:
>
> Which style do you prefer?
>
> .filter(last_name__startswith='b').order_by('last_name').first()    # clear
> and long
> .first(last_name__startswith='b').order_by('last_name')    # first method
> has filter syntax.
> .filter(last_name__startswith='b').first('last_name')   # first method has
> order_by syntax.
>
> So, what do you think?

While investigating the queryset iterator memory leak issue I spotted
some instances of this anti-pattern:
if qs:
obj = qs[0]
else:
obj = ...
This has the side effect of fetching the whole dataset into memory
(except on Oracle).

.get() doesn't work for this use-case as multiple objects returned
will cause an error. One solution is:
qs = qs[0:1]
if qs:
obj = qs[0]
else:
obj = ...
but this isn't exactly clean.

It would be nice if one could write:
obj = qs.first() or ...
this is clean and short. (Or equivalently: obj = qs.first(); if not
obj: obj = ...).

I propose the following solution: deprecate earliest()/latest() and
replace them by first()/last(). When using first()/last() no match
isn't an exception, instead None is returned. The method returns the
first/last object by:
  1) the given by_fields
  2) if no by_fields, then Meta.ordering (for earliest/latest this was
by Meta.get_latest_by)
  3) raise error (similar to earilest/latest() - another option would
be implicit ordering by pk, but explicit is better...).

The underlying problem is that there is no simple way to return the
first object of the queryset in a way that multiple or no objects
matched isn't an exception. So, users are either going to write code
using the above mentioned antipattern or forced to use exceptions for
logic.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: websockets

[Aymeric Augustin]

2013/4/17 Daniel Swarbrick 

> On the pure Django side of things, one of the challenges I encountered was
> "IDLE IN TRANSACTION" hanging DB connections in the long-running WebSocket
> views. I really ought to research a more elegant solution to this, but for
> now I'm just doing all my DB queries early in the view, then fairly
> brutally closing the DB connection before entering the long-running WS poll
> loop.
>

The new transaction management introduced in Django 1.6 may help with this
problem, but it isn't a good idea not to maintain one database connection
per websocket on a website with more than a few users anyway :)

I believe you need a connection pooler and asynchronous database I/O if you
want to talk to the database from a websocket handler. I haven't looked at
this in detail.


Is there any possibility that Tulip could be backported to Python 2.7, or
> are we kinda past caring about 2.x?
>

No, it cannot be backported to Python 3.2 or earlier because it requires
the "yield from" syntax introduced in Python 3.3.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: websockets

[Daniel Swarbrick]

On Wednesday, April 17, 2013 8:10:15 AM UTC+2, Aymeric Augustin wrote:
>
>
> Yes, that's why https://github.com/aaugustin/django-c10k-demo/ builds 
> upon Tulip.
>
> Unfortunately, that choice makes it unsuitable for inclusion in Django 
> until we drop support for Python 3.3 and all earlier versions, which isn't 
> going to happen soon.
>
> There's another issue: Django is designed to be deployed over WSGI, and 
> WSGI is incompatible with websockets. A new revision of WSGI would help.
>
>
It's encouraging to hear that WebSockets are at least on some sort of 
hypothetical roadmap. I've been using gevent-socketio in a Django app in 
small scale production use, for about one year now. One of the biggest 
drawbacks of WebSockets has been the very slow response from the likes of 
Apache and Nginx to offer support for them. For this reason, I deployed my 
setup using a custom worker class in gunicorn (custom, mainly for the 
reason of supporting SSL).

WebSockets are difficult, if not impossible, to (reverse) proxy via 
traditional means. Apache's mod_proxy certainly doesn't work, nor will any 
other HTTP 1.0 proxy (of which there are still a few), since the lack the 
ability to handle the protocol upgrade. I've seen a few abortive attempts 
at writing simple TCP proxy modules, but often these will do the WebSocket 
protocol upgrade too early, before it gets to your WSGI app - which is a 
problem for gevent-socketio at least, since it needs to actually see 
whether the client is offering a WS protocol upgrade, and choose an 
alternate transport if not. IIRC, the main reason why "strict" WSGI does 
not support WebSockets is that WSGI expects to specify the response content 
length, which is obviously not possible for a long running WebSocket 
connection.

uWSGI are doing some interesting stuff with WebSockets, and since version 
1.4, they are supported natively by the uWSGI app server. I've read some 
reports that mod_uwsgi (for Apache) can also handle WebSockets in an Apache 
environment. I believe it does this by passing the raw HTTP body through to 
the WSGI app. Since the uWSGI app server can terminate HTTP/S connections 
directly though, it would be feasible to run this without any Apache / 
Nginx at all - although I'm not sure whether this is recommended in a 
production setup.

On the pure Django side of things, one of the challenges I encountered was 
"IDLE IN TRANSACTION" hanging DB connections in the long-running WebSocket 
views. I really ought to research a more elegant solution to this, but for 
now I'm just doing all my DB queries early in the view, then fairly 
brutally closing the DB connection before entering the long-running WS poll 
loop. I suspect that manual transaction handling would be a better 
alternative to this, but it does raise an interesting point - namely, that 
Django has long been designed for a fairly vanilla request/response method 
of operation, and may need some rethinking if it is to support long-running 
connections.

Is there any possibility that Tulip could be backported to Python 2.7, or 
are we kinda past caring about 2.x?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: websockets

[Aymeric Augustin]

On 16 avr. 2013, at 23:03, Jonathan Slenders  
wrote:

> Maybe it's worth noting that Guido is working on a Tulip, a specification for 
> an asynchronous API in Python 3, this to get some consensus. Right now, there 
> is almost zero compatibility between al the different approaches: twisted, 
> tornado, gevent, etc...
> 
> If we decide to go for one technology, better be future proof.

Yes, that's why https://github.com/aaugustin/django-c10k-demo/ builds upon 
Tulip.

Unfortunately, that choice makes it unsuitable for inclusion in Django until we 
drop support for Python 3.3 and all earlier versions, which isn't going to 
happen soon.

There's another issue: Django is designed to be deployed over WSGI, and WSGI is 
incompatible with websockets. A new revision of WSGI would help.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.