Re: Feedback on Django Channels

[Andrew Godwin]

The channel layer could indeed use SECRET_KEY, since it's loaded in via a
Django codepath in both the protocol server and the worker server.

It's better done as a channel layer feature rather than in Channels/Daphne
directly, as then it requires no extra supporting code in anything else
that does ASGI (like the WSGI adapter). The layer could also have options
to turn it on only for certain channels (e.g. only http.request) - it would
be harder to make deeper encryption choices, though, as things like the
body content and headers both come through as a single message key. Maybe
let people provide a regex to match against e.g. http.request and
websocket.connect's `path` key.

It's probably something we could pay for someone to work on for the "main"
(Redis-backed) layer implementation? I could add it in too, but it's not as
high priority as some other stuff I have to get to.

Andrew
On 21 Mar 2016 5:50 pm, "Josh Smeaton"  wrote:

> Assuming the frontend has access to DJANGO_SETTINGS_MODULE, couldn't it
> use the SECRET_KEY to encrypt the message before passing it to the message
> broker? On message receipt it could then use the SECRET_KEY to decrypt the
> message. It'd be nice if encryption were an option encoded within the
> message or route somehow, so it could be optionally turned on for sensitive
> data. I'm guessing that there's probably an issue with my assumptions
> though.
>
> On Tuesday, 22 March 2016 02:56:12 UTC+11, Andrew Godwin wrote:
>>
>>
>>
>> On Mon, Mar 21, 2016 at 12:11 PM, Sean Brant  wrote:
>>
>>> How does the new channels model handle requests with sensitive
>>> information? Say you have a login view and the user is submitting a
>>> username/password. Does the password get serialized into the message queue
>>> as plain text? If so is that a security concern users should be aware of?
>>>
>>>
>> With the default backend (redis), yes, it goes over the wire and into
>> Redis as plain text - the overhead of both on-the-wire and at-rest
>> encryption would be reasonably high, so I didn't include it by default;
>> however, I very much hope we'll get at least one channel layer
>> implementation that does do this.
>>
>> For Redis, at least, you can run a TLS tunnel between machines and point
>> Channels at the local end of the tunnel to get transport security (as it
>> has none of its own). At-rest is a bit harder - I plan to just turn my
>> redis nodes into memory-only mode so that the data is never persisted, at
>> least (channels doesn't need persistance beyond about a minute), but I can
>> imagine a channel layer where you pass a symmetric encryption key in its
>> configuration or similar.
>>
>> Andrew
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/3002448c-f36d-4f70-a2fd-c4d5e9bb2c34%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urDr0LWvgK%2BRx9maEPezoFxqyFXCzNQbHedJwwvJM9pgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feedback on Django Channels

[Josh Smeaton]

Assuming the frontend has access to DJANGO_SETTINGS_MODULE, couldn't it use 
the SECRET_KEY to encrypt the message before passing it to the message 
broker? On message receipt it could then use the SECRET_KEY to decrypt the 
message. It'd be nice if encryption were an option encoded within the 
message or route somehow, so it could be optionally turned on for sensitive 
data. I'm guessing that there's probably an issue with my assumptions 
though.

On Tuesday, 22 March 2016 02:56:12 UTC+11, Andrew Godwin wrote:
>
>
>
> On Mon, Mar 21, 2016 at 12:11 PM, Sean Brant  > wrote:
>
>> How does the new channels model handle requests with sensitive 
>> information? Say you have a login view and the user is submitting a 
>> username/password. Does the password get serialized into the message queue 
>> as plain text? If so is that a security concern users should be aware of?
>>
>>
> With the default backend (redis), yes, it goes over the wire and into 
> Redis as plain text - the overhead of both on-the-wire and at-rest 
> encryption would be reasonably high, so I didn't include it by default; 
> however, I very much hope we'll get at least one channel layer 
> implementation that does do this.
>
> For Redis, at least, you can run a TLS tunnel between machines and point 
> Channels at the local end of the tunnel to get transport security (as it 
> has none of its own). At-rest is a bit harder - I plan to just turn my 
> redis nodes into memory-only mode so that the data is never persisted, at 
> least (channels doesn't need persistance beyond about a minute), but I can 
> imagine a channel layer where you pass a symmetric encryption key in its 
> configuration or similar.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3002448c-f36d-4f70-a2fd-c4d5e9bb2c34%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Shai Berger]

Hi,

On Monday 21 March 2016 13:01:27 hcharg...@medici.tv wrote:
> 
> I agree with Karen that documenting that some functions don't work is not
> helping. Especially since this is Django's fault, not MySQL's. I mean,
> sure, MySQL has some very weird isolation modes, I definitely agree with
> Shai on that, but an ORM's job is to know all that, and to use the database
> in such ways that it is safe. I reckon those safe ways exist,

I disagree. The ORM cannot keep you safe against MySql's REPEATABLE READ.

> and that the
> problem in Django is not the MySQL default isolation level, but the
> DELETE-SELECT-INSERT way of updating M2Ms.

The reason for that, and the major point you seem to be missing, is that 
Django 1.8's DELETE-SELECT-INSERT dance for updating M2M's is not reserved to 
Django's internals; you can be quite certain that there are a *lot* of similar 
examples in user code. And that user code, making assumptions that hold 
everywhere except with MRR, are not buggy. It is MySql's documented behavior 
that is insensible.

> Incidentally, to prove my point,
> this has been changed in Django 1.9 and data-loss doesn't happen anymore,
> in that same default isolation level.
> 

That indeed seems to support your point, but it is far from proving it.

It is trivial to show that MRR simply isn't repeatable read:

create table yoyo (id int, age int, rank int);
insert yoyo values(1,2,3);

Now, using MySql's so-called REPEATABLE READ,

SESSION A   SESSION B
==
SET autocommit=0;   SET autocommit=0;

SELECT * FROM yoyo; SELECT age FROM yoyo;
(1,2,3) (2)

UPDATE yoyo set age=5;  
(ok, 1 row affected)

UPDATE yoyo set rank=7;
(blocks)

COMMIT;
(ok)
(resumes)
(ok, 1 row affected)


SELECT age from yoyo;
(5)


Session B sees, in its reads, changes made by session A after B's transaction 
started. This is called, in the SQL standard, "a non-repeatable read". This is 
*exactly* the situation that REPEATABLE READ is required to prevent. Calling a 
transaction isolation level that allows this "REPEATABLE READ" is a lie. At 
the very least, MySql needs to change the name of this isolation level to 
something less misleading.

> It seems that setting a READ COMMITTED default level would help in that
> particular schedule of operations that is described in the ticket, but I
> disagree that it will work always and not cause problems elsewhere. For
> example, given a different schedule of the same operations needed to update
> a M2M, (DELETE-SELECT-INSERT), under READ COMMITTED, data-loss could still
> happen (see attachment).
> 

I can reproduce this. But frankly, I cannot understand it. Can you explain 
what happens here? Point to some documentation? The way I understand it, the 
DELETE from session B either attempts to delete the pre-existing record (in 
which case, when it resumes after the lock, it should report "0 records 
affected" -- that record has been already deleted) or the one inserted by 
session A (in which case, the second SELECT in session B should retrieve no 
records). In either case, the SELECT in session B should not be finding any 
record which has already been deleted, because all deletions have either been 
committed (and we are in READ COMMITTED) or have been done in its own 
transaction. What is going on here?

The point of the above question, besides academic interest, is that this 
behavior is also suspect of being a bug. And if it is, Django should try to 
work around it and help users work around it, but not base its strategic 
decisions on its existence.

> 
> In my opinion, the fix should not be to set a different default isolation
> level, as that could trigger other problems that may be very hard to find.
> I was lucky to find this one. I think that Django should find sequences of
> operations that are proven to be safe under specific isolation levels.

Again: The burden of "finding operation sequences" on Django is relatively 
small -- most ORM operations are single queries. Most of the burden created by 
MySql's REPEATABLE READ falls on users, who should not be expected to deal 
with such blatant violations of common standards and terminology.

Shai.

Re: Index expressions

[Tim Graham]

Sorry, but if you haven't started a proposal yet and haven't contributed to 
Django before, it's unlikely you'll develop a successful proposal in less 
than a week (the application deadline is Friday).

On Monday, March 21, 2016 at 2:59:43 PM UTC-4, aleksand...@gmail.com wrote:
>
> Hello, Tim.
>
> Is there free tasks for me, I'm a new one in django(0.5 year), but it's 
> really interesting for me and I want to be useful for us and have a good 
> summer with great experience.
>
> waiting your answer and making a contribution, Alex
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cf2f89cd-9b6e-4870-b8b4-365603768b42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: code.djangoproject vs docs.djangoproject

[Tim Graham]

Copying my response from the ticket, "The wiki isn't part of the official 
documentation so a Trac ticket isn't needed. Just make sure you are logged 
in and use the "Edit this page" button at the bottom of the page. Thanks!"

On Monday, March 21, 2016 at 2:41:54 PM UTC-4, Becka R. wrote:
>
> Hi!
>
> I just made a ticket, and want to follow it up and fix some of the links 
> on the Django resources 
>  page.  Where is the 
> source for the code.djangoproject? Is it in the same repo as the  Sorry if 
> this is in the docs; I'm trying to figure out my way around. 
>
> Thanks!
>
> Becka
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3c7d00f0-a404-44ce-9190-d20d2af8e63c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Index expressions

[aleksandrsofienko]

Hello, Tim.

Is there free tasks for me, I'm a new one in django(0.5 year), but it's 
really interesting for me and I want to be useful for us and have a good 
summer with great experience.

waiting your answer and making a contribution, Alex

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/37085db8-91bb-4d9c-b583-c0323c813456%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

code.djangoproject vs docs.djangoproject

[Becka]

Hi!

I just made a ticket, and want to follow it up and fix some of the links on
the Django resources 
page.  Where is the source for the code.djangoproject? Is it in the same
repo as the  Sorry if this is in the docs; I'm trying to figure out my way
around.

Thanks!

Becka

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANOxe%2B13PH1UW0pSUYmzFFswWUWs3jGurPacn%3DMzePLK9zzo9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Aymeric Augustin]

On 21 Mar 2016, at 15:02, Hugo Chargois  wrote:
> 
> But here, we have "two non-conflicting updates (no-ops, even) causes data to 
> be lost". I bet no one would call this safe.

Yes, this one was clearly a bug (and, if I understand correctly, it was 
resolved by a not-directly-related change, originally considered a performance 
optimization).

What I meant, but failed to convey clearly, is: can we derive general and 
useful rules from this bug, or are we stuck with dealing with such problems on 
a case by case basis?

In theory, we could:

1. make a list of “atomic” ORM operations (create, update, delete, etc.) — 
things you write in one line and thus expect to complete consistently
2. figure out which operations incur multiple queries — concrete inheritance 
and many-to-many relationships come to mind
3. investigate what unexpected results could occur in the presence of 
concurrent queries — I have no idea how to do that exhaustively

Furthermore, I suspect the answer to 3. will be “anything could happen” at 
lower isolation levels and “that looks be safe” at higher isolation levels. I 
also suspect there’s little we could do to improve the situation at lower 
isolation levels, except if we have other instances of unoptimized M2M code. 
(Optimizing for performance means making fewer queries means improving chances 
of transactional correctness.)

In the end, “figure out where the ORM could do fewer queries” sounds like a 
easier plan than “figure out all possible sequences of queries and whether 
they’re safe”.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/44DC4C9E-74FC-43A0-8394-F0035DED3970%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: relative path in {% extends "...base.html" %} like relative import

[Vitaly Bogomolov]

>
> There are backwards-compatibility concerns as a leading dot can no longer 
> be used in a template name, though if developers are following the advice 
> about template namespacing it wouldn't be a problem.
>

Mimicking for python import paths it's just my proposal. If this violate 
some guidelines, any syntax can be implemented. I'm put a comment to code, 
where it can be done. You can make it yourself or notify me, what Django 
team prefer.
 

>
> Still, I am not sure if the Django Template Language should evolve new 
> features like this. Is there any prior art in Jinja2? I couldn't find any 
> based on this answer: 
> http://stackoverflow.com/questions/2180247/does-the-jinja2-templating-language-have-the-concept-of-here-current-director
>
> I see you created a pull request: 
> https://github.com/django/django/pull/6318
>

fuh, 'All checks have passed' :)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/da7e47f1-7289-4fb3-aad3-c172b4096660%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: relative path in {% extends "...base.html" %} like relative import

[Tim Graham]

There are backwards-compatibility concerns as a leading dot can no longer 
be used in a template name, though if developers are following the advice 
about template namespacing it wouldn't be a problem.

Still, I am not sure if the Django Template Language should evolve new 
features like this. Is there any prior art in Jinja2? I couldn't find any 
based on this answer: 
http://stackoverflow.com/questions/2180247/does-the-jinja2-templating-language-have-the-concept-of-here-current-director

I see you created a pull request: https://github.com/django/django/pull/6318

On Monday, March 21, 2016 at 4:47:49 AM UTC-4, Florian Apolloner wrote:
>
>
>
> On Monday, March 21, 2016 at 8:25:46 AM UTC+1, Vitaly Bogomolov wrote:
>>
>> Here is code, that implement this feature for Django 1.4/1.9:
>> https://github.com/vb64/django.templates.relative.path 
>> 
>>
>
> Since we are (usually) talking about paths, I would prefer "./base.html" 
> and "../base.html", if at all. Mimicking python import paths here seems 
> counter intuitive to me. If the changes against core Django are relatively 
> minimal and a nice patch with tests were put up, I would not stand in the 
> way of merging.
>
> Cheers,
> Florian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dc04291b-e48b-4033-b628-56ed6751f643%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feedback on Django Channels

[Andrew Godwin]

On Mon, Mar 21, 2016 at 12:11 PM, Sean Brant  wrote:

> How does the new channels model handle requests with sensitive
> information? Say you have a login view and the user is submitting a
> username/password. Does the password get serialized into the message queue
> as plain text? If so is that a security concern users should be aware of?
>
>
With the default backend (redis), yes, it goes over the wire and into Redis
as plain text - the overhead of both on-the-wire and at-rest encryption
would be reasonably high, so I didn't include it by default; however, I
very much hope we'll get at least one channel layer implementation that
does do this.

For Redis, at least, you can run a TLS tunnel between machines and point
Channels at the local end of the tunnel to get transport security (as it
has none of its own). At-rest is a bit harder - I plan to just turn my
redis nodes into memory-only mode so that the data is never persisted, at
least (channels doesn't need persistance beyond about a minute), but I can
imagine a channel layer where you pass a symmetric encryption key in its
configuration or similar.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1upA2gAvRru8Gfn1DkEx8Bfy_j4ZPKeKkBZQH5aMEaH92w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feedback on Django Channels

[Andrew Godwin]

On Mon, Mar 21, 2016 at 9:55 AM, David Evans  wrote:

> On the static files question, I'm about to release v3 of WhiteNoise (
> http://whitenoise.evans.io/en/latest/changelog.html) which provides the
> option to integrate via Django middlware, rather than wsgi middleware. (It
> uses the FileResponse class, which didn't exist when I first wrote
> WhiteNoise.) I'm hoping (though I haven't tested it yet) that this will
> work out of the box with Channels, as it's only using standard Django APIs.
>
>
Yes, as long as you use the standard Django file response stuff it should
work just fine (I made sure of that so staticfiles' serving class worked).

The only difference is that Channels prefers a chunking size of around
256KB, whereas the default is 2KB, which causes a whole lot of messages,
but I can move the code that adjusts it up for FileResponses to the main
handling code rather than just Staticfiles, so it should be fine.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uoPd-2XZOBWbRORBRQ1V%2Bt5D_BVfeRfS2EAzroTtmCW6g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feedback on Django Channels

[Sean Brant]

How does the new channels model handle requests with sensitive information?
Say you have a login view and the user is submitting a username/password.
Does the password get serialized into the message queue as plain text? If
so is that a security concern users should be aware of?

Sean

On Mon, Mar 21, 2016 at 7:55 AM, David Evans  wrote:

> On the static files question, I'm about to release v3 of WhiteNoise (
> http://whitenoise.evans.io/en/latest/changelog.html) which provides the
> option to integrate via Django middlware, rather than wsgi middleware. (It
> uses the FileResponse class, which didn't exist when I first wrote
> WhiteNoise.) I'm hoping (though I haven't tested it yet) that this will
> work out of the box with Channels, as it's only using standard Django APIs.
>
> Dave
>
>
> On Thursday, 17 March 2016 16:42:05 UTC, Jacob Kaplan-Moss wrote:
>>
>> Hi folks (and especially Andrew):
>>
>> I've just completed writing an example Channels app [1] for an article
>> about Channels [2]. Overall it was a super-pleasant experience: Channels
>> seems pretty solid, the APIs make sense to me, and I couldn't be more
>> excited about the new things this'll let me do!
>>
>> In the interests of making this thing as solid as possible before we
>> merge it into Django, I do have some feedback on some of the hiccups I
>> encountered. Roughly in order of severity (as I perceive it), they are:
>>
>> 1. Debugging errors:
>>
>> I found debugging errors that happen in a consumer to be *really*
>> difficult -- errors mostly presented as things just silently not working.
>> It took a ton of messing around with logging setups before I could get
>> anything of use dumped to the console. This isn't strictly a Channels issue
>> (I've noted similar problems with AJAX views and errors in Celery tasks),
>> but I think Channels sorta brings the issue to a head.
>>
>> I think we need some better defaults, and simple, clear documentation, to
>> make sure that exceptions go somewhere useful.
>>
>> 2. Static files:
>>
>> I had trouble getting static files served. I'm used to using Whitenoise (
>> http://whitenoise.evans.io/en/stable/) for small-to-medium-ish sites
>> that don't need a proper static server, but of course it doesn't work with
>> Channels since Channels doesn't use WSGI! I found the (undocumented)
>> StaticFilesConsumer (
>> https://github.com/jacobian/channels-example/blob/master/chat/routing.py#L5-L8),
>> but that feels less than ideal.
>>
>> I think this might be an opportunity, however. If Daphne learns how to
>> serve static files (perhaps via optional integration with Whitenoise?),
>> this would actually make static media in Django a bit easier by default.
>>
>> [I would be happy to work on this if I get a thumbsup.]
>>
>> 3. WebSocket routing:
>>
>> Channels routes all WebSocket connections to a single set of consumers
>> (the `websocket.*` consumers). This means that if you want multiple
>> WebSocket URLs in a single app you need to manually parse the path. And, to
>> make things more complicated, you only get the WebSocket path passed in the
>> connection message, so you have to also use a channel session to keep track.
>>
>> This feels like a distinct step back from the URL routing we already have
>> in Django, and it was surprising to me to have to do this by hand. It
>> definitely felt like Channels is missing some sort of WebSocket URL router.
>>
>> I had a brief chat with Andrew, who indicates that he'd planned for this
>> to be a post-1.0 feature. I'm not sure I agree - it feels pretty
>> fundamental - but I'd like to hear other thoughts.
>>
>> [This is another thing I'd be interested in working on, assuming a
>> thumbs.]
>>
>> ---
>>
>> Has anyone else here played with Channels? Are there other things I'm
>> missing that might need to be included before we merge this?
>>
>> Jacob
>>
>> [1] https://github.com/jacobian/channels-example
>>
>> [2]
>> https://blog.heroku.com/archives/2016/3/17/in_deep_with_django_channels_the_future_of_real_time_apps_in_django
>> ?
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ba29cc7a-90fc-4cf2-a65d-da782e83b3ca%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from t

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Hugo Chargois]

Le lundi 21 mars 2016 13:35:19 UTC+1, Aymeric Augustin a écrit :
>
>
> The problem is to determine what “safe” means here :-) I’m afraid this 
> won’t be possible in general (at least not in a remotely efficient fashion) 
> because Django inevitably depends on the guarantees of the underlying 
> database, and those guarantees are weak at lower isolation levels.
>

Sure, some of it is left to interpretation. There are some tricky cases, 
like two conflicting updates happening at the same time. Would safe mean 
that they should they both succeed, even if there would be a lost update? 
Or would it mean that one of should them fail? But here, we have "two 
non-conflicting updates (no-ops, even) causes data to be lost". I bet no 
one would call this safe.

The guarantees offered by the database may not be perfect, but my point is 
that those guarantees are used badly in Django. Consistent read is a 
"feature", it can be used to make things "safer", but in that bug, it's 
simply used incorrectly because of false assumptions. I reckon that given 
those same guarantees, and knowing how they may be different from backend 
to backend, they can be used to have a "safer" behavior than this.

That's also why I don't think lowering the isolation level is a good idea. 
The higher the isolation level, the more "guarantees" to build upon there 
are. This bug is an unfortunate case where an assumption that is true in a 
lower isolation level is not anymore in a higher one. That doesn't mean 
that the assumption was safe to make in the first place.

I applied the patch proposed by Shai (thanks!) in the ticket, that replaces 
the SELECT by a SELECT FOR UPDATE. This effectively fixes the bug. I think 
this is the right kind of fix. Fix the code, do not change the isolation 
level because it fixes the code by chance.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/08fed447-1ad9-4e38-a23c-e170d255cb07%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feedback on Django Channels

[David Evans]

On the static files question, I'm about to release v3 of WhiteNoise 
(http://whitenoise.evans.io/en/latest/changelog.html) which provides the 
option to integrate via Django middlware, rather than wsgi middleware. (It 
uses the FileResponse class, which didn't exist when I first wrote 
WhiteNoise.) I'm hoping (though I haven't tested it yet) that this will 
work out of the box with Channels, as it's only using standard Django APIs.

Dave

On Thursday, 17 March 2016 16:42:05 UTC, Jacob Kaplan-Moss wrote:
>
> Hi folks (and especially Andrew):
>
> I've just completed writing an example Channels app [1] for an article 
> about Channels [2]. Overall it was a super-pleasant experience: Channels 
> seems pretty solid, the APIs make sense to me, and I couldn't be more 
> excited about the new things this'll let me do! 
>
> In the interests of making this thing as solid as possible before we merge 
> it into Django, I do have some feedback on some of the hiccups I 
> encountered. Roughly in order of severity (as I perceive it), they are:
>
> 1. Debugging errors:
>
> I found debugging errors that happen in a consumer to be *really* 
> difficult -- errors mostly presented as things just silently not working. 
> It took a ton of messing around with logging setups before I could get 
> anything of use dumped to the console. This isn't strictly a Channels issue 
> (I've noted similar problems with AJAX views and errors in Celery tasks), 
> but I think Channels sorta brings the issue to a head. 
>
> I think we need some better defaults, and simple, clear documentation, to 
> make sure that exceptions go somewhere useful.
>
> 2. Static files:
>
> I had trouble getting static files served. I'm used to using Whitenoise (
> http://whitenoise.evans.io/en/stable/) for small-to-medium-ish sites that 
> don't need a proper static server, but of course it doesn't work with 
> Channels since Channels doesn't use WSGI! I found the (undocumented) 
> StaticFilesConsumer (
> https://github.com/jacobian/channels-example/blob/master/chat/routing.py#L5-L8),
>  
> but that feels less than ideal.
>
> I think this might be an opportunity, however. If Daphne learns how to 
> serve static files (perhaps via optional integration with Whitenoise?), 
> this would actually make static media in Django a bit easier by default.
>
> [I would be happy to work on this if I get a thumbsup.]
>
> 3. WebSocket routing:
>
> Channels routes all WebSocket connections to a single set of consumers 
> (the `websocket.*` consumers). This means that if you want multiple 
> WebSocket URLs in a single app you need to manually parse the path. And, to 
> make things more complicated, you only get the WebSocket path passed in the 
> connection message, so you have to also use a channel session to keep track.
>
> This feels like a distinct step back from the URL routing we already have 
> in Django, and it was surprising to me to have to do this by hand. It 
> definitely felt like Channels is missing some sort of WebSocket URL router. 
>
> I had a brief chat with Andrew, who indicates that he'd planned for this 
> to be a post-1.0 feature. I'm not sure I agree - it feels pretty 
> fundamental - but I'd like to hear other thoughts. 
>
> [This is another thing I'd be interested in working on, assuming a thumbs.]
>
> ---
>
> Has anyone else here played with Channels? Are there other things I'm 
> missing that might need to be included before we merge this?
>
> Jacob
>
> [1] https://github.com/jacobian/channels-example
>
> [2] 
> https://blog.heroku.com/archives/2016/3/17/in_deep_with_django_channels_the_future_of_real_time_apps_in_django
> ?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ba29cc7a-90fc-4cf2-a65d-da782e83b3ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Aymeric Augustin]

Hello,

> On 21 Mar 2016, at 12:01, hcharg...@medici.tv wrote:
> 
> In my opinion, the fix should not be to set a different default isolation 
> level, as that could trigger other problems that may be very hard to find. I 
> was lucky to find this one. I think that Django should find sequences of 
> operations that are proven to be safe under specific isolation levels.

The problem is to determine what “safe” means here :-) I’m afraid this won’t be 
possible in general (at least not in a remotely efficient fashion) because 
Django inevitably depends on the guarantees of the underlying database, and 
those guarantees are weak at lower isolation levels.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6966EE96-8A48-403A-A39A-2B97BB88203F%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Cristiano Coelho]

Shai Berger, this explanation is pure gold! Definetely better than MySQL's 
one. 

Now I may agree on that changing the isolation level should be probably 
left to a major release, or either have a huge warning on the patch notes. 
I personally can't think of any project I have done that this would be an 
issue since database errors are usually well handled (which I think doing 
this change would increase, since you will be reading more modified data 
from other transactions if I read correct) and critical operations should 
be handled with either a select for update or transactional operations 
(such as doing col = col + 1).

El lunes, 21 de marzo de 2016, 4:27:59 (UTC-3), Shai Berger escribió:
>
> First of all, I would like to say that I strongly support the move to READ 
> COMITTED, including backporting it to 1.8.x. 
>
> But we also need to explain: REPEATABLE READ is a higher transaction 
> isolation 
> level than READ COMMITTED. If you have problematic code, it should lead to 
> more deadlocks and/or transactions failing at commit time (compared to 
> READ 
> COMMITTED), not to data loss. The reason we get data losses is MySql's 
> unique 
> interpretation of REPEATABLE READ. If you're interested in the details 
> (and if 
> you use MySql, you should be), read on. 
>
> With MySql's REPEATABLE READ, the "read" operations -- SELECT statements 
> -- 
> indeed act like they act in the usual REPEATABLE READ: Once you've read 
> some 
> table, changes made to that table by other transactions will not be 
> visible 
> within your transaction. But "write" operations -- UPDATE, DELETE, INSERT 
> and 
> the like -- act as if they're under READ COMMITTED, affecting (and 
> affected by) 
> changes committed by other transactions. The result is, essentially, that 
> within a transaction, the reads are not guaranteed to be consistent with 
> the 
> writes [1]. 
>
> In particular, in the bug[2] that caused this discussion, we get the 
> following   
> behavior in one transaction: 
>
> (1) BEGIN TRANSACTION 
>
> (2) SELECT ... FROM some_table WHERE some_field=some_value 
> (1 row returned) 
>
> (3) (some other transactions commit) 
>
> (4) SELECT ... FROM some_table WHERE some_field=some_value 
> (1 row returned, same as above) 
>
> (5) DELETE some_table WHERE some_field=some_value 
> (answer: 1 row deleted) 
>
> (6) SELECT ... FROM some_table WHERE some_field=some_value 
> (1 row returned, same as above) 
>
> (7) COMMIT 
> (the row that was returned earlier is no longer in the 
> database) 
>
> Take a minute to read this. Up to step (5), everything is as you would 
> expect; 
> you should find steps (6) and (7) quite surprising. 
>
> This happens because the other transactions in (3) deleted the row that is 
> returned in (2), (4) & (6), and inserted another one where 
> some_field=some_value; that other row is the row that was deleted in (5). 
> The 
> row that this transaction selects was not seen by the DELETE, and hence 
> not 
> changed by it, and hence continues to be visible by the SELECTs in our 
> transaction. But when we commit, the row (which has been deleted) no 
> longer 
> exists. 
>
> I have expressed elsewhere my opinion of this behavior as a general 
> database 
> feature, and feel no need to repeat it here; but I think that, if 
> possible, it 
> is Django's job as a framework to protect its users from it, at least as a 
> default. 
>
> On Monday 21 March 2016 02:25:37 Cristiano Coelho wrote: 
> > What performance changes can you expect doing this change? It is 
> probably 
> > that default on MySQL for a good reason. 
>
> The Django project is usually willing to give up quite a lot of 
> performance in 
> order to prevent data losses. I agree that this default on MySql is 
> probably 
> for a reason, but I don't think it can be a good reason for Django. 
>
> Have fun, 
> Shai. 
>
> [1] https://dev.mysql.com/doc/refman/5.7/en/innodb-consistent-read.html 
> [2] https://code.djangoproject.com/ticket/26347 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9a0de040-4943-4a57-9153-312b30ee8524%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [GSoC 2016]Proposal: Validity check at client and dynamic form framework

[Cristiano Coelho]

I don't have enough enough knowledge on how GSoC works and what the django 
team expects from people proposing projects but if it was me I would rather 
have a very well implemented, tested and with a good level of customization 
client side validation than two not so well features but that's up to the 
times used. I'm sure someone more related to this will be able to give you 
a better answer but the two ideas are definitely interesting :)


El domingo, 20 de marzo de 2016, 23:52:32 (UTC-3), 7sDream escribió:
>
> Thanks for your advice! 
>
> My first thought is the client side validation too. Because I used some 
> Forms in my Django project, after write javascript for each form, I feel 
> this is redundant.
>
> Then I thought of another common demand of Form, the dynamic form. After I 
> draw the chart, I also find it is a little complicated so I arranged for 
> eight weeks to finish it. 
>
> Do you think I should focus on client side validation (delete dynamic form 
> part)? Or describe the complexity of dynamic form and give a more detailed 
> schedule?
>
> Thanks again!
>
> 在 2016年3月21日星期一 UTC+8上午2:23:16，Cristiano Coelho写道：
>>
>> The client side validation is a very good idea, other frameworks such as 
>> ASP.NET MVC already has some basic client side validation tied to model 
>> fields (equivalent to django forms) and also provides a very easy way to 
>> add custom javascript validation and tie to the model/form.
>>
>> For the second part about dynamic forms, it really looks complicated to 
>> implement, looks like a job for more than one person.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d73fce50-82a6-4404-8f89-a7f49fe42d3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Feature suggestion: Count database queries

[Shai Berger]

Hi,

When working with large projects, it is sometimes the case that you run into 
problems where some edge case causes a pathologically high number of queries 
for processing a specific request. These cases are hard to debug, because it is 
not always easy to reconstruct the edge-case and even identify the exact 
request (because post-data is not always available). Django collects all the 
queries on a connection if DEBUG=True, but when it's hard to reproduce, you 
sometimes need some sort of monitoring in production, where DEBUG=True is not 
really an option.

Something that could be very helpful is just to count the queries -- a counter 
cab be incremented with no noticeable overhead in CursorWrapper, and it can be 
reset and/or examined at request start/end points (even with a Middleware, as 
long as we have them). This would enable a project to define alerts or errors 
or whatever when there are two many database queries, at a level where enough 
information can be given to help solve the problem.

Similar measures may be applicable to other resources as well, if they are 
handled by thread-local objects like the database connections. I'm not sure 
which ones apply.

Thoughts?

Thanks,
Shai.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[hchargois]

Hi,

I'm the ticket OP.

As a user, I definitely expect that my ORM won't eat my records when I save 
something, even with concurrency. Furthermore, I expect it to work with the 
default configuration of the backend, or if I have to change it, the ORM 
should refuse to work until the configuration is right, or at least display 
a huge warning.

I agree with Karen that documenting that some functions don't work is not 
helping. Especially since this is Django's fault, not MySQL's. I mean, 
sure, MySQL has some very weird isolation modes, I definitely agree with 
Shai on that, but an ORM's job is to know all that, and to use the database 
in such ways that it is safe. I reckon those safe ways exist, and that the 
problem in Django is not the MySQL default isolation level, but the 
DELETE-SELECT-INSERT way of updating M2Ms. Incidentally, to prove my point, 
this has been changed in Django 1.9 and data-loss doesn't happen anymore, 
in that same default isolation level.

It seems that setting a READ COMMITTED default level would help in that 
particular schedule of operations that is described in the ticket, but I 
disagree that it will work always and not cause problems elsewhere. For 
example, given a different schedule of the same operations needed to update 
a M2M, (DELETE-SELECT-INSERT), under READ COMMITTED, data-loss could still 
happen (see attachment).

Granted, this would be less of a problem, because it would (I think) 
require a very close synchronicity of the two sets of operations. This 
requirement is not needed in the ticket's case. If we refer to the 
step-by-step in the ticket, this isn't evident from it, but between the two 
SELECTs in sessions A and B, and the DELETE in session A, a "great" deal of 
time happens. This is because (if I understand correctly) a transaction is 
started to update each of the model's fields, and that is that single 
transaction that includes the update to the M2M field. Thus, the 
requirement is only that those two long transactions are synchronized.

By a matter of fact, this means that the bigger the model, the easier it 
will be to trigger the bug. I tried to replicate it in a minimal toy 
project, and I couldn't have enough synchronicity to replicate it (I only 
tried by double-clicking fast on the save button in the admin site, 
though). On my production project, I can reproduce the bug every time.

In my opinion, the fix should not be to set a different default isolation 
level, as that could trigger other problems that may be very hard to find. 
I was lucky to find this one. I think that Django should find sequences of 
operations that are proven to be safe under specific isolation levels. 
Those may need to be tailored for each backend. Maybe for PostgreSQL, that 
DELETE-SELECT-INSERT is safe, but it simply isn't in MySQL, neither in 
REPEATABLE READ nor READ COMMITTED. Thus this sequence could be used for 
PostgreSQL, but another one should be found for MySQL.

Le lundi 21 mars 2016 09:41:00 UTC+1, Shai Berger a écrit :
>
> My recommendation to backport is based on the observation that the 
> peculiar REPEATABLE READ behavior is highly conductive to data loss in the 
> presence of concurrency, combined with a sense that it is not very well 
> known; I find it much more likely that the change will fix broken code than 
> break really working code. 
>
> On 21 במרץ 2016 09:59:25 GMT+02:00, "Anssi Kääriäinen"  > wrote:
>>
>> I'm strongly -1 on changing the default isolation level in a minor
>> release. We can recommend users switch the level and complain loudly
>> if they don't. But just changing the isolation level has potential for
>> breaking working code.
>>
>>  - Anssi
>>
>> On Mon, Mar 21, 2016 at 9:27 AM, Shai Berger > > wrote:
>>
>>>  First of all, I would like to say that I strongly support the move to READ
>>>  COMITTED, including backporting it to 1.8.x.
>>>
>>>  But we also need to explain: REPEATABLE READ is a higher transaction 
>>> isolation
>>>  level than READ COMMITTED. If you have problematic code, it should lead to
>>>  more deadlocks and/or transactions failing at commit time (compared to READ
>>>  COMMITTED), not to data loss. The reason we get data losses is MySql's 
>>> unique
>>>  interpretation of REPEATABLE READ. If you're interested in the details 
>>> (and if
>>>  you use MySql, you should be), read on.
>>>
>>>  With MySql's REPEATABLE READ, the "read" operations -- SELECT statements --
>>>  indeed act like they act in the usual REPEATABLE READ: Once you've read 
>>> some
>>>  table, changes made to that table by other transactions will not be visible
>>>  within your transaction. But "write" operations -- UPDATE, DELETE, INSERT 
>>> and
>>>  the like -- act as if they're under READ COMMITTED, affecting (and 
>>> affected by)
>>>  changes committed by other transactions. The result is, essentially, that
>>>  within a transaction, the reads are not guaranteed to be consistent with 
>>> the
>>>  writes [1].
>>>
>>>  In particular, i

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Aymeric Augustin]

We’ve had, and known about, such problems for so long that I’m also
inclined not to make that change in already released versions.

-- 
Aymeric.

> On 21 Mar 2016, at 12:33, Anssi Kääriäinen  wrote:
> 
> This being MySQL I wouldn't be surprised if changing the isolation
> level would introduce new problems. Also, user code might rely on
> Django using repeatable read. If we introduce the change in to stable
> releases, we risk breaking sites that work perfectly well currently.
> To me this is against our backwards compatibility rules which state
> that we avoid doing such changes except when completely unavoidable.
> 
> - Anssi
> 
> On Mon, Mar 21, 2016 at 10:40 AM, Shai Berger  wrote:
>> My recommendation to backport is based on the observation that the peculiar
>> REPEATABLE READ behavior is highly conductive to data loss in the presence
>> of concurrency, combined with a sense that it is not very well known; I find
>> it much more likely that the change will fix broken code than break really
>> working code.
>> 
>> On 21 במרץ 2016 09:59:25 GMT+02:00, "Anssi Kääriäinen" 
>> wrote:
>>> 
>>> I'm strongly -1 on changing the default isolation level in a minor
>>> release. We can recommend users switch the level and complain loudly
>>> if they don't. But just changing the isolation level has potential for
>>> breaking working code.
>>> 
>>> - Anssi
>>> 
>>> On Mon, Mar 21, 2016 at 9:27 AM, Shai Berger  wrote:
 
 First of all, I would like to say that I strongly support the move to
 READ
 COMITTED, including backporting it to 1.8.x.
 
 But we also need to explain: REPEATABLE READ is a higher transaction
 isolation
 level than READ COMMITTED. If you have problematic code, it should lead
 to
 more deadlocks and/or transactions failing at commit time (compared to
 READ
 COMMITTED), not to data loss. The reason we get data losses is MySql's
 unique
 interpretation of REPEATABLE READ. If you're interested in the details
 (and if
 you use MySql, you should be), read on.
 
 With MySql's REPEATABLE READ, the "read" operations -- SELECT statements
 --
 indeed act like they act in the usual REPEATABLE READ: Once you've read
 some
 table, changes made to that table by other transactions will not be
 visible
 within your transaction. But "write" operations -- UPDATE, DELETE,
 INSERT and
 the like -- act as if they're under READ COMMITTED, affecting (and
 affected by)
 changes committed by other transactions. The result is, essentially,
 that
 within a transaction, the reads are not guaranteed to be consistent with
 the
 writes [1].
 
 In particular, in the bug[2] that caused this discussion, we get the
 following
 behavior in one transaction:
 
 (1) BEGIN TRANSACTION
 
 (2) SELECT ... FROM some_table WHERE some_field=some_value
 (1 row returned)
 
 (3) (some other transactions commit)
 
 (4) SELECT ... FROM some_table WHERE some_field=some_value
 (1 row returned, same as above)
 
 (5) DELETE some_table WHERE some_field=some_value
 (answer: 1 row deleted)
 
 (6) SELECT ... FROM some_table WHERE some_field=some_value
 (1 row returned, same as above)
 
 (7) COMMIT
 (the row that was returned earlier is no longer in the
 database)
 
 Take a minute to read this. Up to step (5), everything is as you would
 expect;
 you should find steps (6) and (7) quite surprising.
 
 This happens because the other transactions in (3) deleted the row that
 is
 returned in (2), (4) & (6), and inserted another one where
 some_field=some_value; that other row is the row that was deleted in
 (5). The
 row that this transaction selects was not seen by the DELETE, and hence
 not
 changed by it, and hence continues to be visible by the SELECTs in our
 transaction. But when we commit, the row (which has been deleted) no
 longer
 exists.
 
 I have expressed elsewhere my opinion of this behavior as a general
 database
 feature, and feel no need to repeat it here; but I think that, if
 possible, it
 is Django's job as a framework to protect its users from it, at least as
 a
 default.
 
 On Monday 21 March 2016 02:25:37 Cristiano Coelho wrote:
> 
> What performance changes can you expect doing this change? It is
> probably
> that default on MySQL for a good reason.
 
 
 The Django project is usually willing to give up quite a lot of
 performance in
 order to prevent data losses. I agree that this default on MySql is
 probably
 for a reason, but I don't think it can be a good reason for Django.
 
 Have fun,
 Shai.
 
 [1] https://d

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Anssi Kääriäinen]

This being MySQL I wouldn't be surprised if changing the isolation
level would introduce new problems. Also, user code might rely on
Django using repeatable read. If we introduce the change in to stable
releases, we risk breaking sites that work perfectly well currently.
To me this is against our backwards compatibility rules which state
that we avoid doing such changes except when completely unavoidable.

 - Anssi

On Mon, Mar 21, 2016 at 10:40 AM, Shai Berger  wrote:
> My recommendation to backport is based on the observation that the peculiar
> REPEATABLE READ behavior is highly conductive to data loss in the presence
> of concurrency, combined with a sense that it is not very well known; I find
> it much more likely that the change will fix broken code than break really
> working code.
>
> On 21 במרץ 2016 09:59:25 GMT+02:00, "Anssi Kääriäinen" 
> wrote:
>>
>> I'm strongly -1 on changing the default isolation level in a minor
>> release. We can recommend users switch the level and complain loudly
>> if they don't. But just changing the isolation level has potential for
>> breaking working code.
>>
>>  - Anssi
>>
>> On Mon, Mar 21, 2016 at 9:27 AM, Shai Berger  wrote:
>>>
>>>  First of all, I would like to say that I strongly support the move to
>>> READ
>>>  COMITTED, including backporting it to 1.8.x.
>>>
>>>  But we also need to explain: REPEATABLE READ is a higher transaction
>>> isolation
>>>  level than READ COMMITTED. If you have problematic code, it should lead
>>> to
>>>  more deadlocks and/or transactions failing at commit time (compared to
>>> READ
>>>  COMMITTED), not to data loss. The reason we get data losses is MySql's
>>> unique
>>>  interpretation of REPEATABLE READ. If you're interested in the details
>>> (and if
>>>  you use MySql, you should be), read on.
>>>
>>>  With MySql's REPEATABLE READ, the "read" operations -- SELECT statements
>>> --
>>>  indeed act like they act in the usual REPEATABLE READ: Once you've read
>>> some
>>>  table, changes made to that table by other transactions will not be
>>> visible
>>>  within your transaction. But "write" operations -- UPDATE, DELETE,
>>> INSERT and
>>>  the like -- act as if they're under READ COMMITTED, affecting (and
>>> affected by)
>>>  changes committed by other transactions. The result is, essentially,
>>> that
>>>  within a transaction, the reads are not guaranteed to be consistent with
>>> the
>>>  writes [1].
>>>
>>>  In particular, in the bug[2] that caused this discussion, we get the
>>> following
>>>  behavior in one transaction:
>>>
>>>  (1) BEGIN TRANSACTION
>>>
>>>  (2) SELECT ... FROM some_table WHERE some_field=some_value
>>>  (1 row returned)
>>>
>>>  (3) (some other transactions commit)
>>>
>>>  (4) SELECT ... FROM some_table WHERE some_field=some_value
>>>  (1 row returned, same as above)
>>>
>>>  (5) DELETE some_table WHERE some_field=some_value
>>>  (answer: 1 row deleted)
>>>
>>>  (6) SELECT ... FROM some_table WHERE some_field=some_value
>>>  (1 row returned, same as above)
>>>
>>>  (7) COMMIT
>>>  (the row that was returned earlier is no longer in the
>>> database)
>>>
>>>  Take a minute to read this. Up to step (5), everything is as you would
>>> expect;
>>>  you should find steps (6) and (7) quite surprising.
>>>
>>>  This happens because the other transactions in (3) deleted the row that
>>> is
>>>  returned in (2), (4) & (6), and inserted another one where
>>>  some_field=some_value; that other row is the row that was deleted in
>>> (5). The
>>>  row that this transaction selects was not seen by the DELETE, and hence
>>> not
>>>  changed by it, and hence continues to be visible by the SELECTs in our
>>>  transaction. But when we commit, the row (which has been deleted) no
>>> longer
>>>  exists.
>>>
>>>  I have expressed elsewhere my opinion of this behavior as a general
>>> database
>>>  feature, and feel no need to repeat it here; but I think that, if
>>> possible, it
>>>  is Django's job as a framework to protect its users from it, at least as
>>> a
>>>  default.
>>>
>>>  On Monday 21 March 2016 02:25:37 Cristiano Coelho wrote:

  What performance changes can you expect doing this change? It is
 probably
  that default on MySQL for a good reason.
>>>
>>>
>>>  The Django project is usually willing to give up quite a lot of
>>> performance in
>>>  order to prevent data losses. I agree that this default on MySql is
>>> probably
>>>  for a reason, but I don't think it can be a good reason for Django.
>>>
>>>  Have fun,
>>>  Shai.
>>>
>>>  [1] https://dev.mysql.com/doc/refman/5.7/en/innodb-consistent-read.html
>>>  [2] https://code.djangoproject.com/ticket/26347
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributi

Re: relative path in {% extends "...base.html" %} like relative import

[Florian Apolloner]

On Monday, March 21, 2016 at 8:25:46 AM UTC+1, Vitaly Bogomolov wrote:
>
> Here is code, that implement this feature for Django 1.4/1.9:
> https://github.com/vb64/django.templates.relative.path 
> 
>

Since we are (usually) talking about paths, I would prefer "./base.html" 
and "../base.html", if at all. Mimicking python import paths here seems 
counter intuitive to me. If the changes against core Django are relatively 
minimal and a nice patch with tests were put up, I would not stand in the 
way of merging.

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/52be4613-1a5b-4f02-bc11-abe01281b96a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Shai Berger]

My recommendation to backport is based on the observation that the peculiar 
REPEATABLE READ behavior is highly conductive to data loss in the presence of 
concurrency, combined with a sense that it is not very well known; I find it 
much more likely that the change will fix broken code than break really working 
code. 

On 21 במרץ 2016 09:59:25 GMT+02:00, "Anssi Kääriäinen"  
wrote:
>I'm strongly -1 on changing the default isolation level in a minor
>release. We can recommend users switch the level and complain loudly
>if they don't. But just changing the isolation level has potential for
>breaking working code.
>
> - Anssi
>
>On Mon, Mar 21, 2016 at 9:27 AM, Shai Berger  wrote:
>> First of all, I would like to say that I strongly support the move to
>READ
>> COMITTED, including backporting it to 1.8.x.
>>
>> But we also need to explain: REPEATABLE READ is a higher transaction
>isolation
>> level than READ COMMITTED. If you have problematic code, it should
>lead to
>> more deadlocks and/or transactions failing at commit time (compared
>to READ
>> COMMITTED), not to data loss. The reason we get data losses is
>MySql's unique
>> interpretation of REPEATABLE READ. If you're interested in the
>details (and if
>> you use MySql, you should be), read on.
>>
>> With MySql's REPEATABLE READ, the "read" operations -- SELECT
>statements --
>> indeed act like they act in the usual REPEATABLE READ: Once you've
>read some
>> table, changes made to that table by other transactions will not be
>visible
>> within your transaction. But "write" operations -- UPDATE, DELETE,
>INSERT and
>> the like -- act as if they're under READ COMMITTED, affecting (and
>affected by)
>> changes committed by other transactions. The result is, essentially,
>that
>> within a transaction, the reads are not guaranteed to be consistent
>with the
>> writes [1].
>>
>> In particular, in the bug[2] that caused this discussion, we get the
>following
>> behavior in one transaction:
>>
>> (1) BEGIN TRANSACTION
>>
>> (2) SELECT ... FROM some_table WHERE some_field=some_value
>> (1 row returned)
>>
>> (3) (some other transactions commit)
>>
>> (4) SELECT ... FROM some_table WHERE some_field=some_value
>> (1 row returned, same as above)
>>
>> (5) DELETE some_table WHERE some_field=some_value
>> (answer: 1 row deleted)
>>
>> (6) SELECT ... FROM some_table WHERE some_field=some_value
>> (1 row returned, same as above)
>>
>> (7) COMMIT
>> (the row that was returned earlier is no longer in
>the database)
>>
>> Take a minute to read this. Up to step (5), everything is as you
>would expect;
>> you should find steps (6) and (7) quite surprising.
>>
>> This happens because the other transactions in (3) deleted the row
>that is
>> returned in (2), (4) & (6), and inserted another one where
>> some_field=some_value; that other row is the row that was deleted in
>(5). The
>> row that this transaction selects was not seen by the DELETE, and
>hence not
>> changed by it, and hence continues to be visible by the SELECTs in
>our
>> transaction. But when we commit, the row (which has been deleted) no
>longer
>> exists.
>>
>> I have expressed elsewhere my opinion of this behavior as a general
>database
>> feature, and feel no need to repeat it here; but I think that, if
>possible, it
>> is Django's job as a framework to protect its users from it, at least
>as a
>> default.
>>
>> On Monday 21 March 2016 02:25:37 Cristiano Coelho wrote:
>>> What performance changes can you expect doing this change? It is
>probably
>>> that default on MySQL for a good reason.
>>
>> The Django project is usually willing to give up quite a lot of
>performance in
>> order to prevent data losses. I agree that this default on MySql is
>probably
>> for a reason, but I don't think it can be a good reason for Django.
>>
>> Have fun,
>> Shai.
>>
>> [1]
>https://dev.mysql.com/doc/refman/5.7/en/innodb-consistent-read.html
>> [2] https://code.djangoproject.com/ticket/26347
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Django developers  (Contributions to Django itself)" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to django-developers+unsubscr...@googlegroups.com.
>To post to this group, send email to
>django-developers@googlegroups.com.
>Visit this group at https://groups.google.com/group/django-developers.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/django-developers/CALMtK1EOwq7CC2bF6nL7-GuHPv_RJ_cf_giCtchtk2gzYYMr8g%40mail.gmail.com.
>For more options, visit https://groups.google.com/d/optout.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop rec

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Anssi Kääriäinen]

I'm strongly -1 on changing the default isolation level in a minor
release. We can recommend users switch the level and complain loudly
if they don't. But just changing the isolation level has potential for
breaking working code.

 - Anssi

On Mon, Mar 21, 2016 at 9:27 AM, Shai Berger  wrote:
> First of all, I would like to say that I strongly support the move to READ
> COMITTED, including backporting it to 1.8.x.
>
> But we also need to explain: REPEATABLE READ is a higher transaction isolation
> level than READ COMMITTED. If you have problematic code, it should lead to
> more deadlocks and/or transactions failing at commit time (compared to READ
> COMMITTED), not to data loss. The reason we get data losses is MySql's unique
> interpretation of REPEATABLE READ. If you're interested in the details (and if
> you use MySql, you should be), read on.
>
> With MySql's REPEATABLE READ, the "read" operations -- SELECT statements --
> indeed act like they act in the usual REPEATABLE READ: Once you've read some
> table, changes made to that table by other transactions will not be visible
> within your transaction. But "write" operations -- UPDATE, DELETE, INSERT and
> the like -- act as if they're under READ COMMITTED, affecting (and affected 
> by)
> changes committed by other transactions. The result is, essentially, that
> within a transaction, the reads are not guaranteed to be consistent with the
> writes [1].
>
> In particular, in the bug[2] that caused this discussion, we get the following
> behavior in one transaction:
>
> (1) BEGIN TRANSACTION
>
> (2) SELECT ... FROM some_table WHERE some_field=some_value
> (1 row returned)
>
> (3) (some other transactions commit)
>
> (4) SELECT ... FROM some_table WHERE some_field=some_value
> (1 row returned, same as above)
>
> (5) DELETE some_table WHERE some_field=some_value
> (answer: 1 row deleted)
>
> (6) SELECT ... FROM some_table WHERE some_field=some_value
> (1 row returned, same as above)
>
> (7) COMMIT
> (the row that was returned earlier is no longer in the 
> database)
>
> Take a minute to read this. Up to step (5), everything is as you would expect;
> you should find steps (6) and (7) quite surprising.
>
> This happens because the other transactions in (3) deleted the row that is
> returned in (2), (4) & (6), and inserted another one where
> some_field=some_value; that other row is the row that was deleted in (5). The
> row that this transaction selects was not seen by the DELETE, and hence not
> changed by it, and hence continues to be visible by the SELECTs in our
> transaction. But when we commit, the row (which has been deleted) no longer
> exists.
>
> I have expressed elsewhere my opinion of this behavior as a general database
> feature, and feel no need to repeat it here; but I think that, if possible, it
> is Django's job as a framework to protect its users from it, at least as a
> default.
>
> On Monday 21 March 2016 02:25:37 Cristiano Coelho wrote:
>> What performance changes can you expect doing this change? It is probably
>> that default on MySQL for a good reason.
>
> The Django project is usually willing to give up quite a lot of performance in
> order to prevent data losses. I agree that this default on MySql is probably
> for a reason, but I don't think it can be a good reason for Django.
>
> Have fun,
> Shai.
>
> [1] https://dev.mysql.com/doc/refman/5.7/en/innodb-consistent-read.html
> [2] https://code.djangoproject.com/ticket/26347

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CALMtK1EOwq7CC2bF6nL7-GuHPv_RJ_cf_giCtchtk2gzYYMr8g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Aymeric Augustin]

On 21 Mar 2016, at 01:25, Cristiano Coelho  wrote:
> 
> What performance changes can you expect doing this change? It is probably 
> that default on MySQL for a good reason.

Barring implementation weirdness, reducing the isolation level is supposed to 
improve performance (if it makes a difference at all).

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/651A7CF7-D982-4584-990F-071DDCF2FFA8%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: relative path in {% extends "...base.html" %} like relative import

[Vitaly Bogomolov]

>
>
> Either way, I fail to see how that feature would be really useful.
>

I was need a refactor for large hierarchy of django templates. Moving 
branches inside hierarchy is simple, if inheritance uses relative paths.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/89b30fb7-4ee6-4a91-b2aa-57977b3214d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Shai Berger]

First of all, I would like to say that I strongly support the move to READ 
COMITTED, including backporting it to 1.8.x.

But we also need to explain: REPEATABLE READ is a higher transaction isolation 
level than READ COMMITTED. If you have problematic code, it should lead to 
more deadlocks and/or transactions failing at commit time (compared to READ 
COMMITTED), not to data loss. The reason we get data losses is MySql's unique 
interpretation of REPEATABLE READ. If you're interested in the details (and if 
you use MySql, you should be), read on.

With MySql's REPEATABLE READ, the "read" operations -- SELECT statements -- 
indeed act like they act in the usual REPEATABLE READ: Once you've read some 
table, changes made to that table by other transactions will not be visible 
within your transaction. But "write" operations -- UPDATE, DELETE, INSERT and 
the like -- act as if they're under READ COMMITTED, affecting (and affected by) 
changes committed by other transactions. The result is, essentially, that 
within a transaction, the reads are not guaranteed to be consistent with the 
writes [1].

In particular, in the bug[2] that caused this discussion, we get the following  
behavior in one transaction:

(1) BEGIN TRANSACTION

(2) SELECT ... FROM some_table WHERE some_field=some_value
(1 row returned)

(3) (some other transactions commit)

(4) SELECT ... FROM some_table WHERE some_field=some_value
(1 row returned, same as above)

(5) DELETE some_table WHERE some_field=some_value
(answer: 1 row deleted)

(6) SELECT ... FROM some_table WHERE some_field=some_value
(1 row returned, same as above)

(7) COMMIT
(the row that was returned earlier is no longer in the database)

Take a minute to read this. Up to step (5), everything is as you would expect; 
you should find steps (6) and (7) quite surprising.

This happens because the other transactions in (3) deleted the row that is 
returned in (2), (4) & (6), and inserted another one where 
some_field=some_value; that other row is the row that was deleted in (5). The 
row that this transaction selects was not seen by the DELETE, and hence not 
changed by it, and hence continues to be visible by the SELECTs in our 
transaction. But when we commit, the row (which has been deleted) no longer 
exists.

I have expressed elsewhere my opinion of this behavior as a general database 
feature, and feel no need to repeat it here; but I think that, if possible, it 
is Django's job as a framework to protect its users from it, at least as a 
default.

On Monday 21 March 2016 02:25:37 Cristiano Coelho wrote:
> What performance changes can you expect doing this change? It is probably
> that default on MySQL for a good reason.

The Django project is usually willing to give up quite a lot of performance in 
order to prevent data losses. I agree that this default on MySql is probably 
for a reason, but I don't think it can be a good reason for Django.

Have fun,
Shai.

[1] https://dev.mysql.com/doc/refman/5.7/en/innodb-consistent-read.html
[2] https://code.djangoproject.com/ticket/26347

Re: relative path in {% extends "...base.html" %} like relative import

[Vitaly Bogomolov]

 

> Hmm... I suppose the closest alternative we have would be to store the 
> base template name in a variable and pass it in from the view.
>

If code for support template inheritance lives in views, it's 
not convenient.

Here is code, that implement this feature for Django 1.4/1.9:
https://github.com/vb64/django.templates.relative.path

But this extension disable the useful rule 'extends must be first tag in 
template', and that make me unhappy. :)
If Django core team approve my proposal, i can make pool request.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/eddbb199-468a-468c-9146-92a17e791e09%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.