Re: Channels, Websockets and 'Backpressure'

[Hank Sims]

>
> You set it in the channel layer configuration in Django, like this:
> https://github.com/django/asgi_redis/#usage
>

Ah, thank you. Sorry I missed that.


> How would you propose this worked? The only alternative to closing the
> socket is to buffer the messages in memory and retry sending them, at which
> point you might have the case where the client thinks they have a working
> connection but it's not actually delivered anything for 30 seconds. Hard
> failure is preferable in distributed systems in my experience; trying to
> solve the problem with soft failure and retry just makes problems even more
> difficult to detect and debug.
>

I guess the "hard failure" I would prefer in this case -- though maybe not
all cases -- is simply discarding new outbound messages when their queue is
full. Or else some sort of mechanism from within my consumers.py that would
allow me to forgo writing to a channel if its queue is full.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2BD3Ovis1T0jDD_d7kwS3ZA1EmGeG5bPqoME7kT6%2BsBdG3aicA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels, Websockets and 'Backpressure'

[Hank Sims]

Thanks, Andrew. A few follow-up questions:

1. How would one go about increasing the default maximum queue size? I saw 
some reference to this when I was researching the problem yesterday, but I 
couldn't find the setting that would change it. 

2. Shouldn't there be a way to resolve the backpressure by draining the 
queue before allowing new messages to be written to it? It seems like 
cutting the connection between client and server would exacerbate the 
problem, rather than remedying it. In my particular case, it wouldn't be 
that big if a block of messages were skipped. But closing the socket when 
maximum queue size is reached seems to cause a cascade of problems.

Thanks for your response, and even more thanks for your work on this 
project.



On Thursday, December 1, 2016 at 12:34:22 PM UTC-8, Andrew Godwin wrote:
>
> "Backpressure" is designed exactly for what you describe, which is when 
> clients are making requests of the server faster than you can handle them. 
> Each channel has a maximum capacity of messages (100 by default), beyond 
> which trying to add a new one results in an error.
>
> Webservers, when they see this, return an error to the client to try and 
> resolve the overload situation. If they didn't, then the server would clog 
> up trying to buffer all the pending requests. It's like returning a 503 
> error on a webpage when a server is overloaded.
>
> To solve the situation, just provision more workers so the channel is 
> drained as fast as messages are put onto it.
>
> If you want to monitor the size of channels to anticipate this stuff, 
> there's a plan for an API in ASGI that would let you do that but it's not 
> in place yet. You may look at the length of the Redis lists directly in the 
> meantime if you wish (there's one list per channel).
>
> Andrew
>
>
>
> On Thu, Dec 1, 2016 at 11:26 AM, hank...@gmail.com  <
> hank...@gmail.com > wrote:
>
>> Can someone help me understand the concept of websocket “backpressure” in 
>> a Django Channels project? What is it? How do diagnose it? At what level of 
>> the stack does it occur? How do I cure it? The docs are a little hazy on 
>> this.
>>
>>
>> I wired up a quick Channels project for my mid-sized website. Before 
>> deploying the project, I load-tested it with thor 
>>  and started scaling up. When I 
>> reached two Daphne processes and four worker processes, it seemed like I 
>> had enough power behind the project to handle the load on my site. It was 
>> able to handle 2000 simultaneous websocket connections without errors, 
>> according to thor. That should have been more than enough.
>>
>>
>> I deployed, and everything went fine for a while. After a bit, though, 
>> the websockets got slow and the server started to drop connections. 
>> Eventually the whole project stalled out. I looked through the Daphne logs 
>> and found a flurry of lines like this:
>>
>>
>> 2016-12-01 14:35:14,513 WARNING WebSocket force closed for 
>>> websocket.send!QbxCqPhvyxVt due to receive backpressure
>>
>>
>> I restarted all the server and worker processes to no effect. I was able 
>> to put the project back online by manually deleting all the “asgi:*” keys 
>> in Redis. But then, after a while, the backpressure built up and everything 
>> crashed again.
>>
>>
>> The problem, I suppose, has something to do with the high frequency of 
>> messages that were to be passed via websocket in this particular project. A 
>> click triggers a message in each direction, and people were encouraged to 
>> click rapidly. So I probably have to throttle this, or else launch more 
>> workers and/or servers.
>>
>>
>> But I'd like to know what, specifically, triggers these “backpressure” 
>> disconnections, and where I might look to monitor them /before/ errors 
>> start to occur. In one of the Redis queues, I suppose? If so, which one(s) 
>> – inbound or outbound? I suppose my idea, here, is that I might be able to 
>> automatically scale up if the queues start to fill up.
>>
>>
>> Thank you in advance. Fun project!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com .
>> To post to this group, send email to django...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/79eb8c4d-0223-4320-8295-c936ebc4a68f%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an 

Re: Figuring out what has changed at save()

[Hank Sims]

For what it's worth, I can imagine on way of doing this. I could use
signals to monkeypatch the instance at post_init, adding an attribute
called "post_init_status." Then I could compare this value with
"status" at post_save.

Please save me from doing something so ugly.

On Mar 6, 12:28 am, Rama Vadakattu  wrote:
> So you need to trigger an action the first time you save an object in
> to the database.
>
> --
>
> Class A:
>
>#self.id is None whenever a particular instance has not been saved
> into a database
>
>#override save method in the object as follows
>
>def save(self) :
>firstime = false
>
>if ~self.id :
>firsttime = true
>
>super.save()
>
>#this code only fires only when it is being saved for the first
> time.
>if firstime :
>   #Trigger an action whichevery you want.
>
> -
>
> Hope the above code helps you in resolving the problem.
>
> On Mar 6, 11:02 am, "hanks...@gmail.com"  wrote:
>
> > Sorry for the unwieldy title, but nothing else strikes me at the
> > moment.
>
> > I have a blog app -- a version of basic.blog, actually. There's a
> > field in the model called "status" with two options: "draft" and
> > "public."
>
> > What I want to do is trigger an action the first time (and /only/ the
> > first time) a particular instance is saved as "public."
>
> > So:
>
> > Write a post in the admin, save as draft --> No action
> > Edit the post some, save as draft again --> No action
> > Edit more, publish   --> Action triggered!
> > Edit again   --> No action.
>
> > How would I go about this? My thought is to override save(), but I
> > can't figure out how to inspect the instance at that point to
> > determine if the status attribute has changed since it was created. I
> > assume that this information must be in there somewhere, since it
> > seems like you'd need it to generate the SQL statement.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: Figuring out what has changed at save()

[Hank Sims]

Rama:

Thanks, but that's no good. That would trigger the action whether or
not the post is saved as "draft."

Hank Sims

On Mar 6, 12:28 am, Rama Vadakattu <ramaakrish...@gmail.com> wrote:
> So you need to trigger an action the first time you save an object in
> to the database.
>
> --
>
> Class A:
>
>#self.id is None whenever a particular instance has not been saved
> into a database
>
>#override save method in the object as follows
>
>def save(self) :
>firstime = false
>
>if ~self.id :
>firsttime = true
>
>super.save()
>
>#this code only fires only when it is being saved for the first
> time.
>if firstime :
>   #Trigger an action whichevery you want.
>
> -
>
> Hope the above code helps you in resolving the problem.
>
> On Mar 6, 11:02 am, "hanks...@gmail.com" <hanks...@gmail.com> wrote:
>
> > Sorry for the unwieldy title, but nothing else strikes me at the
> > moment.
>
> > I have a blog app -- a version of basic.blog, actually. There's a
> > field in the model called "status" with two options: "draft" and
> > "public."
>
> > What I want to do is trigger an action the first time (and /only/ the
> > first time) a particular instance is saved as "public."
>
> > So:
>
> > Write a post in the admin, save as draft --> No action
> > Edit the post some, save as draft again --> No action
> > Edit more, publish   --> Action triggered!
> > Edit again   --> No action.
>
> > How would I go about this? My thought is to override save(), but I
> > can't figure out how to inspect the instance at that point to
> > determine if the status attribute has changed since it was created. I
> > assume that this information must be in there somewhere, since it
> > seems like you'd need it to generate the SQL statement.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

GeoDjango Lookup Woes

[Hank Sims]

 Hello:

 I'm trying to get up on GeoDjango, but I can't get lookups to work at
all. All geographic operations via the GeoManager -- filter, exclude, get --
fail with the same error: "Operations on two geometries with different
SRIDs."
 Strangely, though, I'm perfectly able to perform geometric operations
like "intersects," "touches," "contains," etc., outside the GeoManager. And
I'm pretty sure that the geometries in question don't have different SRIDs
because they come from the same model.
 Here's a typical shell session. The first error listed is an IPython
error -- I don't think that's it, because it fails the same way in the plain
Python shell, but I'll leave it here in case I'm missing something.
 Probably I'm doing something extremely boneheaded.  Any ideas?

 Thanks in advance,

 Hank Sims


In [1]: from gistest.watershed.models import River

In [2]: river0=River.objects.all()[0]

In [3]: river1=River.objects.all()[1]

In [4]: river0.line.touches(river1.line)
Out[4]: True

In [5]: river0.line.srid
Out[5]: 4326

In [6]: River.objects.filter(line__touches=river0.line)
Out[6]:
---
   Traceback (most recent call last)

/home/hanksims/gis_projects/gistest/ in ()

/var/lib/python-support/python2.5/IPython/Prompts.py in __call__(self, arg)
521
522 # and now call a possibly user-defined print mechanism
--> 523 manipulated_val = self.display(arg)
524
525 # user display hooks can change the variable to be
stored in

/var/lib/python-support/python2.5/IPython/Prompts.py in _display(self, arg)
545 """
546
--> 547 return self.shell.hooks.result_display(arg)
548
549 # Assign the default display method:

/var/lib/python-support/python2.5/IPython/hooks.py in __call__(self, *args,
**kw)
132 #print "prio",prio,"cmd",cmd #dbg
133 try:
--> 134 ret = cmd(*args, **kw)
135 return ret
136 except ipapi.TryNext, exc:

/var/lib/python-support/python2.5/IPython/hooks.py in result_display(self,
arg)
160
161 if self.rc.pprint:
--> 162 out = pformat(arg)
163 if '\n' in out:
164 # So that multi-line strings line up with the left
column of

/home/hanksims/gis_projects/gistest/pprint.py in pformat(self, object)
109 def pformat(self, object):
110 sio = _StringIO()
--> 111 self._format(object, sio, 0, 0, {}, 0)
112 return sio.getvalue()
113

/home/hanksims/gis_projects/gistest/pprint.py in _format(self, object,
stream, indent, allowance, context, level)
127 self._readable = False
128 return
--> 129 rep = self._repr(object, context, level - 1)
130 typ = _type(object)
131 sepLines = _len(rep) > (self._width - 1 - indent -
allowance)

/home/hanksims/gis_projects/gistest/pprint.py in _repr(self, object,
context, level)
193 def _repr(self, object, context, level):
194 repr, readable, recursive = self.format(object, context.copy
(),
--> 195 self._depth, level)
196 if not readable:
197 self._readable = False

/home/hanksims/gis_projects/gistest/pprint.py in format(self, object,
context, maxlevels, level)
205 and whether the object represents a recursive construct.
206 """
--> 207 return _safe_repr(object, context, maxlevels, level)
208
209

/home/hanksims/gis_projects/gistest/pprint.py in _safe_repr(object, context,
maxlevels, level)
290 return format % _commajoin(components), readable, recursive
291
--> 292 rep = repr(object)
293 return rep, (rep and not rep.startswith('<')), False
294

/usr/lib/python2.5/site-packages/django/db/models/query.py in __repr__(self)
106
107 def __repr__(self):
--> 108 return repr(self._get_data())
109
110 def __len__(self):

/usr/lib/python2.5/site-packages/django/db/models/query.py in
_get_data(self)
484 def _get_data(self):
485 if self._result_cache is None:
--> 486 self._result_cache = list(self.iterator())
487 return self._result_cache
488

/usr/lib/python2.5/site-packages/django/db/models/query.py in iterator(self)
187
188 cursor = connection.cursor()
--> 189 cursor.execute("SELECT " + (self._distinct and "DISTINCT "
or "") + ",".join(select) + sql, params)
190
191 fill_cache = self._select_related

/usr/lib/python2.5/site-packages/django/db/backends/util.py in execute(self,
sql, params)
 16 start = time(