Re: Share unpickleable object across processes

[dieter]

Israel Brewster  writes:

> I have a Flask/UWSGI web app that serves up web socket connections. When a 
> web socket connection is created, I want to store a reference to said web 
> socket so I can do things like write messages to every connected 
> socket/disconnect various sockets/etc. UWSGI, however, launches multiple 
> child processes which handle incoming connections, so the data structure that 
> stores the socket connections needs to be shared across all said processes. 
> How can I do this?

At low (= Posix) level, socket objects are represented as
ints, representing an "open communication object".

Usually, a child process shares the parent's "open communication object"s
*AT THE TINE OF THE CHILD CREATION*. However, often, this sharing
is broken very quickly: the parent may have set up those objects
to close automatically in the child; the parent may close its
"open communication object" after the spawning of the child.

If two processes have not inherited the same "open communication object"
from a common ancestor, there is no way that they can share
the communication object.


I am quite convinced that UWSGI will
be set up to close "open communication object"s passed on to
child processes. Thus, I see little chance that your current approach
can succeed.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Share unpickleable object across processes

[Chris Angelico]

On Fri, Nov 3, 2017 at 11:00 AM, Dennis Lee Bieber
 wrote:
> On Thu, 2 Nov 2017 12:32:35 -0800, Israel Brewster 
> declaimed the following:
>
>
>>
>>I *must* be thinking about this wrong. Take even a basic chat app that uses 
>>websockets. Client a, which connected to process 1, sends a message to the 
>>server. There are three other clients connected, each of which needs to 
>>receive said message. Given that the way UWSGI works said clients could have 
>>connected to any one of the worker processes, how can the server push the 
>>message out to *all* clients? What am I missing here?
>>
>
> This is beginning to sound like a form of publisher/subscriber system.
>
> http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pubsub.html
>
> Though I have to admit I don't quite see how to set up bidirectional
> communication. Having the workers subscribe to receive data from a single
> publisher is the easier part -- but how a worker can send data to the
> publisher for distribution is not clear; the distribution process can't
> subscribe to every worker automatically... Perhaps something using
>
> http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/devices/forwarder.html
>
> with each worker making both publisher and subscriber connections. That
> could let a worker publish to the forwarder, which then distributes to all
> subscribed workers. Might need some way to identify one's own message
> (perhaps each worker has a topic, and filters its own topic out of the
> return stream).

Websockets are perfect for this sort of thing. It's not difficult
but you need to have all the clients managed by a single central hub,
which is the exact problem the OP faced.

Here's a simple websocket app that does this sort of thing:

https://github.com/Rosuav/monopoly-open-bid/blob/master/server.py

It depends on running a single process to handle every client, and
then simply maintains an in-memory list of concurrent clients. (This
uses asyncio, but equivalent models using threads are also possible.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Share unpickleable object across processes

[Chris Angelico]

On Fri, Nov 3, 2017 at 8:10 AM, Israel Brewster  wrote:
> Gotcha, thanks. The hesitation I have there is that the UWSGI config is a
> user setting. Sure, I can set up my install to only run one process, but
> what if someone else tries to use my code, and they set up UWSGI to run
> multiple? I hate the idea of my code being so fragile that a simple user
> setting change which I have no control over can break it. But it is what it
> is, and if that's the only option, I'll just put a note in the readme to
> NEVER, under any circumstances, set UWSGI to use multiple processes when
> running this app and call it good :-)

IMO that's not a problem. People have gotten into the bad habit of
assuming that all requests are independent, and that's something that
is OFTEN true, but can leak. That's why some web sites randomly log
you out now and then - because you landed on a different instance of
the web server, and it doesn't have your login (because they messed up
the sharing across nodes). For something like this, where different
clients directly interact with each other, it's entirely reasonable to
demand that they all be handled by a single process.

In MUDs, it's normal to run a single process for all clients. Of
course, MUDs aren't heavy-weight, but it's possible to handle a
thousand concurrent clients without a blip on the CPU monitor. You'd
be surprised how many HTTP clients you can manage on a single process,
as long as you have the RAM for it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Share unpickleable object across processes

[Israel Brewster]

On Nov 2, 2017, at 12:36 PM, Chris Angelico  wrote:
> 
> On Fri, Nov 3, 2017 at 7:35 AM, Israel Brewster  > wrote:
>> On Nov 2, 2017, at 12:30 PM, Chris Angelico  wrote:
>>> 
>>> On Fri, Nov 3, 2017 at 5:54 AM, Israel Brewster  
>>> wrote:
 I have a Flask/UWSGI web app that serves up web socket connections. When a 
 web socket connection is created, I want to store a reference to said web 
 socket so I can do things like write messages to every connected 
 socket/disconnect various sockets/etc. UWSGI, however, launches multiple 
 child processes which handle incoming connections, so the data structure 
 that stores the socket connections needs to be shared across all said 
 processes. How can I do this?
 
>>> 
>>> You're basically going to need to have a single process that manages
>>> all the socket connections. Do you actually NEED multiple processes to
>>> do your work? If you can do it with multiple threads in a single
>>> process, you'll be able to share your socket info easily. Otherwise,
>>> you could have one process dedicated to managing the websockets, and
>>> all the others message that process saying "please send this to all
>>> processes".
>> 
>> Ok, that makes sense, but again: it's UWSGI that creates the processes, not 
>> me. I'm not creating *any* processes or threads. Aside from telling UWSGI to 
>> only use a single worker, I have no control over what happens where. But 
>> maybe that's what I need to do?
>> 
> 
> That's exactly what I mean, yeah. UWSGI should be able to be told to
> use threads instead of processes. I don't know it in detail, but a
> cursory look at the docos suggests that it's happy to use either (or
> even both).

Gotcha, thanks. The hesitation I have there is that the UWSGI config is a user 
setting. Sure, I can set up my install to only run one process, but what if 
someone else tries to use my code, and they set up UWSGI to run multiple? I 
hate the idea of my code being so fragile that a simple user setting change 
which I have no control over can break it. But it is what it is, and if that's 
the only option, I'll just put a note in the readme to NEVER, under any 
circumstances, set UWSGI to use multiple processes when running this app and 
call it good :-)

> 
> ChrisA
> -- 
> https://mail.python.org/mailman/listinfo/python-list 
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Share unpickleable object across processes

[Chris Angelico]

On Fri, Nov 3, 2017 at 7:35 AM, Israel Brewster  wrote:
> On Nov 2, 2017, at 12:30 PM, Chris Angelico  wrote:
>>
>> On Fri, Nov 3, 2017 at 5:54 AM, Israel Brewster  
>> wrote:
>>> I have a Flask/UWSGI web app that serves up web socket connections. When a 
>>> web socket connection is created, I want to store a reference to said web 
>>> socket so I can do things like write messages to every connected 
>>> socket/disconnect various sockets/etc. UWSGI, however, launches multiple 
>>> child processes which handle incoming connections, so the data structure 
>>> that stores the socket connections needs to be shared across all said 
>>> processes. How can I do this?
>>>
>>
>> You're basically going to need to have a single process that manages
>> all the socket connections. Do you actually NEED multiple processes to
>> do your work? If you can do it with multiple threads in a single
>> process, you'll be able to share your socket info easily. Otherwise,
>> you could have one process dedicated to managing the websockets, and
>> all the others message that process saying "please send this to all
>> processes".
>
> Ok, that makes sense, but again: it's UWSGI that creates the processes, not 
> me. I'm not creating *any* processes or threads. Aside from telling UWSGI to 
> only use a single worker, I have no control over what happens where. But 
> maybe that's what I need to do?
>

That's exactly what I mean, yeah. UWSGI should be able to be told to
use threads instead of processes. I don't know it in detail, but a
cursory look at the docos suggests that it's happy to use either (or
even both).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Share unpickleable object across processes

[Israel Brewster]

On Nov 2, 2017, at 11:15 AM, Stefan Ram  wrote:
> 
> Israel Brewster  writes:
>> the data structure that stores the socket connections needs
>> to be shared across all said processes.
> 
>  IIRC that's the difference between threads and
>  processes: threads share a common memory.
> 
>  You can use the standard module mmap to share
>  data between processes.
> 
>  If it's not pickleable, but if you can write code
>  to serialize it to a text format yourself, you
>  can share that text representation via, er, sockets.

If I could serialize it to a text format, then I could pickle said text format 
and store it in redis/some other third party store. :-)

> 
>> In C I might do something like store a void pointer to the
>> object, then cast it to the correct object type
> 
>  Restrictions of the OS or MMU even apply to
>  C code.

Sure, I was just talking in general "ideas". I'm not saying I tried it or it 
would work.

> 
>> , but that's not an option in python. So how can I get around
>> this issue?
> 
>  You can always write parts of a CPython program
>  in C, for example, using Cython.

True, but I just need to be able to share this piece of data - I don't want to 
reinvent the wheel just to write an app that uses web sockets!

I *must* be thinking about this wrong. Take even a basic chat app that uses 
websockets. Client a, which connected to process 1, sends a message to the 
server. There are three other clients connected, each of which needs to receive 
said message. Given that the way UWSGI works said clients could have connected 
to any one of the worker processes, how can the server push the message out to 
*all* clients? What am I missing here?

> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Share unpickleable object across processes

[Israel Brewster]

On Nov 2, 2017, at 12:30 PM, Chris Angelico  wrote:
> 
> On Fri, Nov 3, 2017 at 5:54 AM, Israel Brewster  wrote:
>> I have a Flask/UWSGI web app that serves up web socket connections. When a 
>> web socket connection is created, I want to store a reference to said web 
>> socket so I can do things like write messages to every connected 
>> socket/disconnect various sockets/etc. UWSGI, however, launches multiple 
>> child processes which handle incoming connections, so the data structure 
>> that stores the socket connections needs to be shared across all said 
>> processes. How can I do this?
>> 
> 
> You're basically going to need to have a single process that manages
> all the socket connections. Do you actually NEED multiple processes to
> do your work? If you can do it with multiple threads in a single
> process, you'll be able to share your socket info easily. Otherwise,
> you could have one process dedicated to managing the websockets, and
> all the others message that process saying "please send this to all
> processes".

Ok, that makes sense, but again: it's UWSGI that creates the processes, not me. 
I'm not creating *any* processes or threads. Aside from telling UWSGI to only 
use a single worker, I have no control over what happens where. But maybe 
that's what I need to do?

> 
> ChrisA
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Share unpickleable object across processes

[Chris Angelico]

On Fri, Nov 3, 2017 at 5:54 AM, Israel Brewster  wrote:
> I have a Flask/UWSGI web app that serves up web socket connections. When a 
> web socket connection is created, I want to store a reference to said web 
> socket so I can do things like write messages to every connected 
> socket/disconnect various sockets/etc. UWSGI, however, launches multiple 
> child processes which handle incoming connections, so the data structure that 
> stores the socket connections needs to be shared across all said processes. 
> How can I do this?
>

You're basically going to need to have a single process that manages
all the socket connections. Do you actually NEED multiple processes to
do your work? If you can do it with multiple threads in a single
process, you'll be able to share your socket info easily. Otherwise,
you could have one process dedicated to managing the websockets, and
all the others message that process saying "please send this to all
processes".

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Share unpickleable object across processes

[Israel Brewster]

I have a Flask/UWSGI web app that serves up web socket connections. When a web 
socket connection is created, I want to store a reference to said web socket so 
I can do things like write messages to every connected socket/disconnect 
various sockets/etc. UWSGI, however, launches multiple child processes which 
handle incoming connections, so the data structure that stores the socket 
connections needs to be shared across all said processes. How can I do this?

Tried so far:

1) using a multiprocessing Manager(), from which I have gotten a dict(). This 
just gives me "BlockingIOError: [Errno 35] Resource temporarily unavailable" 
errors whenever I try to access the dict object.
2) Using redis/some other third-party store. This fails because it requires you 
to be able to pickle the object, and the web socket objects I'm getting are not 
pickle able.

In C I might do something like store a void pointer to the object, then cast it 
to the correct object type, but that's not an option in python. So how can I 
get around this issue?



---
Israel Brewster
Systems Analyst II
Ravn Alaska
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7293
---




-- 
https://mail.python.org/mailman/listinfo/python-list