On Tue, Mar 18, 2014 at 5:16 AM, est <electronix...@gmail.com> wrote:
> > IPython.parallel > > > http://ipython.org/ipython-doc/stable/install/install.html#dependencies-for-ipython-parallel-parallel-computing > > It's based on ZeroMQ(PyZMQ), and the `ssh` command. I don't think that's > lightweigh enough for busy web clusters. > You will need to secure you web cluster computations anyway. SSH may be slower that HTTPS, I agree, but I'd still see the benchmarks. IPython is good for handling long processing tasks. For myriad of tiny code+data workers I'd choose Stackless. Not sure about the web server part. > By QMachine I assume that's > > https://github.com/wilkinson/qmachine > > For web server cluster it's really not a good idea to amplify HTTP > requests. One client request amplifies several other HTTP requests on > server clusters. > Right. Because your workers are not trusted you need to distribute the load and validate results with multiple passes. > What I propose is something like Zed Shawn's Mongrel2 project ( > http://mongrel2.org/), use a very lightweight server-side serialization > protocol as cluster IPC, you can pass states/data between nodes (workers) > easily. It should be agnostic to framework or libraries, the objective is > to unite python modules in the realtime web world. Because for > request-response web world, a synchronized gateway like WSGI is good > enough, between each requests, share > nothing<https://docs.djangoproject.com/en/dev/faq/general/#does-django-scale> > . > > But for realtime web, server side state is very much required. There need > to be a fd pool for DBs, external services, and stuff like Server-Side-Push > technologies. > "realtime web" is a very broad term. Need a more concise definition. I see only one difference in "web" over standard protocol - is that client is limited to send operations only and requests to HTTP(S) protocol only. Is that true? All other parts of the system can communicate with whatever protocols they like. So, to unify the network under some standard, we need common base. Stick to limitations of client to make all nodes work the same. Limit choice to bare minimum and extend where it is needed. Let's assume the following scenario: > > One user submits a blog, his follower gets browser/iOS/Android push > notification. Because users are connected different nodes in one big > cluster, we need some kind of mechanism to broadcast this message. > > In such an architecture we can write simpler code like this: > > from django.db.models.signals import post_save > > @receiver(post_save, sender=BlogPostModel) > def my_handler(sender, **kwargs): > msg = "User X just posted a new blog, check it out at http://..." > browser_followers.send(msg) > ios_followers.send(msg) > android_followers.send(msg) > > Currently this library reall shines. > > https://pypi.python.org/pypi/telegraphy/ > > Telegraphy architecture is like this: > > [image: Inline image 1] > > What I propose is to merge Web-app part and the AutobahnPython Gateway > part into *one* based on a community honored standard. > Just a side note - XML-RPC is a bad way of operation and I am going to promote that belief. The key component here that is not depicted is client limitations (able to only request events, and accept events after websocket connection is established with a single server). Channel description (WS, HTTP) are not informative in this regard to capture that limitation that this architecture should deal with. When client (browser) establishes connection to HTTP site, can it open a websocket to the site in other domain? If no - then cross-domain interaction should also be included into problem description before unifying Django and Autobahn. If this limitation exists - the generic clustering problem will include management DNS infrastructure (to make sure client can send requests to any node in the cluster) or clustering will require frontends on servers to reroute requests on established websocket connections to appropriate cluster nodes. Not sure I got the positioning of NodeJS cluster right, so feel free to fix that.
_______________________________________________ Web-SIG mailing list Web-SIG@python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: https://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com