On Mon, 2011-11-14 at 04:07 -0800, Tobias Oberstein wrote: > I am testing a WebSocket based pubsub system .. I have 2 questions .. any > hints welcome! > > Environment: > + FreeBSD 8.2 i386 > + Python 2.7.2 32-bit > + Twisted Trunk > + new kqueue() reactor > > The FreeBSD is tuned for massive connection numbers. I can connect 50k WS > clients. Fine. > > > 1) Massive dispatching > > Essentially, what I'm currently doing is: > > recvs = set([100k instances of protocol.Protocol]) > > data = "..." > for recv in recvs: > recv.transport.write(data) > > Now, writing mass-data to _one_ transport should be done using > producer/consumer. > > But in my case, the data itself is tiny (<100 octets) and the same for all > clients, but the number > of transports to dispatch that data to is massive. > > The problem is: while above loop is running, other stuff is being delayed. > > What would be the right approach to solve that?
Besides callLater, a higher level construct to do this is twisted.internet.task.cooperate(). https://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2010-cooperative-multitasking-with-twisted-getting-things-done-concurrently-11-3352182 is a presentation dreid did, which should really be turned into a howto. > 2) Too many files. > Is there another limit specifically for files, and/or something tunable in > Python/Twisted? That's definitely not a limit on Python or Twisted. That's an OS limit. Are you opening lots of files in addition to your sockets? _______________________________________________ Twisted-Python mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
