Very cool Brian. For those in ruby-land, I've done some work on an evented framework merging ZeroMQ, WebSockets, and HTTP I've named DripDrop.
I've got a sample app bridging ZeroMQ and WebSockets up here: http://stringer.andrewvc.com:3002/ It's a live HTTP stats program. Feel free to send some traffic to : http://stringer.andrewvc.com:3002/test_url to see stats show up in the console. you'll see the stats instantly. It's not the most stable thing in the world, (I think em-websocket may leak sockets sometimes, I need to look into what I'm doing that causes that), but it works. To get a feel for how it bridges these protocols, these two short code samples might be useful. In them, I build an HTTP -> XREQ bridge and WebSocket to two PUB/SUB (one on each side of the conn) bridge as well. WebSockets to pub/sub: http://github.com/andrewvc/dripdrop-webstats/blob/master/lib/dripdrop-webstats.rb#L69 HTTP maps to xreq with ease as well: http://github.com/andrewvc/dripdrop-webstats/blob/master/lib/dripdrop-webstats.rb#L60 The API could still use some work, I think I can clean it up quite a bit. If anyone has any opinions on that I'd appreciate it. Under the hood I'm running EventMachine and Chuck Remes' excellent zmqmachine in parallel. I've tried to normalize the APIs for HTTP, WebSockets, and ZeroMQ as much as possible, and provide a common serialization format (a specially structured BERT or JSON message) between them as well. Currently, I'm writing a javascript library to make hooking into it from the browser as simple as possible, the demo above has some extremely ugly and hackish JS. If you want to check out the code, see: http://github.com/andrewvc/dripdrop and http://github.com/andrewvc/dripdrop-webstats On Wed, Sep 29, 2010 at 9:12 PM, <[email protected]>wrote: > Send zeromq-dev mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of zeromq-dev digest..." > > > Today's Topics: > > 1. Eventlet, zeromq and websockets (Brian Granger) > 2. Re: Assertion failed: fetched (rep.cpp:232) with XREQ + REP > (Brian Granger) > 3. Re: [Request] Custom load balancer/routers (Brian Granger) > 4. Re: pyzmq-static 2.0.8 (Brian Granger) > 5. Re: pyzmq-static 2.0.8 (Brian Granger) > 6. Re: pyzmq-static 2.0.8 (Robert Kern) > 7. Re: IOCP (Steven McCoy) > 8. Re: Upgrade to OpenPGM 5.x (commit 00cd7d49c7f2b532b234) > (Steven McCoy) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 29 Sep 2010 15:17:56 -0700 > From: Brian Granger <[email protected]> > Subject: [zeromq-dev] Eventlet, zeromq and websockets > To: zeromq-dev <[email protected]>, Ben Ford > <[email protected]> > Message-ID: > > <[email protected]<aanlktiku0dzwgrl5puwbqtgbaudea8bx%[email protected]> > > > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, > > I just got a note about this from Ben Ford one of the developers of > Eventlet. Eventlet is a coroutine-base networking framework for > Python that has recently added zeromq support (through pyzmq). Ben > just pointed me to this example of a zeromq<->websocket bridge for a > distributed chat system. I thought it was very nice and though others > might be interested. A websocket to zeromq bridge could be a very > powerful way of exposing zeromq goodness to the WAN/internet. > > https://bitbucket.org/boothead/eventlet/changeset/de4b186ab9b4 > > Cheers, > > Brian > > > -- > Brian E. Granger, Ph.D. > Assistant Professor of Physics > Cal Poly State University, San Luis Obispo > [email protected] > [email protected] > > > ------------------------------ > > Message: 2 > Date: Wed, 29 Sep 2010 15:38:27 -0700 > From: Brian Granger <[email protected]> > Subject: Re: [zeromq-dev] Assertion failed: fetched (rep.cpp:232) with > XREQ + REP > To: Martin Sustrik <[email protected]> > Cc: ZeroMQ development list <[email protected]> > Message-ID: > <[email protected]> > Content-Type: text/plain; charset=ISO-8859-1 > > On Tue, Sep 28, 2010 at 11:37 PM, Martin Sustrik <[email protected]> > wrote: > > On 09/28/2010 11:17 PM, Brian Granger wrote: > >> > >> Thanks everyone for pointing our my mistake here. ?But I then have a > >> secondary question. ?Currently it is possible to crash a REP socket by > >> not adhering to the REQ/REP protocol. ?Don't we want to make it more > >> difficult to crash a REP server than this? > > > > Definitely. I've introduced the logging mechanism (sys://log) to 2.1 so > any > > violation of the protocol can be handled by silently discarding the > > offending message and logging the fact, so that admin can take > appropriate > > measures. > > Great! > > Brian > > > Martin > > > > > > > > -- > Brian E. Granger, Ph.D. > Assistant Professor of Physics > Cal Poly State University, San Luis Obispo > [email protected] > [email protected] > > > ------------------------------ > > Message: 3 > Date: Wed, 29 Sep 2010 15:39:45 -0700 > From: Brian Granger <[email protected]> > Subject: Re: [zeromq-dev] [Request] Custom load balancer/routers > To: ZeroMQ development list <[email protected]> > Cc: Thijs Terlouw <[email protected]> > Message-ID: > > <[email protected]<aanlktikbzw0l7uz0mwo97fsuj_nsuseqm%[email protected]> > > > Content-Type: text/plain; charset=ISO-8859-1 > > On Wed, Sep 29, 2010 at 1:14 PM, MinRK <[email protected]> wrote: > > > > > > On Wed, Sep 29, 2010 at 07:26, Martin Sustrik <[email protected]> > wrote: > >> > >> Thijs, > >> > >> > Would it be possible to make the load-balancer for the 2.1 branch of > >> > ZeroMQ more easily customizable? > >> > > >> > I see mentions to lb_t inside push and xreq but the lb_t class doesn't > >> > appear to be extensible because the lack of virtual functions? It > would > >> > be nice if I could plug in my own custom load balancer based on for > >> > example weights, latency (coupled with more details about the internal > >> > status of sockets possibly) etc. > >> > >> What's the actual use case for this? I recall I was discussing this with > >> someone, where it actually turned out that if high watermark that would > >> account for _all_ the messages on the fly (as opposed to messages > >> buffered in the local peer) was implemented there would be no need for > >> custom load-balancing. > > > > I'm not sure about others, but in IPython the load-balancing metric is > not > > obvious to zmq itself, since it need not correlate directly to network > > traffic - we are balancing the CPU load of the jobs contained in the > > messages. > > We do have the ability to generate a structure corresponding to an > estimated > > load of each possible target, and it would be very helpful to use that in > > routing decisions. > > I don't know how zmq organizes the potential destinations, but assuming > > there was a dict/hashtable/array of weights, then a few useful schemes > would > > be: > > * round robin ignoring weights > > * random ignoring weights > > * weighted random pick > > * always pick the highest weight > > There are also variations on how to randomly select when you have > weights. > > ?For instance, the following scheme has been shown to be substantially > > better than a pure random choice:?select two at random instead of one; > use > > the one with greater weight. > > Now, the default behavior for getting weights in zmq could either be: > > messages waiting to send, total messages sent, or (best match for our > case) > > total messages sent minus total messages received. > > There are many ways to load-balance, and the best answer can vary widely > > based on the situation, so if the selection process can be made > extensible > > without performance penalty in the default mode, I think it would be > quite > > valuable - certainly to us in IPython, and likely to others as well. > > Nice summary Min! > > Brian > > > > -MinRK > > > >> > >> Martin > >> _______________________________________________ > >> zeromq-dev mailing list > >> [email protected] > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > -- > Brian E. Granger, Ph.D. > Assistant Professor of Physics > Cal Poly State University, San Luis Obispo > [email protected] > [email protected] > > > ------------------------------ > > Message: 4 > Date: Wed, 29 Sep 2010 15:47:23 -0700 > From: Brian Granger <[email protected]> > Subject: Re: [zeromq-dev] pyzmq-static 2.0.8 > To: ZeroMQ development list <[email protected]> > Message-ID: > <[email protected]> > Content-Type: text/plain; charset=ISO-8859-1 > > Yes, I saw that about a week ago. Unfortunately it looks like they > actually pull the entire zeromq code base into the python build system > (distutils), which is not very ideal. In the long run, I need to > figure out a way of doing the static linking given the libzmq.a. It > should not be too difficult to do that. > > Cheers, > > Brian > > On Wed, Sep 29, 2010 at 11:06 AM, Martin Sustrik <[email protected]> > wrote: > > I am not a Python developer but this looks like it may be handy: > > > > http://pypi.python.org/pypi/pyzmq-static/2.0.8 > > > > Martin > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > -- > Brian E. Granger, Ph.D. > Assistant Professor of Physics > Cal Poly State University, San Luis Obispo > [email protected] > [email protected] > > > ------------------------------ > > Message: 5 > Date: Wed, 29 Sep 2010 15:51:56 -0700 > From: Brian Granger <[email protected]> > Subject: Re: [zeromq-dev] pyzmq-static 2.0.8 > To: ZeroMQ development list <[email protected]> > Message-ID: > <[email protected]> > Content-Type: text/plain; charset=ISO-8859-1 > > Well, actually I just remembered that we are making some changes to > the pyzmq codebase that will, essentially make static linking > difficult or even impossible....We are splitting up the main _zmq.pyx > file into separate files for each main class (Context, Socket, > Message, etc.). Each of these files will have its own .c and .so > files. Static linking libzmq.a into all of those .so files > individually would almost impossible. > > Cheers, > > Brian > > On Wed, Sep 29, 2010 at 3:47 PM, Brian Granger <[email protected]> > wrote: > > Yes, I saw that about a week ago. ?Unfortunately it looks like they > > actually pull the entire zeromq code base into the python build system > > (distutils), which is not very ideal. ?In the long run, I need to > > figure out a way of doing the static linking given the libzmq.a. ?It > > should not be too difficult to do that. > > > > Cheers, > > > > Brian > > > > On Wed, Sep 29, 2010 at 11:06 AM, Martin Sustrik <[email protected]> > wrote: > >> I am not a Python developer but this looks like it may be handy: > >> > >> http://pypi.python.org/pypi/pyzmq-static/2.0.8 > >> > >> Martin > >> _______________________________________________ > >> zeromq-dev mailing list > >> [email protected] > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >> > > > > > > > > -- > > Brian E. Granger, Ph.D. > > Assistant Professor of Physics > > Cal Poly State University, San Luis Obispo > > [email protected] > > [email protected] > > > > > > -- > Brian E. Granger, Ph.D. > Assistant Professor of Physics > Cal Poly State University, San Luis Obispo > [email protected] > [email protected] > > > ------------------------------ > > Message: 6 > Date: Wed, 29 Sep 2010 18:04:42 -0500 > From: Robert Kern <[email protected]> > Subject: Re: [zeromq-dev] pyzmq-static 2.0.8 > To: [email protected] > Message-ID: <[email protected]> > Content-Type: text/plain; charset=UTF-8; format=flowed > > On 9/29/10 5:47 PM, Brian Granger wrote: > > Yes, I saw that about a week ago. Unfortunately it looks like they > > actually pull the entire zeromq code base into the python build system > > (distutils), which is not very ideal. In the long run, I need to > > figure out a way of doing the static linking given the libzmq.a. It > > should not be too difficult to do that. > > The main point of the package, near as I can tell, is to allow a "pip > install > pyzmq-static" to work without needing to have built and installed a > separate > libzmq at all. The static linking is just an implementation detail, despite > the > name. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." > -- Umberto Eco > > > > ------------------------------ > > Message: 7 > Date: Thu, 30 Sep 2010 10:17:59 +0800 > From: Steven McCoy <[email protected]> > Subject: Re: [zeromq-dev] IOCP > To: ZeroMQ development list <[email protected]> > Cc: Mikael Helbo Kj?r <[email protected]> > Message-ID: > <[email protected]> > Content-Type: text/plain; charset="iso-8859-1" > > 2010/9/27 Martin Sustrik <[email protected]> > > > Ack. Using named pipes requires some polling mechanism that works on > > named pipe HANDLE first though. > > > > > I found an amusing list of IPC methods on Windows from MSDN, > > Clipboard< > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_the_clipboard_for_ipc > > > COM< > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_com_for_ipc > > > Data Copy< > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_data_copy_for_ipc > > > DDE< > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_dde_for_ipc > > > File Mapping< > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_a_file_mapping_for_ipc > > > Mailslots< > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_a_mailslot_for_ipc > > > Pipes< > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_pipes_for_ipc > > > RPC< > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_rpc_for_ipc > > > Windows Sockets< > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx#base.using_windows_sockets_for_ipc > > > > http://msdn.microsoft.com/en-us/library/aa365574(VS.85).aspx > > It would probably be worth investigating shared memory (File Mapping) or > named pipes for content and a Windows Socket for synchronisation. > > -- > Steve-o > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100930/a2666220/attachment-0001.htm > > ------------------------------ > > Message: 8 > Date: Thu, 30 Sep 2010 12:12:20 +0800 > From: Steven McCoy <[email protected]> > Subject: Re: [zeromq-dev] Upgrade to OpenPGM 5.x (commit > 00cd7d49c7f2b532b234) > To: ZeroMQ development list <[email protected]> > Message-ID: > <[email protected]> > Content-Type: text/plain; charset="iso-8859-1" > > On 29 September 2010 00:43, Martin Lucina <[email protected]> wrote: > > > In the changes to libpgm_diff_flags in src/Makefile.am, I see things like > > -DCONFIG_HAVE_EPOLL, -DCONFIG_HAVE_HPET. Won't this cause a problem on > > SPARC? Also, I'd prefer that these flags be computed by 'configure' > rather > > than being hardcoded in Makefile.am; I can make this change if you're OK > > with that approach... > > > > > I had a look at Autoconf, some are easy some are definitely not: > > Format of list, > > <OpenPGM CFLAGS> > <Autoconf lines> > <Extra notes> > > -DCONFIG_HAVE_GETPROTOBYNAME_R > -DCONFIG_HAVE_GETPROTOBYNAME_R2 > AC_CHECK_FUNC([getprotobyname_r],[AC_DEFINE(HAVE_GETPROTOBYNAME_R)]) > Need to check for 4 or 5 parameters, Googling shows output of Autoconf > detecting the number of parameters but I didn't find any matching code on > quick search. > > -DCONFIG_HAVE_ISO_VARARGS > > Need to check for pre-processor or compilation errors on the following code > void vararg_func (...); > #define vararg_macro(...) vararg_func (__VA_ARGS__) > vararg_macro (1, 2, 3); > vararg_macro (1, 2); > > -DCONFIG_HAVE_GNUC_VARARGS > > Need to check for pre-processor or compilation errors on the following code > void vararg_func (...); > #define vararg_macro(f...) vararg_func (f) > vararg_macro (1, 2, 3); > vararg_macro (1, 2); > > -DCONFIG_HAVE_ALLOCA_H > AC_FUNC_ALLOCA > > The Autoconf recommendation is a rather long piece of code to copy & paste, > I'm only covering a minor set of platforms. > > -DCONFIG_HAVE_PROC > AC_CHECK_FILE(/proc/cpuinfo,[AC_DEFINE(HAVE_PROC_CPUINFO)]) > > Historically had wider scope, but current release is only the > /proc/cpuinfofile. > > -DCONFIG_HAVE_BACKTRACE > AC_CHECK_FUNC([backtrace],[AC_DEFINE(HAVE_BACKTRACE)]) > > Should be GCC only, Solaris has printstack(3C) for later investigation. > > -DCONFIG_HAVE_PSELECT > AC_CHECK_FUNC([pselect],[AC_DEFINE(HAVE_PSELECT)]) > > -DCONFIG_HAVE_RTC > AC_CHECK_FILE(/dev/rtc,[AC_DEFINE(HAVE_RTC)]) > > -DCONFIG_HAVE_TSC > Check IA32/x86-64 architecture. > > -DCONFIG_HAVE_HPET > AC_CHECK_FILE(/dev/hpet,[AC_DEFINE(HAVE_HPET)]) > > -DCONFIG_HAVE_POLL > ax_have_poll > > -DCONFIG_HAVE_EPOLL > ax_have_epoll > > -DCONFIG_HAVE_GETIFADDRS > AC_CHECK_FUNC([getifaddrs],[AC_DEFINE(HAVE_GETIFADDRS)]) > > -DCONFIG_HAVE_IFR_NETMASK > AC_CHECK_MEMBER([struct ifreq.ifr_netmask], > [AC_DEFINE([HAVE_IFR_NETMASK], [1], [Have ifr_netmask])], > [], > [ > #include <sys/types.h> > #include <sys/socket.h> > #include <net/if.h> > ]) > > Platform may use a #define to union member ifr_ifru.ifru_netmask > > -DCONFIG_HAVE_WSACMSGHDR > AC_CHECK_MEMBER([WSACMSGHDR.cmsg_len], > [AC_DEFINE([HAVE_WSACMSGHDR], [1], [Have WSACMSGHDR])], > [], > [ > #include <ws2tcpip.h> > #include <winsock2.h> > ]) > > -DCONFIG_HAVE_MCAST_JOIN > Check for MCAST_JOIN_GROUP and struct group_req, with disable override for > Windows XP as compiler will present API for Vista+ support. > > -DCONFIG_HAVE_IP_MREQN > AC_CHECK_MEMBER([struct ip_mreqn.imr_ifindex], > [AC_DEFINE([HAVE_IP_MREQN], [1], [Have struct ip_mreqn])], > [], > [ > #include <sys/socket.h> > #include <netinet/in.h> > ]) > > May require different headers for older platforms. > > -DCONFIG_HAVE_SPRINTF_GROUPING > ax_c_printf_thsep > > -DCONFIG_HAVE_VASPRINTF > AC_CHECK_FUNC([vasprintf],[AC_DEFINE(HAVE_VASPRINTF)]) > > -DCONFIG_HAVE_DSO_VISIBILITY > > Check compilation and linking of static lib in dynamic lib with either > __attribute__((visibility("hidden"))) > or > __hidden > > -DCONFIG_BIND_INADDR_ANY > Possibly check getaddrinfo(AI_PASSIVE), or run test build and run time > check > of UDP unicast & multicast send & recv. > > -DCONFIG_HOST_ORDER_IP_LEN > -DCONFIG_HOST_ORDER_IP_OFF > Only affects FreeBSD as far as I can tell, maybe other BSD versions too. > > -DCONFIG_TARGET_WINE > Manual configuration or build & runtime detection of limitations as it is a > moving target: > > 1. No WSARecvMsg(). > 2. GetAdapterAddresses() does not work as expected. > 3. No support of IP_PKTINFO / IP_RECVDSTADDR. > 4. No support of IP_MULTICAST_TTL / IPV6_MULTICAST_HOPS. > > > -DCONFIG_HAVE_GETOPT > AC_CHECK_FUNC([getopt],[AC_DEFINE(HAVE_GETOPT)]) > > -- > Steve-o > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100930/2402dc5c/attachment.htm > > ------------------------------ > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > End of zeromq-dev Digest, Vol 33, Issue 117 > ******************************************* > -- Andrew Cholakian http://www.andrewvc.com
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
