s/_connect/_bind/ ;) the proxy binds two ports, the source and agents connect to it
On Jan 15, 2014, at 22:23, [email protected] wrote: > I'm quite sure that it shouldn't be a problem to have the proxy written in C > and the rest in Java, but anyway the proxy doesn't forward any messages > between the Java programms. During my tests on my Mac I allow the program to > accept connections via the firewall, hence there shouldn't be any problems. I > don't understand the issues with this 25 line c program. > http://pastebin.com/4gvjkv4U > > > Gesendet: Mittwoch, 15. Januar 2014 um 12:11 Uhr > Von: "Andreas Bauer" <[email protected]> > An: "ZeroMQ development list" <[email protected]> > Betreff: Re: [zeromq-dev] Pub/sub pattern: Binding two sources to same > address not working > Thanks for sharing your experiences. Very interesting. Then I'll setup a > proxy in C for test purposes and then I'll look into the p2p approach. > > My idea would be now to set up a proxy per machine. The agents on the same > node emit their messages via this proxy, so I can have start and stops agents > within this machine separately and new agents starting up (on the same > machine or a different) can always listen for message by connecting to the > publisher port of the proxy. > > Finally the virtual addressing (http://hintjens.com/blog:39#toc11), which is > discussed in another thread, would also solve my requirement to have a > dynamic yet predictable connection scheme for new agents, right? In terms of > performance which approach would be preferable? > > Gesendet: Mittwoch, 15. Januar 2014 um 01:33 Uhr > Von: "Bruno D. Rodrigues" <[email protected]> > An: "[email protected]" <[email protected]> > Betreff: Re: [zeromq-dev] Pub/sub pattern: Binding two sources to same > address not working > On my local tests I’ve been getting a rate of 1:4 of performance difference > between all C code (libzmq’s perf local_thr/remote_thr/proxy) and all java > code (jeromq) (and mixed results using the JZMQ bindings, mostly because of > JNI itself, not the code). With some tweaking (rather bug fixing for mac) I > was able to double the performance of the C proxy (and 10x’plicate on the > mac), which is code still pending some cleanup on my github fork (davipt). I > guess I should also look at the jeromq code and see if the same trick > applies. Either way all tests I’ve done have put JeroMQ performance above > 1Gbps, which was my aim, so I’ve been concentrating on other stuff. > > In the past when I did the exact same question and tried to design a way to > have multiple proxies to divide the load (or cross the Gbps per machine), > Pieter convinced me to ditch the proxy and move into a peer-to-peer design. > It’s more complicated to get right, specially with dynamic source nodes and > agents nodes, and in your case with the agents talking back to themselves, > but it’s feasible and removes the bottleneck - or shall I say SPOC of the > proxy. Hence why I didn’t look at JeroMQ’s proxy code yet. > > But early optimization sometimes is overkill. And with ZeroMQ the advantage > is that it’s so damn simple to change from one type of connection to another > one - tcp vs ipc vs inproc, bind vs connect, one connection vs a dozen of > them. What I mean is that if tomorrow you want to get rid of the proxy, your > code won’t change, only the socket’s connect()/bind() stuff (ok you’ll need > some code for node discovery, but that’s different). > > About your numbers, always count not only msg/sec, but bytes/bits per second. > 100K msgs/sec, if at 1K each, is already reaching the 1Gb. > > One thing you should do is design the message types to take advantage of the > subscription prefixes of zeromq, so each agent really gets only the messages > then need to receive, instead of receiving them and discarding like I did on > my sample code. That helps a lot! > > On Jan 14, 2014, at 23:29, Andreas Bauer <[email protected]> wrote: > > Many thanks for the code! It works now. > > I'll take care of the message growth. In fact it shouldn't be a problem, > because each agent only accepts a certain type of message and foremost only > emits a single type of message, i.e. the message are not "blindly" forwarded > by all agents. As mentioned the goal was to provide a fixed filtering point > to new agents where they can listen for the messages they need. > > But now the performance questions arise and I ask for your experience with > 0mq as I'm still learning :) > > Can such a proxy keep the processing performance constant or will it be a > bottle neck if I keep adding agents? Can I expect to keep the processing rate > constant if I design my agents to take care of the suicidal snail pattern? A > short test (without having the agents yet optimized): Without any agents the > source can emit 100.000 message/second. With only 3 agents it is already down > to 22.000. So is this an expected behaviour no matter what I do with the > agents, or can I expect to keep the rate constant. > > I'm asking because this gives me a hint if my optimizations are working and > it saves time if it is impossible anyhow. > > Gesendet: Dienstag, 14. Januar 2014 um 12:59 Uhr > Von: "Bruno D. Rodrigues" <[email protected]> > An: "[email protected]" <[email protected]> > Betreff: Re: [zeromq-dev] Pub/sub pattern: Binding two sources to same > address not working > https://gist.github.com/davipt/8417197 > > Launch *one* proxy. > > Launch one or more sources. > > Launch one or more agents. > > Rethink about agents sending messages to other agents, as that way the > messages will exponentially multiply if you’re not careful :P > > > > > > > On Jan 13, 2014, at 22:47, [email protected] wrote: > > Thanks for your advise. But I'm still struggeling. > > To be honest, I still don't succeed in receiving and publishing messages at > the same time using a proxy. I must be doing something wrong. Maybe you could > have short look. Sometimes I'm too blind to see... > http://pastebin.com/ar6bH8vq > > > > Gesendet: Montag, 13. Januar 2014 um 09:57 Uhr > Von: "Bruno D. Rodrigues" <[email protected]> > An: "[email protected]" <[email protected]> > Betreff: Re: [zeromq-dev] Pub/sub pattern: Binding two sources to same > address not working > Thats the part I couldn’t understand. > > But then again, it shouldn’t be that hard: > > 1. with zeromq it’s irrelevant who binds and who connects. There is no > “client connects to servers”. > > 2. one of the services/node must be well known to the others. This is the one > that should do the bind. If there are multiple, then each will have a > different port, if running on the same machine, or different ip > > 3. a zeromq socket can connect to multiple points; a socket can also bind to > multiple points; heck a socket can even bind and connect at the same time. > > 4. from what I understood you’re trying to do some kind of broadcast bus > where every node connects to listen and rebroadcast stuff (hope they take > care of loop messages). For this case it may be worth to have a separate note > doing a XSUB(bin)-XPROXY-XPUB(bind] so you get a well known port (two) and > each agent or other services can connect to it. > > ctx = zmq.context(1) > s1 = ctx.socket(ZMQ.XSUB); s2 = ctx.socket.ZMQ.XPUB); > s1.bind(“tcp://*.5556”); s2.bind(“tcp://5557”); > ZMQ.proxy(s1,s2, null); > > Then each Agent does a SUB socket connecting to 5557 and, to republish > messages, a PUB socket connecting to 5556. > > > > > On Jan 13, 2014, at 8:13, [email protected] wrote: > > Ok, you're right regarding tcp and maybe I phrased it a bit complicated. > > In fact the sources should be fan-out nodes and the agents fan-in/fan-out > nodes. I'm not sure how to correctly implement it so I don't have to use a > different port for every new node or message type. Or is the only correct way > to specify a different port for every fan-out node? > > Gesendet: Sonntag, 12. Januar 2014 um 23:33 Uhr > Von: "Bruno D. Rodrigues" <[email protected]> > An: "ZeroMQ development list" <[email protected]> > Betreff: Re: [zeromq-dev] Pub/sub pattern: Binding two sources to same > address not working > > It was hard for me to understand what are you trying to do but I guess the > simple answer is no, you can't have the same port (ip+port to be correct) > bound multiple times. That's how TCP works. > > > -- > Bruno Rodrigues > Sent from my iPhone > > No dia 12/01/2014, às 21:51, Andreas Bauer <[email protected]> escreveu: > > > Hello, > > I'm fairly new to ZeroMq and that's why I'm asking for a hint regarding the > correct usage of 0mq. I'm using version 3.2.2. with Java and jeromq version > 0.3.2. My scenario is running on a local machine for test purposes. Here is > what I try to accomplish (snippets: http://pastebin.com/YG9adHEy). > > I have several publishers/sources publishing messages via tcp port 5556. > Furthermore I'm trying to implement various "agents" that receive those > messages by listening at port 5556 and in addtion they publish their messages > also via tcp and port 5556. This is because I want to add new agents > dynamically that can receive messages from existing agents as well as already > running agents can accept messages from the new agents without > reconfiguration. I thought they all need a common port, so I can configure > all agents identically. > > Hence I have implementend a source that sends message via tcp and port 5556. > I've also got (at the moment only) one agent that subscribes to the messages > sent from the source. This works great unless I try to publish messages from > the agent via port 5556. What I try to say is, when I start the agent without > the "Agent - Source" part (see snippet) the agent ("Agent -Receiver) receives > the messages. If I start the agent also with publisher part (Agent - > Source.init()), the agent does not receive any messages. > http://pastebin.com/YG9adHEy > > At the moment I'm a bit stuck, because I'm not sure anymore if it is possible > to have several publishers bind to the same port. I thought it is, but as > mentioned, if try to startup a the publisher within the agent, the agent > can't receive any messages. I thought I can use the Espresso pattern, but it > doesn't work (maybe I've implemented it incorrectly if this was the right > approach). > > I need high throughpt therefore I chose the pub/sub pattern. But 0mq offers > so many different message patterns, maybe you can give me a hint if I'm using > the wrong messaging pattern. > > Many thanks, > > Andreas > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
