Re: [zeromq-dev] Apparent queuing on PUB/SUB fan-in despite HWM=1
Hi Brian, This is more like it! :) I'll have to translate that example into C to explore it further. That'll happen on Thursday or Friday only. Can you log an issue (we should have a new issue tracker running on http://zeromq.jira.com now) and attach your example? Cheers Pieter On Sat, May 14, 2011 at 11:04 PM, Brian Rossa wrote: > Ok, this is far more minimal than my first try but the outcome is the same. > Please see http://pastebin.com/KcAzxM3U. > I rebuilt ZeroMQ from source on Thursday. I am testing on a 3Ghz, 6-core, > 64-bit machine with 16GB of RAM. When I run the script the latency values > (numbers printed to stdout) continue to rise indefinitely. > Hope I haven't missed something. Please let me know what you find. > Cheers! > ~br > >> Date: Sat, 14 May 2011 10:37:40 +0200 >> From: sust...@250bpm.com >> To: zeromq-dev@lists.zeromq.org >> CC: rossa...@hotmail.com >> Subject: Re: [zeromq-dev] Apparent queuing on PUB/SUB fan-in despite HWM=1 >> >> On 05/13/2011 08:02 PM, Brian Rossa wrote: >> > The Question: "Is ZeroMQ suitable for use in real-time and just-in-time >> > systems?" >> >> The answer is: "Yes. It should be." >> >> If it is not, it's a bug. Please report the issue and provide minimal >> test case. Etc. >> >> Martin >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Proposal for new features in ZMQ.
On Sun, May 15, 2011 at 1:25 AM, Fabien Ninoles wrote: > Here the feature set: > Feature 1 - Add a new msg LABEL flag marking a frame as "labeled". Part of redesigning the whole message envelope concept, I think. A good idea. > Feature 2 - Embedded flags inside the msg_t structure. Definitely a good idea, it's hard to justify MORE being a socket property. In czmq I experimented with this being a message property, it works fine. > Feature 3 - Add a timeout sockopt. Nice idea. > Feature 4 - Add a ready sockopt. It'd be impossible IMO to cleanly implement the ready signalling we currently do in the LRU pattern because it's tied into application processing of data. You could do some tighter coordination between the queues at both sides, with low HWM, and this gives you some kind of 'ready'. But it won't be on a message-per-message basis afaics. > Feature 5 - The COLLECTOR socket. Assumes READY can work... > Most of the current existing pattern would be greatly simplified using > them, avoiding weird usage like DEALER socket as worker end point in > LRU, and probably unseen and interesting new patterns could probably > build using them. Note that DEALER allows asynchronous worker 'end points', necessary if your 'end point' actually deals work out to single threaded worker threads. In synchronous workers, we don't use DEALER sockets. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Windows and IOCP
On 14 May 2011 18:58, Martin Sustrik wrote: > Hi Jarred, > > > I've been hacking on it a bit lately. There's a fundamental difference > > between overlapped I/O with IOCP and the typical Unix file descriptor > > polling; IOCP events fire on operation completion and not socket > > readiness. We'd issue overlapped send/recvs (aka asynchronous) and poll > > for "completions" in the event loop. The whole socket ready paradigm > > doesn't matter in this case, and the ZMQ codebase is built on the > > assumption of socket readiness. Throwing a completely different paradigm > > into the mix will require a refactor. > > > > There is a new library that the node.js guys are working on called libuv > > that has a nice network abstraction layer on top of the high performance > > I/O multiplexers https://github.com/joyent/libuv. I'm trying to get some > > inspiration from them, or perhaps give their library a shot. > > > > We basically need to rework the ZMQ I/O and polling infrastructure to > > support asynchronous operations. > > The obvious question is: How can you limit amount of data queued for > sending? If you are sending faster than network can transfer, is IOCP > going to ultimately run out of memory? If you're doing it right, it is all zero copy so all the buffers should be in ZeroMQ or the application. -- Steve-o ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Proposal for new features in ZMQ.
I just terminated most of my porting of the P2P layer of our cluster to ZMQ, (ab-)using a lot the collector device I have presented last week (see http://www.tzone.org/~fabien/hg/index.cgi/collector/). I find out that it shared a lot with two other pattern used in ZMQ, the PUB socket and the LRU device, and so there is probably some stuff to share to make them more easy to implement both, which can probably be useful for other usages. Here the feature set: Feature 1 - Add a new msg LABEL flag marking a frame as "labeled". This flag would allow to distinguish between frames from the original sender and frame added by REQ and ROUTER sockets. This will greatly ease the usage of any socket that connect to a ROUTER socket, including the REQ socket itself. Feature 2 - Embedded flags inside the msg_t structure. By allowing frame flags to be set inside the msg_t structure, it would be far easier to keep forward compatibility with new flags for proxy device that need to copy msg frames in a transparent manner. Just as an example, a queue socket would simply be a while True: out.send(in.recv()) loop, which will now take care of the multipart flags without problem. Feature 3 - Add a timeout sockopt. The timeout would allow the recv method to return with a ETIMEOUT error after a specified amount of time inactivity. It can also be queried by device to implement a similar mechanism inside the device polling scheme. May be the socket could also reset itself, just like if the socket were close and reopen again (removing the need for the user to do it itself). Feature 4 - Add a ready sockopt. This sockopt allow a socket to be marked as ready to read a new message. It will be send on any messages send by the socket as well as the connection handshake time, and will be reset on the next successful recv of the socket (this information does not need to be propagated however). On the other end, sockets could be setup (another sockopt) to only send to socket which were marked as ready (either in a lb or dist manner). Just having this option will make the LRU device a simple queue with a special socket option. You'll be able to use REP device on the worker just as usual and could probably even handled disconnection of the standby workers more efficiently. Feature 5 - The COLLECTOR socket. The COLLECTOR socket is related to the PATCH socket just like the REQ socket is related to XREQ. COLLECTOR socket would allow reception only on sockets that it has send a message previously, and only until it received a READY signal from this socket. All other messages would be dropped, although any READY signal will push back the socket in the eligible list for the next send request. All those features can be emulated in "client" space, but they are clearly less efficient than doing it directly in the library itself. Most of the current existing pattern would be greatly simplified using them, avoiding weird usage like DEALER socket as worker end point in LRU, and probably unseen and interesting new patterns could probably build using them. I'm volunteer to work on all of them, I would just like to have your agreement on it before starting. I'm still a very new to ZMQ and even to network programming, so I could have miss a lot of things in this proposal. Thanks, Fabien ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Trademark policy for 0MQ
On 05/14/2011 08:59 PM, Pieter Hintjens wrote: >> (This tie to github is a bit silly. >> What if it has the repo in svn, hg or somesuch?) > > No answer. We need some kind of formalizable container. Github works > as well as any. Yep, it's OK for now. We can list the projects on a wiki page later on in case there's a request to add a project that's not hosted on github. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Windows and IOCP
Hi Jarred, > I've been hacking on it a bit lately. There's a fundamental difference > between overlapped I/O with IOCP and the typical Unix file descriptor > polling; IOCP events fire on operation completion and not socket > readiness. We'd issue overlapped send/recvs (aka asynchronous) and poll > for "completions" in the event loop. The whole socket ready paradigm > doesn't matter in this case, and the ZMQ codebase is built on the > assumption of socket readiness. Throwing a completely different paradigm > into the mix will require a refactor. > > There is a new library that the node.js guys are working on called libuv > that has a nice network abstraction layer on top of the high performance > I/O multiplexers https://github.com/joyent/libuv. I'm trying to get some > inspiration from them, or perhaps give their library a shot. > > We basically need to rework the ZMQ I/O and polling infrastructure to > support asynchronous operations. The obvious question is: How can you limit amount of data queued for sending? If you are sending faster than network can transfer, is IOCP going to ultimately run out of memory? Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Apparent queuing on PUB/SUB fan-in despite HWM=1
Ok, this is far more minimal than my first try but the outcome is the same. Please see http://pastebin.com/KcAzxM3U. I rebuilt ZeroMQ from source on Thursday. I am testing on a 3Ghz, 6-core, 64-bit machine with 16GB of RAM. When I run the script the latency values (numbers printed to stdout) continue to rise indefinitely. Hope I haven't missed something. Please let me know what you find. Cheers!~br > Date: Sat, 14 May 2011 10:37:40 +0200 > From: sust...@250bpm.com > To: zeromq-dev@lists.zeromq.org > CC: rossa...@hotmail.com > Subject: Re: [zeromq-dev] Apparent queuing on PUB/SUB fan-in despite HWM=1 > > On 05/13/2011 08:02 PM, Brian Rossa wrote: > > The Question: "Is ZeroMQ suitable for use in real-time and just-in-time > > systems?" > > The answer is: "Yes. It should be." > > If it is not, it's a bug. Please report the issue and provide minimal > test case. Etc. > > Martin > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [ANNOUNCE] pyzmq 2.1.7
On Sat, May 14, 2011 at 13:26, Dirkjan Ochtman wrote: > On Sat, May 14, 2011 at 22:00, MinRK wrote: >> I agree, the message may be a bit harsh, since it should work fine on >> everything back to 2.1.0 (though it isn't actively tested all the way >> back). >> >> How about: >> "Detected ZMQ version: , but pyzmq is based on zmq ." >> "libzmq features and fixes introduced after will be unavailable." > > Looks great! I think "based on" is a little bit weird in this context, > maybe we could find some other wording for that? 'targets' is probably more conventional. Thanks! > > Cheers, > > Dirkjan > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [ANNOUNCE] pyzmq 2.1.7
On Sat, May 14, 2011 at 22:00, MinRK wrote: > I agree, the message may be a bit harsh, since it should work fine on > everything back to 2.1.0 (though it isn't actively tested all the way > back). > > How about: > "Detected ZMQ version: , but pyzmq is based on zmq ." > "libzmq features and fixes introduced after will be unavailable." Looks great! I think "based on" is a little bit weird in this context, maybe we could find some other wording for that? Cheers, Dirkjan ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [ANNOUNCE] pyzmq 2.1.7
I agree, the message may be a bit harsh, since it should work fine on everything back to 2.1.0 (though it isn't actively tested all the way back). How about: "Detected ZMQ version: , but pyzmq is based on zmq ." "libzmq features and fixes introduced after will be unavailable." -MinRK On Sat, May 14, 2011 at 12:40, Dirkjan Ochtman wrote: > On Sat, May 14, 2011 at 20:41, Ian Barber wrote: >> Building against 2.1.6 sounds like a bad idea - there's a pretty big bug in >> it, unless you've pulled patches separately? > > Yeah, the 2.1.6 we're distributing is patched up for #206. > > Cheers, > > Dirkjan > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ File Server
Excellent, this is the info I need, now I should be able to figure it out. Thanks! I will probably have more questions, I will post it once I get it figured out. My plan is to make a simple api to embed a simple zmq file server into a python app. Then use it in salt. On May 14, 2011 1:38 PM, "Pieter Hintjens" wrote: > On Sat, May 14, 2011 at 5:12 PM, Thomas S Hatch wrote: > >> Great! This is what I was hoping to hear! If the entire file is getting read >> into ram anyway, how would I go about making a file server that would send >> the files in chunks and not read the whole thing into ram? I think that you >> can see me intention here, but I gues I am still not entirely sure how to do >> it yet. > > So here's one design. You read/write files in chunks of say 1MB, each > a 0MQ message. Your protocol to send a file is: > > * OPEN filename - requests to send a new file, reply gives a staging > handle and the size of the file already received, if GT 0, > * STAGE message - sends a chunk to a staging handle. > * DELIVER handle - closes the file staging, publishes the file > > I'd use a router/dealer socket pair so that I can do the open and > deliver commands synchronously, and the stage command asynchronously. > > This gives you a good level of performance and lets you send massive > files that don't fit in memory, with recovery if there's a network > issue or node crash. > > If you're sending 10K config files, of course, this is overkill, just > slurp and send the file in one message. > > -Pieter > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [ANNOUNCE] pyzmq 2.1.7
On Sat, May 14, 2011 at 20:41, Ian Barber wrote: > Building against 2.1.6 sounds like a bad idea - there's a pretty big bug in > it, unless you've pulled patches separately? Yeah, the 2.1.6 we're distributing is patched up for #206. Cheers, Dirkjan ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ File Server
On Sat, May 14, 2011 at 5:12 PM, Thomas S Hatch wrote: > Great! This is what I was hoping to hear! If the entire file is getting read > into ram anyway, how would I go about making a file server that would send > the files in chunks and not read the whole thing into ram? I think that you > can see me intention here, but I gues I am still not entirely sure how to do > it yet. So here's one design. You read/write files in chunks of say 1MB, each a 0MQ message. Your protocol to send a file is: * OPEN filename - requests to send a new file, reply gives a staging handle and the size of the file already received, if GT 0, * STAGE message - sends a chunk to a staging handle. * DELIVER handle - closes the file staging, publishes the file I'd use a router/dealer socket pair so that I can do the open and deliver commands synchronously, and the stage command asynchronously. This gives you a good level of performance and lets you send massive files that don't fit in memory, with recovery if there's a network issue or node crash. If you're sending 10K config files, of course, this is overkill, just slurp and send the file in one message. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Trademark policy for 0MQ
On Sat, May 14, 2011 at 11:49 AM, Martin Sustrik wrote: > There are projects out there that are not covered by trademark grant (as > they are not part of zeromq organisation), however, unless they are doing > something hostile nobody's going to pester them about that. > > If the project wants to be covered by the trademark grant it can ask to be > added to the github zeromq organisation. > > Any github organisation admin has right to add a new project to the > organisation. Yes. > (This tie to github is a bit silly. > What if it has the repo in svn, hg or somesuch?) No answer. We need some kind of formalizable container. Github works as well as any. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [ANNOUNCE] pyzmq 2.1.7
On Saturday, 14 May 2011 at 13:12, Dirkjan Ochtman wrote: > Nice. I have included it in Gentoo Linux's package tree. > > I noticed the configure whined a bit because I was compiling it > against zeromq-2.1.6 (although all the tests passed). The README says > that for 2.1.x range, all versions of pyzmq-2.1.x should remain > compatible with zeromq-2.1.x. If the latter is true, maybe the > configure warning could be softened a bit? Building against 2.1.6 sounds like a bad idea - there's a pretty big bug in it, unless you've pulled patches separately? Ian ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Windows and IOCP
I've been hacking on it a bit lately. There's a fundamental difference between overlapped I/O with IOCP and the typical Unix file descriptor polling; IOCP events fire on operation completion and not socket readiness. We'd issue overlapped send/recvs (aka asynchronous) and poll for "completions" in the event loop. The whole socket ready paradigm doesn't matter in this case, and the ZMQ codebase is built on the assumption of socket readiness. Throwing a completely different paradigm into the mix will require a refactor. There is a new library that the node.js guys are working on called libuv that has a nice network abstraction layer on top of the high performance I/O multiplexers https://github.com/joyent/libuv. I'm trying to get some inspiration from them, or perhaps give their library a shot. We basically need to rework the ZMQ I/O and polling infrastructure to support asynchronous operations. On Wed, Apr 13, 2011 at 3:22 AM, Martin Sustrik wrote: > Hi Tore, > > > Are there any plans to use IOCP for tcp connections on windows? > > Yes. It was discussed many times (search the mailing list archive for > IOCP) and couple people even gave implementation a try. But, somehow, it > haven't materialised so far. > > Martin > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Jira instance
Hi, Jira instance is now up and running. The issues have been exported from Github and they will be imported on Tuesday (after this Jira can be opened for business). If there are issues being created / updated in the meanwhile I'll add another import request later. Thanks, Mikko ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Simple GFS-like System
On Sat, May 14, 2011 at 12:41 AM, Pieter Hintjens wrote: > On Sat, May 14, 2011 at 8:37 AM, Wolfgang Richter wrote: > > > Backed by ZeroMQ (right now, only REQ-REP sockets; planning on using > > PUSH-PULL for pipelining later) and Redis (for metadata storage): > > https://github.com/theonewolf/TripleD/ > > Neat! I've sent a tweet to my three loyal followers. > > It's really great to see ZeroMQ being used as the transport layer for > smart new projects like this. If you want maximum visibility for > Triple-D, I'd suggest writing a blog posting that explains your goals, > and also acts as a simple tutorial for ZeroMQ. > > -Pieter > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > This is a great concept! I use MooseFS a great deal in my datacenters and over and over again I say "we need this written with ZeroMQ and to use redis as the metadata back end"! You are on the right track, and I hope to see this grow into something more powerful! -Thomas S Hatch ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [ANNOUNCE] pyzmq 2.1.7
On Sat, May 14, 2011 at 6:12 AM, Dirkjan Ochtman wrote: > On Sat, May 14, 2011 at 02:03, MinRK wrote: > > I cut pyzmq-2.1.7 today, to match libzmq-2.1.7's release. This is the > > first version to include (experimental, and probably only partial) > > support for libzmq 3.0. > > > > https://github.com/zeromq/pyzmq/downloads > > Nice. I have included it in Gentoo Linux's package tree. > > I noticed the configure whined a bit because I was compiling it > against zeromq-2.1.6 (although all the tests passed). The README says > that for 2.1.x range, all versions of pyzmq-2.1.x should remain > compatible with zeromq-2.1.x. If the latter is true, maybe the > configure warning could be softened a bit? > > Cheers, > > Dirkjan > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > Thanks for pyzmq! I use it a LOT! Updated the Arch Linux packages: https://aur.archlinux.org/packages.php?ID=47713 https://aur.archlinux.org/packages.php?ID=47714 -Thomas S Hatch ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ File Server
On Sat, May 14, 2011 at 8:32 AM, Paul Colomiets wrote: > > > On Sat, May 14, 2011 at 6:40 AM, Thomas S Hatch wrote: > >> I have just posed a small, quick article on the back end required for a >> VERY simple file server in ZeroMQ, for those of you interested: >> >> http://red45.wordpress.com/2011/05/13/zeromq-file-server/ >> >> I am, of course, and as usual, open to suggestions! >> > > Adding to Martin's and Pieter's suggestions: > * you use isfile() which does stat() call anyway, so use getsize() instead > and allocate buffer all at once (as Martin said file is held in memory > anyway) > * you *must* check the path against limited set of directories to send > from, even if you will run server in fully controlled environment (and use > realpath() for checking) > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > Great! This is what I was hoping to hear! If the entire file is getting read into ram anyway, how would I go about making a file server that would send the files in chunks and not read the whole thing into ram? I think that you can see me intention here, but I gues I am still not entirely sure how to do it yet. And yes Pieter, I am very much interested in seeing your information on how to best make a zeroMQ file server! ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How to cancel a timer in zloop
Hi Pieter, After looking into the source, I find there is no easy way to add support for removing a timer without breaking current API. A reader is associated with a socket, so we can unregistered reader by calling zloop_cancel (zloop_t *self, void *socket) with a specific socket as parameter. But for a timer, when created and registered in a zloop, there is no handle or something similar returned by any API calls. So my attempt to patch it by adding a similar API like zloop_cancel_timer failed. Any ideas to maintain compatibility? Kind regards, - Bo mail.xia...@gmail.com On 14 May 2011, at 6:54 PM, Pieter Hintjens wrote: > Do you want to try patching zloop to add this? Otherwise I'll do it, it'll > take a few days, I'm traveling right now. > > -Pieter > > On 14 May 2011 11:41, "Bo Xiao" wrote: > > Hi all, > > I'm using zeromq c binding czmq master in my project. czmq provide a > > runloop implementation that's easy to use, but there is no way to cancel a > > timer registered in a zloop, which is very inconvenient. any ideas to this > > problem? > > > > thanks in advance. > > > > > > > > Kind regards, > > > > - Bo > > mail.xia...@gmail.com > > > > > > > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ File Server
On Sat, May 14, 2011 at 6:40 AM, Thomas S Hatch wrote: > I have just posed a small, quick article on the back end required for a > VERY simple file server in ZeroMQ, for those of you interested: > > http://red45.wordpress.com/2011/05/13/zeromq-file-server/ > > I am, of course, and as usual, open to suggestions! > Adding to Martin's and Pieter's suggestions: * you use isfile() which does stat() call anyway, so use getsize() instead and allocate buffer all at once (as Martin said file is held in memory anyway) * you *must* check the path against limited set of directories to send from, even if you will run server in fully controlled environment (and use realpath() for checking) ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] HWM default
On Thu, May 12, 2011 at 9:56 AM, Martin Sustrik wrote: > On 05/09/2011 02:41 PM, Paul Colomiets wrote: > > Well, I have no practical experience, but a lot of alloc_assert's in the >> code says that it will be aborted. BTW, it will be so not only if >> overcommit_memory is set right, but also if ulimit is set for the >> application's memory. Which is awfully bad IMO. >> > > What else would you do if you run out of memory? Any recovery mechanism in > user space is likely to fail because there's no memory available :) Killing > (and possibly restarting) the app seems like a reasonable option to me. On the application side you can flush buffers, flush cache, close connections. On zeromq side you can stop accepting connections, stop processing incoming data, drop incoming messages, drop messages already in queue (delivery is not reliable anyway), wait (for data to be processed), notify user about memory failure. Enought? For some applications it's crucial, e.g. for databases which has a write cache, for persistent queues which postpone their writes, for game servers having some state in memory, whatever. You right that it's reasonable for some *applications*. But it's quite mad for networking library. It will be fixed in kernel implementation anyway (may be using other methods). Still I'm pretty sure it should be fixed in a library. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [ANNOUNCE] pyzmq 2.1.7
On Sat, May 14, 2011 at 02:03, MinRK wrote: > I cut pyzmq-2.1.7 today, to match libzmq-2.1.7's release. This is the > first version to include (experimental, and probably only partial) > support for libzmq 3.0. > > https://github.com/zeromq/pyzmq/downloads Nice. I have included it in Gentoo Linux's package tree. I noticed the configure whined a bit because I was compiling it against zeromq-2.1.6 (although all the tests passed). The README says that for 2.1.x range, all versions of pyzmq-2.1.x should remain compatible with zeromq-2.1.x. If the latter is true, maybe the configure warning could be softened a bit? Cheers, Dirkjan ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How to cancel a timer in zloop
Do you want to try patching zloop to add this? Otherwise I'll do it, it'll take a few days, I'm traveling right now. -Pieter On 14 May 2011 11:41, "Bo Xiao" wrote: > Hi all, > I'm using zeromq c binding czmq master in my project. czmq provide a runloop implementation that's easy to use, but there is no way to cancel a timer registered in a zloop, which is very inconvenient. any ideas to this problem? > > thanks in advance. > > > > Kind regards, > > - Bo > mail.xia...@gmail.com > > > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Trademark policy for 0MQ
On 05/14/2011 11:16 AM, Pieter Hintjens wrote: > The current model is working, and I'd not change it for now, until we > see specific problems cropping up. What about summarising it like this: There are projects out there that are not covered by trademark grant (as they are not part of zeromq organisation), however, unless they are doing something hostile nobody's going to pester them about that. If the project wants to be covered by the trademark grant it can ask to be added to the github zeromq organisation. (This tie to github is a bit silly. What if it has the repo in svn, hg or somesuch?) Any github organisation admin has right to add a new project to the organisation. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] How to cancel a timer in zloop
Hi all, I'm using zeromq c binding czmq master in my project. czmq provide a runloop implementation that's easy to use, but there is no way to cancel a timer registered in a zloop, which is very inconvenient. any ideas to this problem? thanks in advance. Kind regards, - Bo mail.xia...@gmail.com ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Trademark policy for 0MQ
On Sat, May 14, 2011 at 11:00 AM, Martin Sustrik wrote: > That's what I meant. By granting trademarks to github zeromq organisation > any organisation admin to extend the trademarks to arbitrary new project by > adding it to the organisation. Indeed. > That kind of process is definitely an option. The other option would be to > have a voting process etc. The current model is working, and I'd not change it for now, until we see specific problems cropping up. What's clear to me is that regular (every six months or so?) meetings for core contributors would be a good thing. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Trademark policy for 0MQ
Hi Pieter, >> 5. The trademarks are currently owned by iMatix. Given that iMatix is a >> commercial entity, ownership of trademarks on behalf of community can be a >> bit delicate. There's still an option to pass to ownership to some >> foundation, such as spi (owns trademarks for debian, libreoffice etc.) > > To be honest, the trademarks represent non-trivial value to iMatix and > it would be hard to literally give them away. This is not really an > option, though it's one I've considered. There would have to be > compelling reasons (e.g. real dysfunction that puts the community at > risk), and it'd have to include the domain names. Someone has to own the trademarks, so iMatix is as good as any other guy. > >> 6. Whoever owns the trademarks on behalf of the community, there still have >> to be a governance process to manage those trademarks. > > Not quite what I meant. > > The governance process is to manage what falls under whatever grant > iMatix makes. In my initial proposal, the grant covers any project > held under the ZeroMQ organization at github.com, and the community > manages this organization. That's what I meant. By granting trademarks to github zeromq organisation any organisation admin to extend the trademarks to arbitrary new project by adding it to the organisation. That kind of process is definitely an option. The other option would be to have a voting process etc. For record, current admins of zeromq organisation at github are: Chuck Remes (ffi-rzmq) Brian Granger (pyzmq) Gonzalo Diethelm (jzmq) Daisuke Maki (ZeroMQ-Perl) Martin Lucina (libzmq build system, man pages) Pieter Hintjens (zguide, czmq) Martin Sustrik (libzmq) Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Windows throughput perfomance
On 05/14/2011 08:08 AM, Pieter Hintjens wrote: > No idea why some apps run at full speed and 0MQ apps run at 1/20th > their normal speed... It definitely needs investigation. However, I have no windows network to test it on, so I'll leave this issue for others to check. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Apparent queuing on PUB/SUB fan-in despite HWM=1
On 05/13/2011 08:02 PM, Brian Rossa wrote: > The Question: "Is ZeroMQ suitable for use in real-time and just-in-time > systems?" The answer is: "Yes. It should be." If it is not, it's a bug. Please report the issue and provide minimal test case. Etc. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Duplicate messages on subscriber reconnect in pub/sub pair
On 05/13/2011 06:31 PM, Benn Bollay wrote: >> Not even EPOLLERR? > > I haven't experimented directly, but the documentation indicates that > it just gets cleaned up. There was some mildly related discussion on > this point here: http://lwn.net/Articles/430793/ > > I haven't banged together a test case, but if it is generating an > EPOLLERR, then 0mq surely isn't cleaning up the lingering reference > internally. Ok. But what are we actually discussing here? 0MQ is surely not expected to be resilient against attaching it to gdb and messing with its internal structures. Are you seeing any problem in real usage? Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Identities as cookies
Hi Oliver, > If this is all sounding feasible... I personally believe that explicit identities are a misdesigned feature and should not be used. So I'm leaving the question to be asnwered by those who are in favour of them... > - How would I poke ZeroMQ to give me a specific delivery tolerance? > sock.setsockopt(zmq.TIMEOUT, 30 * 1000) What's delivery tolerance? You mean timouting on recv()? If so, currently it has to be done via zmq_poll() which has a timeout parameter. > - Is there a way to tell the XREP socket on the server to flush any > backlog/close the underlying tcp socket associated with a particular > identity? No. > - Most importantly, where and how much overhead does the identity impose > across a single ZMQ-hop? > clientserver > Is it only in the handshaking? Is it repeated periodically? Or is it a > per-message overhead? It only happens when TCP connection is established. It's not even a handshake. Each peer just sends the identity and that's it. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] EPGM Congestion Control
On 05/13/2011 09:58 PM, Steven McCoy wrote: > To reiterate for Michael, this is how ZeroMQ provides high > performance. Messages are only delayed when the socket pushes back, > your latencies are driven by NIC performance and any interrupt > coalescing. Right. If you want to minimise latency, turning off interrupt coalescing in your NIC is probably the best start. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ File Server
On 05/14/2011 08:20 AM, Pieter Hintjens wrote: > Thomas, > >> I have just posed a small, quick article on the back end required for a VERY >> simple file server in ZeroMQ, for those of you interested: >> http://red45.wordpress.com/2011/05/13/zeromq-file-server/ > > Very nice. Making a file server was one of the items on the (very > long) list for the Guide. If you do want to make a general purpose > server, I'd consider: > > * using a large block size, e.g. 64K at least, for performance reasons > * make it work both from server-to-client and client-to-server > * implementing a simple staging protocol to allow restartable file transfers Also note that multipart messages are held in memory and sent as an atomic unit. Thus they aren't really suited for sending a file split into chunks. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
