Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
2010/11/16 Martin Sustrik sust...@250bpm.com:
If someone actually tests that setting FD_SETSIZE to 1024 in 0MQ MSVC
project doesn't break client applications with different FD_SETSIZE
values (say default 64), I'll apply the change straight away.
I think it's not good idea. Maybe I don't undestand code correctly,
but this part:
// Store the file descriptor.
fd_entry_t entry = {fd_, events_};
fds.push_back (entry);
// Ensure we do not attempt to select () on more than FD_SETSIZE
// file descriptors.
zmq_assert (fds.size () = FD_SETSIZE);
push into fds descriptors, but where is pop? If FD_SETSIZE = 1024,
application fail on 1024 cycle, right?
--
С уважением,
Олег Севостьянов
JabberID: boo...@jabber.ru
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Strange client application fail
Hi. Maybe is not a bug, but I have got strange error while debuging with MSVC 2005. I have test client application (http://pastebin.com/emer9049) and test server application (http://pastebin.com/YTW732Xb). I run both under MSVC in debug. In server application I set break point (see place in comments). When breakpoint is triggered I stop debugging (kill application). In client application I get error null pointer reference in object.cpp line 85. Call stack information - http://pastebin.com/jprBVFM8. Dump of Locals - http://pastebin.com/mMEepQyP. P.S. Sorry for my English. -- С уважением, Олег Севостьянов JabberID: boo...@jabber.ru ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
On 11/16/2010 09:33 AM, Олег Севостьянов wrote:
push into fds descriptors, but where is pop?
Here it is:
// Destroy retired event sources.
if (retired) {
fds.erase (std::remove_if (fds.begin (), fds.end (),
zmq::select_t::is_retired_fd), fds.end ());
retired = false;
}
Martin
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
Am 16.11.2010 10:21, schrieb Martin Sustrik: More or less. It would be nice to call select() in client app to see whether it works ok. Is there a particular patch I should apply to master beforehand? Christian ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
2010/11/16 Martin Sustrik sust...@250bpm.com: Nope. Yust set FD_SETSIZE to 1024 in 0MQ MSVC project. I try to change select.cpp - remove FD_SETSIZE and set 1024 as const, and then run my test. // Ensure we do not attempt to select () on more than FD_SETSIZE // file descriptors. zmq_assert (fds.size () = 1024); Test failed on 511 cycle with message zmq error: Too many open files. -- С уважением, Олег Севостьянов JabberID: boo...@jabber.ru ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
Am 16.11.2010 10:27, schrieb Олег Севостьянов: I try to change select.cpp - remove FD_SETSIZE and set 1024 as const, and then run my test. FD_SETSIZE is used internally by the Winsock headers as well so just replacing it with a constant in the 0MQ sources will not do the trick. Did you actually try to define FD_SETSIZE via the project options? Christian ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
On Tue, Nov 16, 2010 at 10:27 AM, Олег Севостьянов oleg@gmail.com wrote: I try to change select.cpp - remove FD_SETSIZE and set 1024 as const, and then run my test. You can either: * #define FD_SETSIZE 1024 in the 4 places that 0MQ includes winsock.h or winsock2.h (zmq.h, select.cpp, select.hpp, windows.hpp afair) * -DFD_SETSIZE=1024 in the project -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
On Tue, Nov 16, 2010 at 10:35 AM, Pieter Hintjens p...@imatix.com wrote: * #define FD_SETSIZE 1024 in the 4 places that 0MQ includes winsock.h or winsock2.h (zmq.h, select.cpp, select.hpp, windows.hpp afair) Sorry, I'm not being clear. You need to define FD_SETSIZE _before_ including winsock.h/winsock2.h. This constant defines the size of the fd_set type used for all calls to select() and the FD_CLR/FD_SET/etc. macros in a specific source file. The simplest is to define this externally, in the MSVC project file, as a macro definition that gets passed to all source files regardless. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] [PATCH] Maximum message size
Hi
It has been a little while since I posted the first patch for this. This patch
applies ON TOP OF the previous patch and fixes the comments that I received on
the earlier patch (errors I had made really). It includes support for the
one_byte_size case and uses the logic that was agreed on for determining the
size.
I investigated a way to notify the sender of violating the maximum size if he
set it himself but I think this should wait until sys://log and the like is in
place. Conceivable you could then have a pgm broadcast stating a node's
settings and thus give client a chance to use this as well, but that should be
entirely up to the application.
Regards,
Mikael
From 822c0d33f8c73eec93c00f29fb27c0b7097ff85c Mon Sep 17 00:00:00 2001
From: Mikael Kjaer m...@designtech.dk
Date: Fri, 12 Nov 2010 21:23:08 +0100
Subject: [PATCH] Fix maximum message size option and decoder.
Signed-off-by: Mikael Kjaer m...@designtech.dk
---
src/decoder.cpp |7 +++--
src/decoder.hpp |4 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/decoder.cpp b/src/decoder.cpp
index 45fd4ef..1ee2029 100644
--- a/src/decoder.cpp
+++ b/src/decoder.cpp
@@ -54,8 +54,9 @@ bool zmq::decoder_t::one_byte_size_ready ()
next_step (tmpbuf, 8, decoder_t::eight_byte_size_ready);
else {
+size_t size = (size_t)*tmpbuf;
// There has to be at least one byte (the flags) in the message).
-if (!*tmpbuf) {
+if (!size || over_maximum_size(size)) {
decoding_error ();
return false;
}
@@ -63,7 +64,7 @@ bool zmq::decoder_t::one_byte_size_ready ()
// in_progress is initialised at this point so in theory we should
// close it before calling zmq_msg_init_size, however, it's a 0-byte
// message and thus we can treat it as uninitialised...
-int rc = zmq_msg_init_size (in_progress, *tmpbuf - 1);
+int rc = zmq_msg_init_size (in_progress, size - 1);
if (rc != 0 errno == ENOMEM) {
decoding_error ();
return false;
@@ -82,7 +83,7 @@ bool zmq::decoder_t::eight_byte_size_ready ()
size_t size = (size_t) get_uint64 (tmpbuf);
// There has to be at least one byte (the flags) in the message).
-if (!size !under_maximum_size(size)) {
+if (!size || over_maximum_size(size)) {
decoding_error ();
return false;
}
diff --git a/src/decoder.hpp b/src/decoder.hpp
index 6dc234e..47c4a51 100644
--- a/src/decoder.hpp
+++ b/src/decoder.hpp
@@ -159,9 +159,9 @@ namespace zmq
next = NULL;
}
-inline bool under_maximum_size (size_t size)
+inline bool over_maximum_size (const size_t size)
{
-return (max_msgsize != 0 size max_msgsize);
+return (max_msgsize != 0 size max_msgsize);
}
private:
--
1.7.0.2.msysgit.0
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
Am 16.11.2010 10:21, schrieb Martin Sustrik: More or less. It would be nice to call select() in client app to see whether it works ok. FD_SETSIZE is still #defined to 64 in client applications even if 0MQ itself got compiled with FD_SETSIZE=1024. select() still appears to work, but that's no surprise since it's FD_SETSIZE agnostic and just looks at the fd_set.fd_count member. The tests have been run using libzmq.dll compiled with MSVC++ 2010 Express. Christian ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
2010/11/16 Pieter Hintjens p...@imatix.com: On Tue, Nov 16, 2010 at 10:27 AM, Олег Севостьянов oleg@gmail.com wrote: You can either: * -DFD_SETSIZE=1024 in the project I add -DFD_SETSIZE=1024 in the project under Compiler-Command Line, rebuild libzmq.dll and my test application. Test fail with unhadled exception in own.cpp:141, fnd with 0MQ error message before - Too many open files. -- С уважением, Олег Севостьянов JabberID: boo...@jabber.ru ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Assertion failed: fds.size() = FD_SETSIZE in select.cpp:67
Oleg, I add -DFD_SETSIZE=1024 in the project under Compiler-Command Line, rebuild libzmq.dll and my test application. Test fail with unhadled exception in own.cpp:141, fnd with 0MQ error message before - Too many open files. I've checked your code. You are creating connection in a loop. You have to run out of file descriptors at some point. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [PATCH] Maximum message size
Hi Mikael,
Your patch does not apply to the master:
sust...@istvan:~/zeromq2$ git apply
0001-Implement-maximum-message-size-and-a-socket-option-t.patch
0001-Implement-maximum-message-size-and-a-socket-option-t.patch:9:
trailing whitespace.
#define ZMQ_MAX_MSGSIZE 20
0001-Implement-maximum-message-size-and-a-socket-option-t.patch:23:
trailing whitespace.
zmq::decoder_t::decoder_t (size_t bufsize_, uint64_t max_msgsize_) :
0001-Implement-maximum-message-size-and-a-socket-option-t.patch:24:
trailing whitespace.
decoder_base_t decoder_t (bufsize_, max_msgsize_),
0001-Implement-maximum-message-size-and-a-socket-option-t.patch:33:
trailing whitespace.
if (!size !under_maximum_size(size)) {
0001-Implement-maximum-message-size-and-a-socket-option-t.patch:45:
trailing whitespace.
#include stdint.hpp
error: patch failed: include/zmq.h:194
error: include/zmq.h: patch does not apply
error: patch failed: src/decoder.cpp:25
error: src/decoder.cpp: patch does not apply
error: patch failed: src/decoder.hpp:25
error: src/decoder.hpp: patch does not apply
error: patch failed: src/encoder.hpp:93
error: src/encoder.hpp: patch does not apply
error: patch failed: src/options.cpp:39
error: src/options.cpp: patch does not apply
error: patch failed: src/options.hpp:73
error: src/options.hpp: patch does not apply
error: patch failed: src/pgm_receiver.cpp:211
error: src/pgm_receiver.cpp: patch does not apply
error: patch failed: src/zmq_engine.cpp:35
error: src/zmq_engine.cpp: patch does not apply
Can you rebase your repo to match the master and send the patch once again?
Thanks.
Martin
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] publishing to the same port from multiple threads
Hi, What is the best approach to publish messages to the same port from the multiple threads? We are using zmq 2.0.10. Thank you. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] publishing to the same port from multiple threads
On 11/16/2010 02:20 PM, T-zex wrote: Hi, What is the best approach to publish messages to the same port from the multiple threads? We are using zmq 2.0.10. Use in-process forwarder device to collect messages from different threads and push them to the network. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails
sust...@250bpm.com said: On 11/12/2010 11:34 AM, Christian Gudrian wrote: Hello, again! test_shutdown_stress proves stressing under Windows, too: The send call at mailbox.cpp:76 might return SOCKET_ERROR with GetLastError() == WSAEWOULDBLOCK. The wsa_assert macro ignores this special case and allows program execution as if nothing had happened. Eventually it's the following zmq_assert which aborts the program. Is there any reasonable way to appropriately handle this condition? The buffer resizing code, such as the one in POSIX implementation of mailbox_t::send should exist on windows as well. As Martin says, there is no reasonable way to handle WSAEWOULDBLOCK in mailbox_t::send(). If it's not failing on that right now then that's a bug and it should just assert. As for the buffer-resizing code for Windows, feel free to port the code from the POSIX implementation. I have no idea if on-the-fly resizing of a socket's SNDBUF will work reliably on Winsock. Even then the whole buffer-resizing operation is more a hack than anything else; unfortunately all the other options we investigated with Martin Sustrik for reimplementing the way mailbox_t works seem to be dead ends right now :-( -mato ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Logging format for sys://log transport
On Mon, Nov 15, 2010 at 11:34 PM, Pieter Hintjens p...@imatix.com wrote: OK, /me found a real keyboard and can explain this better. Here's my proposal for the topic header: * Number of repeated characters indicates severity from 1 to 3 (X = information, XX = warning, XXX = error) * Character indicates category of error: '?' indicates internal 0MQ error, '*' indicates bad data, '#' indicates resource error, etc. Now I can subscribe to keys like '?' (any internal error), '***' (only serious bad data errors and up), '##' (any resource errors apart from informative). It is trivial to turn XXX into 3 if that's needed. This is just a sketch, we may want more severity levels eventually. I'd probably not use version numbers in the API since it would be very hard to manage multiple versions of the error subsystem at once. And we don't need multipart data, that is too complex. Just a topic that specifies the severity and category, and then a textual message, IMO. The simpler the better at this point. One potential downside with this approach is that it could make it challenging to search for characters like '?'. You could get the same functionality through http-style number status codes, where informational messages are 1xx, client errors are 4xx, server errors are 5xx. You can subscribe to all of the error codes by subscribing to '4' and '5', or if you want just a specific message you subscribe to '401'. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Logging format for sys://log transport
Point is that subscription happens on a pure prefix of the topic key. -Pieter On 16 Nov 2010 19:01, Erick Tryzelaar erick.tryzel...@gmail.com wrote: On Mon, Nov 15, 2010 at 11:34 PM, Pieter Hintjens p...@imatix.com wrote: OK, /me found a real keyboard and can explain this better. Here's my proposal for the topic header: * Number of repeated characters indicates severity from 1 to 3 (X = information, XX = warning, XXX = error) * Character indicates category of error: '?' indicates internal 0MQ error, '*' indicates bad data, '#' indicates resource error, etc. Now I can subscribe to keys like '?' (any internal error), '***' (only serious bad data errors and up), '##' (any resource errors apart from informative). It is trivial to turn XXX into 3 if that's needed. This is just a sketch, we may want more severity levels eventually. I'd probably not use version numbers in the API since it would be very hard to manage multiple versions of the error subsystem at once. And we don't need multipart data, that is too complex. Just a topic that specifies the severity and category, and then a textual message, IMO. The simpler the better at this point. One potential downside with this approach is that it could make it challenging to search for characters like '?'. You could get the same functionality through http-style number status codes, where informational messages are 1xx, client errors are 4xx, server errors are 5xx. You can subscribe to all of the error codes by subscribing to '4' and '5', or if you want just a specific message you subscribe to '401'. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] PyZMQ 2.0.10
Hello list, I cut pyzmq 2.0.10 last night, and pushed it to the downloads section on github and pypi (easy_install/pip) and it should have some useful bits: Headline: * It should build without warnings (in most standard cases) and pass all tests on Python 2.5, 2.6, 2.7, and 3.1 on Mac OSX, Linux, and Windows * significant file reorganization - zeromq bindings in zmq.core, and any extended functionality is in other subpackages (previous import patterns still work just fine) * The reorganization is done in such a way that other Cython projects can link directly against the Cython objects in pyzmq, and are not restricted to the pure Python interface. * Non-copying sends of large messages (e.g. numpy arrays) requires that you know when a message is actually sent, before it is safe to edit the buffer. A 'track' argument has been added to send/recv and the Message constructor, which allows users to track when zeromq is done with a Message. Tracking is only performed on request, so your messages won't be tracked if you don't ask for it. * A Python logging handler that relays system logging messages over a zmq socket (useful for hiding latency of heavy logging applications) * BackgroundDevice: A class that provides a simple wrapper for launching a zmq_device in a background process or thread without blocking the main thread * MonitoredQueueDevice: A Device like ZMQ_QUEUE, but sends all messages coming through the queue in either direction also on an additional side socket. I also added a pyzmq-dev download to GitHub, which is to be a manual pseudo-nightly build of git master (now 2.1.0dev). This will be a way to download a reasonably-current development version, but without depending on Cython. The bindings:python page on zeromq.org should now be up to date with this information. -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Logging format for sys://log transport
i still think this is being clever, rather than just subscribing to the level you want. for your case, just do two subscribes, one of \002 and one of \003. existing practice is for several levels (~7), and do we really want to count that many out? On Nov 16, 2010, at 11:19 AM, Pieter Hintjens wrote: Point is that subscription happens on a pure prefix of the topic key. -Pieter On 16 Nov 2010 19:01, Erick Tryzelaar erick.tryzel...@gmail.com wrote: On Mon, Nov 15, 2010 at 11:34 PM, Pieter Hintjens p...@imatix.com wrote: OK, /me found a real keyboard and can explain this better. Here's my proposal for the topic header: * Number of repeated characters indicates severity from 1 to 3 (X = information, XX = warning, XXX = error) * Character indicates category of error: '?' indicates internal 0MQ error, '*' indicates bad data, '#' indicates resource error, etc. Now I can subscribe to keys like '?' (any internal error), '***' (only serious bad data errors and up), '##' (any resource errors apart from informative). It is trivial to turn XXX into 3 if that's needed. This is just a sketch, we may want more severity levels eventually. I'd probably not use version numbers in the API since it would be very hard to manage multiple versions of the error subsystem at once. And we don't need multipart data, that is too complex. Just a topic that specifies the severity and category, and then a textual message, IMO. The simpler the better at this point. One potential downside with this approach is that it could make it challenging to search for characters like '?'. You could get the same functionality through http-style number status codes, where informational messages are 1xx, client errors are 4xx, server errors are 5xx. You can subscribe to all of the error codes by subscribing to '4' and '5', or if you want just a specific message you subscribe to '401'. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Andrew Hume (best - Telework) +1 623-551-2845 and...@research.att.com (Work) +1 none currently ATT Labs - Research; member of USENIX and LOPSA ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Logging format for sys://log transport
On 11/16/2010 07:24 PM, Andrew Hume wrote: existing practice is for several levels (~7), and do we really want to count that many out? +1 Rarely more than 3 levels are used in practice. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Logging format for sys://log transport
It lets you subscribe to both GE severity and EQ category in one go. Yes, it's being clever... sorry about that. But it makes use simpler IMO. Think about taking subscriptions from a config file. Anyhow, I'm pretty sure we don't even need categories here. -Pieter On 16 Nov 2010 19:25, Andrew Hume and...@research.att.com wrote: ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Preferred spelling of ØMQ?
aha, I see someone finally registered xn--mq-kka.org Unfortunately the cachet is lost with an ANSI redirect. -- Steve-o On 6 April 2010 09:19, Steven McCoy steven.mc...@miru.hk wrote: Doesn't look like the IDN domain name is taken either, although where would one register it as Network Solutions doesn't seem to take the punycode. ØMQ.org = xn--mq-kka.org :-) Still looking for anyone using IDN, not in Hong Kong or Taiwan for sure even though HKDNR will sell you a IDN domain. -- Steve-o On 6 April 2010 21:09, Michael Glaesemann g...@seespotcode.net wrote: Looking around, I've seen ØMQ spelled a variety of ways. ØMQ LATIN CAPITAL LETTER O WITH STROKE, LATIN CAPITAL LETTER M, LATIN CAPITAL LETTER Q 0MQ DIGIT ZERO, LATIN CAPITAL LETTER M, LATIN CAPITAL LETTER Q zmq zeromq What's the preferred spelling? Are there usage differences between these spellings? Michael Glaesemann grzm seespotcode net ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PyZMQ 2.0.10
Many thanks to Min for his excellent work on PyZMQ for this release!!! PyZMQ has really taken a huge step forward because of his efforts. Cheers, Brian On Tue, Nov 16, 2010 at 10:21 AM, MinRK benjami...@gmail.com wrote: Hello list, I cut pyzmq 2.0.10 last night, and pushed it to the downloads section on github and pypi (easy_install/pip) and it should have some useful bits: Headline: * It should build without warnings (in most standard cases) and pass all tests on Python 2.5, 2.6, 2.7, and 3.1 on Mac OSX, Linux, and Windows * significant file reorganization - zeromq bindings in zmq.core, and any extended functionality is in other subpackages (previous import patterns still work just fine) * The reorganization is done in such a way that other Cython projects can link directly against the Cython objects in pyzmq, and are not restricted to the pure Python interface. * Non-copying sends of large messages (e.g. numpy arrays) requires that you know when a message is actually sent, before it is safe to edit the buffer. A 'track' argument has been added to send/recv and the Message constructor, which allows users to track when zeromq is done with a Message. Tracking is only performed on request, so your messages won't be tracked if you don't ask for it. * A Python logging handler that relays system logging messages over a zmq socket (useful for hiding latency of heavy logging applications) * BackgroundDevice: A class that provides a simple wrapper for launching a zmq_device in a background process or thread without blocking the main thread * MonitoredQueueDevice: A Device like ZMQ_QUEUE, but sends all messages coming through the queue in either direction also on an additional side socket. I also added a pyzmq-dev download to GitHub, which is to be a manual pseudo-nightly build of git master (now 2.1.0dev). This will be a way to download a reasonably-current development version, but without depending on Cython. The bindings:python page on zeromq.org should now be up to date with this information. -MinRK -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgran...@calpoly.edu elliso...@gmail.com ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PyZMQ support for socket options added in ZeroMQ 2.1
For those interested in ZMQ_LINGER, etc. on pyzmq, since 2.0.10 was cut last night, master now targets 2.1.0, and I have added the new sockopt constants. From now on, we will maintain a pyzmq-dev release in the downloads section on github, where you can get the current (or at least recent) state of master, and like real releases it will not require cython. -MinRK On Thu, Nov 11, 2010 at 06:01, Martin Lucina m...@kotelna.sk wrote: m...@kotelna.sk said: Ok, I'll try with Cython 13 tomorrow, it's late here ... if that doesn't work I guess I can always hand-edit the generated Cython code in the interim. Cython 13 installed from source works, thanks for the hint. -mato ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] how to integrate zmq to normal epoll web server
Hello all, I am a newbie of ZMQ. I want to write a web server in C++ that use ZMQ to transfer the job to worker and receive the respone from the worker. The web server using epoll runs in a process separated from the worker process. In the web server process we have 1 main thread and two child threads. when initializing, the main thread creates two child threads: one for transferring the jobs to the PUSH queue (called C1), one for receive the response from the SUB queue (like Mongrel2),called C2. the main thread uses epoll to accept the connection from the client, parses HTTP header and then pushes the job to a queue then the C1 gets job from internal queue and push to ZMQ PUSH queue. My question is: - Is this architect OK? - Sholud I use 1 or 2 ZMQ I/O thread for C1 and C2 ? - Because ZMQ does not support HTTP, is there any way to integrate HTTP socket accepted from epoll into ZMQ polling ? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] More security thoughts
Hi, We (the pyzmq devs) have been thinking and talking a lot about security lately and Min has been prototyping various ideas. I wanted to report back on our findings as I think they are relevant to the larger security discussion. For the record, there are 3 main security approaches that are being discussed for zeromq: 1. Message layer security. Use a regular TCP socket and encrypt/authenticate each message part individually. 2. Tunnel zeromq through a VPN or SSH tunnel. 3. Make an actual SSL/TLS zeromq transport. Min has prototyped both the (1) and (2) options. For (1) the idea was: * Use PyCrypto (pycrypto.org) to encrypt (AES) and authenticate (HMAC) each message part. * For now assume that the key distribution/exchange problem is solved (ha, ha!) (no PKI for now). What we found is that this approach has a significant performance penalty (I think it was in the 3-10x range above a copying send/recv). A performance penalty was expected, BUT there was an aspect of this that we had not thought about previously. One of the core ideas in zeromq is that in application code, a send/recv returns instantly. This is absolutely critical in writing high performance applications that have low latency. This is possible as the actual send/recv is done by zeromq in the IO threads. BUT, when you do message level encryption/authentication all of the padding/encrypting/HMAC/etc is done in the application thread. Thus, every call to send/recv sees the performance hit. This is a huge effect and at least for our applications is quite difficult to tolerate. Of course, zeromq could be built with message level security that happens in the IO threads, but that is very different and would require lots of low level modifications that may affect performance in the IO threads (you don't really want CPU bound work in IO threads). Another thing that we realized is that message level security completely destroys the nice non-copy send/recv capabilities of zeromq. The reason for this is that each message has to go through a padding/encyption/HMAC chain and that makes multiple copies of the data. Sure the final data can be sent in a noncopying manner, but only after you make multiple copies first. Part of the difficulty is that this has to be done for each message and the transport can't make any decisions about how to chuck the data over the network in an efficient manner. Summary: even if we can figure out how to make message level security bulletproof, there are some serious performance issues. For (2), the idea was: * Tunnel our insecure zeromq connections over SSH tunnels. * Rely on the security provided by SSH entirely. This worked fairly well and the performance hit with each send/recv is avoided. There is still a performance penalty, but it does not occur in the application thread. But this approach leaves a huge gapping security hole that we can't see how to plug. An ssh tunnel/forwarding has to have an insecure socket listening on the loopback of the server. The forward decrypts the secure data and then sends it on to the insecure socket. The problem with this is that *any* user on that host has full access to the data stream and can do anything they want. In some environments this is not an issue, but having shared hosts with untrusted users is not that uncommon in real life. Summary: tunnels have a serious security hole in the untrusted localhost environment. Tunnels only work for securing remote connections. Alright, that is all for now. Cheers, Brian -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgran...@calpoly.edu elliso...@gmail.com ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] how to integrate zmq to normal epoll web server
I know you may be tied to C++, but the Python bindings have a nice event loop that is integrates with the Tornado web server. Cheers, Brian On Tue, Nov 16, 2010 at 8:04 PM, technical issue techwe...@gmail.comwrote: Hello all, I am a newbie of ZMQ. I want to write a web server in C++ that use ZMQ to transfer the job to worker and receive the respone from the worker. The web server using epoll runs in a process separated from the worker process. In the web server process we have 1 main thread and two child threads. when initializing, the main thread creates two child threads: one for transferring the jobs to the PUSH queue (called C1), one for receive the response from the SUB queue (like Mongrel2),called C2. the main thread uses epoll to accept the connection from the client, parses HTTP header and then pushes the job to a queue then the C1 gets job from internal queue and push to ZMQ PUSH queue. My question is: - Is this architect OK? - Sholud I use 1 or 2 ZMQ I/O thread for C1 and C2 ? - Because ZMQ does not support HTTP, is there any way to integrate HTTP socket accepted from epoll into ZMQ polling ? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgran...@calpoly.edu elliso...@gmail.com ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] how to integrate zmq to normal epoll web server
Thank for your quick reply. But my project must use C++ for other reason. On Wed, Nov 17, 2010 at 11:18 AM, Brian Granger elliso...@gmail.com wrote: I know you may be tied to C++, but the Python bindings have a nice event loop that is integrates with the Tornado web server. Cheers, Brian On Tue, Nov 16, 2010 at 8:04 PM, technical issue techwe...@gmail.comwrote: Hello all, I am a newbie of ZMQ. I want to write a web server in C++ that use ZMQ to transfer the job to worker and receive the respone from the worker. The web server using epoll runs in a process separated from the worker process. In the web server process we have 1 main thread and two child threads. when initializing, the main thread creates two child threads: one for transferring the jobs to the PUSH queue (called C1), one for receive the response from the SUB queue (like Mongrel2),called C2. the main thread uses epoll to accept the connection from the client, parses HTTP header and then pushes the job to a queue then the C1 gets job from internal queue and push to ZMQ PUSH queue. My question is: - Is this architect OK? - Sholud I use 1 or 2 ZMQ I/O thread for C1 and C2 ? - Because ZMQ does not support HTTP, is there any way to integrate HTTP socket accepted from epoll into ZMQ polling ? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgran...@calpoly.edu elliso...@gmail.com ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Preferred spelling of ØMQ?
It wasn't iMatix, in any case, our registrar couldn't handle the name. I assume it's a squatter. -Pieter On 16 Nov 2010 22:25, Steven McCoy steven.mc...@miru.hk wrote: aha, I see someone finally registered xn--mq-kka.org Unfortunately the cachet is lost with an ANSI redirect. -- Steve-o On 6 April 2010 09:19, Steven McCoy steven.mc...@miru.hk wrote: Doesn't look like the IDN domain name is taken either, although where would one register it as Network Solutions doesn't seem to take the punycode. ØMQ.org = xn--mq-kka.org :-) Still looking for anyone using IDN, not in Hong Kong or Taiwan for sure even though HKDNR will sell you a IDN domain. -- Steve-o On 6 April 2010 21:09, Michael Glaesemann g...@seespotcode.net wrote: Looking around, I've seen ØMQ spelled a variety of ways. ØMQ LATIN CAPITAL LETTER O WITH STROKE, LATIN CAPITAL LETTER M, LATIN CAPITAL LETTER Q 0MQ DIGIT ZERO, LATIN CAPITAL LETTER M, LATIN CAPITAL LETTER Q zmq zeromq What's the preferred spelling? Are there usage differences between these spellings? Michael Glaesemann grzm seespotcode net ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Preferred spelling of ØMQ?
On 11/17/2010 08:17 AM, Pieter Hintjens wrote: It wasn't iMatix, in any case, our registrar couldn't handle the name. I assume it's a squatter. Interestingly, it redirects to zeromq.org Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Preferred spelling of ØMQ?
Hah... I take it back. Can you do a whois on the name? (Don't have that on my Android yet...) - Pieter On 17 Nov 2010 08:19, Martin Sustrik sust...@250bpm.com wrote: On 11/17/2010 08:17 AM, Pieter Hintjens wrote: It wasn't iMatix, in any case, our registrar couldn't handle the name. I assume it's a squatter. Interestingly, it redirects to zeromq.org Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Preferred spelling of ØMQ?
On 11/17/2010 08:21 AM, Pieter Hintjens wrote: Hah... I take it back. Can you do a whois on the name? (Don't have that on my Android yet...) Domain ID:D160113145-LROR Domain Name:XN--MQ-KKA.ORG Created On:10-Sep-2010 14:40:59 UTC Last Updated On:10-Nov-2010 03:49:40 UTC Expiration Date:10-Sep-2011 14:40:59 UTC Sponsoring Registrar:Name.com, LLC (R1288-LROR) Status:CLIENT TRANSFER PROHIBITED Registrant ID:ncr-207-cb17 Registrant Name:Gregory Burd Registrant Organization:Ossus, LLC Registrant Street1:300 McCauley Street Registrant Street2: Registrant Street3: Registrant City:Chapel Hill Registrant State/Province:NC Registrant Postal Code:27516-2722 Registrant Country:US Registrant Phone:+1.6178690997 Registrant Phone Ext.: Registrant FAX: Registrant FAX Ext.: Registrant Email:gb...@ossus.com Admin ID:nca-208-099b Admin Name:Gregory Burd Admin Organization:Ossus, LLC Admin Street1:300 McCauley Street Admin Street2: Admin Street3: Admin City:Chapel Hill Admin State/Province:NC Admin Postal Code:27516-2722 Admin Country:US Admin Phone:+1.6178690997 Admin Phone Ext.: Admin FAX: Admin FAX Ext.: Admin Email:gb...@ossus.com Tech ID:nct-209-47ea Tech Name:Gregory Burd Tech Organization:Ossus, LLC Tech Street1:300 McCauley Street Tech Street2: Tech Street3: Tech City:Chapel Hill Tech State/Province:NC Tech Postal Code:27516-2722 Tech Country:US Tech Phone:+1.6178690997 Tech Phone Ext.: Tech FAX: Tech FAX Ext.: Tech Email:gb...@ossus.com Name Server:NS1.NAME.COM Name Server:NS2.NAME.COM Name Server:NS3.NAME.COM Name Server:NS4.NAME.COM Name Server: Name Server: Name Server: Name Server: Name Server: Name Server: Name Server: Name Server: Name Server: IDN Script:da Unicode Hex:U+00F8;U+006D;U+0071;.org Unicode HTML:#248;mq.org DNSSEC:Unsigned Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] how to integrate zmq to normal epoll web server
On 11/17/2010 05:22 AM, technical issue wrote: I want to write a web server in C++ that use ZMQ to transfer the job to worker and receive the respone from the worker. The web server using epoll runs in a process separated from the worker process. In the web server process we have 1 main thread and two child threads. when initializing, the main thread creates two child threads: one for transferring the jobs to the PUSH queue (called C1), one for receive the response from the SUB queue (like Mongrel2),called C2. the main thread uses epoll to accept the connection from the client, parses HTTP header and then pushes the job to a queue then the C1 gets job from internal queue and push to ZMQ PUSH queue. My question is: - Is this architect OK? I think so. AFAIK Mongrel2 does it the same way. - Sholud I use 1 or 2 ZMQ I/O thread for C1 and C2 ? 1 should be enough. - Because ZMQ does not support HTTP, is there any way to integrate HTTP socket accepted from epoll into ZMQ polling? Either use zmq_poll instead of EPOLL, or get the file descriptor from 0MQ socket useing getsockopt(ZMQ_FD) and use epoll. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Mac OS X: test_shutdown_stress sometimes fails
Hi Martin,
On Sat, Nov 13, 2010 at 5:47 PM, Dhammika Pathirana dhamm...@gmail.com wrote:
Hi Martin,
On Sat, Nov 13, 2010 at 2:18 AM, Martin Sustrik sust...@250bpm.com wrote:
On 11/13/2010 11:13 AM, Dhammika Pathirana wrote:
In finalize_initialization(), we detach the engine and pass it over to
app thread.
Actually, it's passed to another I/O thread rather than to the app thread,
but that's irrelevant to the problem.
correct, my bad.
engine-unplug ();
send_attach (session, engine, peer_identity, false);
engine = NULL;
terminate ();
But there's a stack reference to this engine object,
engine-out_event() callback.
How does the stack look like when the out_event() is called on destroyed
engine?
I'll check if this shows up in valgrind.
I added following dbg prints,
@ -51,6 +51,7 @@ zmq::zmq_engine_t::zmq_engine_t (fd_t fd_, const
options_t options_) :
zmq::zmq_engine_t::~zmq_engine_t ()
{
zmq_assert (!plugged);
+ printf(~engine() %p\n, (void *)this);
}
void zmq::zmq_engine_t::plug (io_thread_t *io_thread_, i_inout *inout_)
@@ -160,6 +161,9 @@ void zmq::zmq_engine_t::out_event ()
}
}
+ if (!plugged)
+ printf(engine %p unplugged\n, (void *)this);
+
stdout just before crash,
engine 0x63e5130 unplugged
~engine() 0x63d3320
~engine() 0x63dbfa0
~engine() 0x63e5130
~engine() 0x63f7810
~engine() 0x63f3410
~engine() 0x63d7bb0
~engine() 0x63e0cd0
engine 0x63f7810 unplugged
Bad file descriptor
~engine() 0x63eefb0
nbytes != -1 (tcp_socket.cpp:197)
Killed
Note ~engine() 0x63f7810 before engine 0x63f7810 unplugged msg.
Here's stack trace,
(gdb) bt
#0 0x05ec8a75 in *__GI_raise (sig=value optimized out) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x05ecc5c0 in *__GI_abort () at abort.c:92
#2 0x04e97962 in zmq::tcp_socket_t::write (this=0x63f7828,
data=0x63f9a30, size=2) at tcp_socket.cpp:197
#3 0x04e9fd3b in zmq::zmq_engine_t::out_event
(this=0x63f7810) at zmq_engine.cpp:169
#4 0x04e76db3 in zmq::epoll_t::loop (this=0x639b1a0) at epoll.cpp:156
#5 0x04e76ee2 in zmq::epoll_t::worker_routine
(arg_=0x639b1a0) at epoll.cpp:173
#6 0x04e97c9a in zmq::thread_t::thread_routine
(arg_=0x639b210) at thread.cpp:79
#7 0x054d09ca in start_thread (arg=value optimized out) at
pthread_create.c:300
#8 0x05f7b70d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9 0x in ?? ()
(gdb) fr 3
#3 0x04e9fd3b in zmq::zmq_engine_t::out_event
(this=0x63f7810) at zmq_engine.cpp:169
169 int nbytes = tcp_socket.write (outpos, outsize);
(gdb) p this
$1 = (zmq::zmq_engine_t * const) 0x63f7810
I could be reading this wrong, and I had to run it few dozen times to
get it to crash.
Dhammika
How about adding a callback to notify when poller is about to destroy
retired event handle?
Something like,
// Destroy retired event sources.
for (retired_t::iterator it = retired.begin (); it !=
retired.end (); it ++) {
// Notify poller is about to delete poll handle
(*it)-events-toss();
delete *it;
}
retired.clear ();
Engine can implement toss() to check a refcount, and delete it if
there's no other object referencing it.
This won't affect data path, but will add some overhead to poller loop.
Dhammika
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
