Re: [zeromq-dev] Interrupted system call
On Thu, Jul 28, 2016 at 2:28 PM, Frédéric wrote: > Le jeudi 28 juillet 2016, MinRK a écrit : > > > Following the behavior of Python 3.5, interrupted system calls are > > retried, starting in pyzmq 14.7. > > I tried added a manual retry, but this does not help. > > > Are you creating the sockets after forking with multiprocessing? You > > shouldn't use any sockets that were created in the host process in any > > of the forks. > > Mmm, no, I don't instanciate a new socket :o/ > > I have to figure out how to do that with my Singleton logger and > multiprocessing module... > > This is strange that it work fine a simple example, or even in my complete > application on my desktop PC, and fail on Rpi... What can cause such > behaviour difference? > > BTW, could someone resume what is needed to re-create or not in a > multi-threaded and multi-processing context? Is ok to share Context? When > do I need to use zmq.Context() or zmq.Context.instance() ? > You must create a new `zmq.Context()` in each process. > > Thanks, > > -- > Frédéric > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Interrupted system call
Following the behavior of Python 3.5, interrupted system calls are retried, starting in pyzmq 14.7. Are you creating the sockets after forking with multiprocessing? You shouldn't use any sockets that were created in the host process in any of the forks. -MinRK On Thu, Jul 28, 2016 at 1:15 PM, Frédéric wrote: > Le lundi 25 juillet 2016, Frédéric a écrit : > > > Mmm, this error only occurs on my RPi3; all work fine on my desktop > > computer. > > Ok, zmq/pyzmq are not in the same versions on both machines: > > libzmq@desktop = 4.1.5 > pyzmq@desktop = 15.2.0 > python@rpi = 2.7.12 > > libzmq@rpi = 4.0.5 > pyzmq@rpi = 14.4.0 > python@rpi = 2.7.9 > > So, I installed pyzqm using pip, which gave me: > > libzmq@rpi = 4.1.5 > pyzmq@rpi = 15.3.0 > > Now, I don't get the 'ZMQError: Interrupted system call' anymore when the > log handler send the message from the process, but instead it blocks > forever on this call :o/ > > -- > Frédéric > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Thread safe Pub/Sub and Multicast
On Wed, Mar 16, 2016 at 11:06 AM, Pieter Hintjens wrote:
> Off-topic, but I need to ask, how do you do the syntax highlighting in
> emails?
>
http://markdown-here.com/
>
> On Mon, Mar 14, 2016 at 4:42 PM, MinRK wrote:
> > On Sat, Mar 12, 2016 at 9:00 PM, Dave Kuhlman
> > wrote:
> >>
> >> For those of us who use Python and the pyzmq library for ZeroMQ, I'm
> >> wondering whether the solution to the problem and need for thread
> >> safety is to avoid threads and to use either the tornado ioloop
> >> support or asyncio ioloop. Am I right that doing so would give a
> >> Python developer the capabilities of multiple threads without the
> >> worries about thread safety.
> >>
> >> One reason I'm asking is that I'm the one who added the ioloop
> >> examples to zguide (in examples/Python/tornado_ioloop and
> >> examples/Python/ascyncio_ioloop). A number of those examples use
> >> (what I understand as) multiple concurrent tasks within a single
> >> thread. So, I'm wondering whether that really is thread safe (or
> >> whatever the equivalent is for tasks), as I believe it is.
> >
> > Yup, asyncio/tornado are single-threaded eventloops, so you won’t get
> > threadsafety problems if everything happens in a callback/coroutine on
> those
> > loops. You can also use these to make threadsafe actions, even when using
> > actual threads. In the tornado case, there is one (and only one)
> threadsafe
> > method, and it’s IOLoop.add_callback. You can use this to schedule calls
> to
> > be made on the IOLoop, from other threads. For instance:
> >
> > loop = start_loop_in_background_thread()
> >
> > def threadsafe_send(socket, msg):
> > loop.add_callback(lambda : socket.send_multipart(msg))
> >
> > In this way, you can schedule sends from any number of threads perfectly
> > safely, because the actual zmq socket is only ever operated on in the IO
> > thread. Threadsafe recv is similarly doable, but a bit more code.
> >
> > On the other hand, you can also write coroutine-unsafe code if you have
> > yields in the middle of sending multipart messages, for instance:
> >
> > @coroutine
> > def bad_coroutine_send(socket, msg):
> > for frame in msg[:-1]:
> > yield socket.send(frame, zmq.SNDMORE)
> > yield socket.send(msg[-1])
> >
> > Where each yield is an opportunity to take over and do some other
> operation
> > on the socket. I think it is a lot easier to write coroutine-safe code
> than
> > thread-safe code, since all context switching is explicit (make sure you
> > have no yield calls in the middle of an operation).
> >
> > -MinRK
> >>
> >>
> >> Any enlightenment will be welcome.
> >>
> >> And, by the way, I suspect that something equivalent could be done
> >> in Node.js (JavaScript), which also uses a single threaded approach
> >> to "concurrency" (psuedo-concurrency, quasi-concurrency, whatever).
> >> In Node.js, you could use callbacks, but there are libraries that
> >> can wrap your code to make is seem more simple and parallel. For
> >> example, see async.js. Here are a few quick examples from the
> >> async.js Web page (https://www.npmjs.com/package/async):
> >>
> >> async.map([`file1`,`file2`,`file3`], fs.stat, function(err,
> results){
> >> // results is now an array of stats for each file
> >> });
> >>
> >> async.filter([`file1`,`file2`,`file3`], fs.exists,
> function(results){
> >> // results now equals an array of the existing files
> >> });
> >>
> >> async.parallel([
> >> function(){ ... },
> >> function(){ ... }
> >> ], callback);
> >>
> >> async.series([
> >> function(){ ... },
> >> function(){ ... }
> >> ]);
> >>
> >> As a bit of justification for this question, the main intension of
> >> the Python ioloop support in tornado and asyncio is to address
> >> problems of network and I/O delays and to enable our code to do
> >> something useful during those delays. That seems like a central
> >> concern for users of ZeroMQ.
> >>
> >> Dave
> >>
> >>
> >> On Sat, Mar 12, 2016 at 03:00:58AM +0200, Doron Somech wrote:
> >> >Hi All,
> >> >During the ZeroMQ hackathon I have added the Radio/Dish pattern
> >> > (thre
Re: [zeromq-dev] Thread safe Pub/Sub and Multicast
On Sat, Mar 12, 2016 at 9:00 PM, Dave Kuhlman
wrote:
For those of us who use Python and the pyzmq library for ZeroMQ, I'm
> wondering whether the solution to the problem and need for thread
> safety is to avoid threads and to use either the tornado ioloop
> support or asyncio ioloop. Am I right that doing so would give a
> Python developer the capabilities of multiple threads without the
> worries about thread safety.
>
> One reason I'm asking is that I'm the one who added the ioloop
> examples to zguide (in examples/Python/tornado_ioloop and
> examples/Python/ascyncio_ioloop). A number of those examples use
> (what I understand as) multiple concurrent tasks within a single
> thread. So, I'm wondering whether that really is thread safe (or
> whatever the equivalent is for tasks), as I believe it is.
>
Yup, asyncio/tornado are single-threaded eventloops, so you won’t get
threadsafety problems if everything happens in a callback/coroutine on
those loops. You can also use these to make threadsafe actions, even when
using actual threads. In the tornado case, there is one (and only one)
threadsafe method, and it’s IOLoop.add_callback. You can use this to
schedule calls to be made on the IOLoop, from other threads. For instance:
loop = start_loop_in_background_thread()
def threadsafe_send(socket, msg):
loop.add_callback(lambda : socket.send_multipart(msg))
In this way, you can *schedule* sends from any number of threads perfectly
safely, because the actual zmq socket is only ever operated on in the IO
thread. Threadsafe recv is similarly doable, but a bit more code.
On the other hand, you can also write coroutine-unsafe code if you have
yields in the middle of sending multipart messages, for instance:
@coroutinedef bad_coroutine_send(socket, msg):
for frame in msg[:-1]:
yield socket.send(frame, zmq.SNDMORE)
yield socket.send(msg[-1])
Where each yield is an opportunity to take over and do some other operation
on the socket. I think it is a lot easier to write coroutine-safe code than
thread-safe code, since all context switching is explicit (make sure you
have no yield calls in the middle of an operation).
-MinRK
> Any enlightenment will be welcome.
>
> And, by the way, I suspect that something equivalent could be done
> in Node.js (JavaScript), which also uses a single threaded approach
> to "concurrency" (psuedo-concurrency, quasi-concurrency, whatever).
> In Node.js, you could use callbacks, but there are libraries that
> can wrap your code to make is seem more simple and parallel. For
> example, see async.js. Here are a few quick examples from the
> async.js Web page (https://www.npmjs.com/package/async):
>
> async.map([`file1`,`file2`,`file3`], fs.stat, function(err, results){
> // results is now an array of stats for each file
> });
>
> async.filter([`file1`,`file2`,`file3`], fs.exists, function(results){
> // results now equals an array of the existing files
> });
>
> async.parallel([
> function(){ ... },
> function(){ ... }
> ], callback);
>
> async.series([
> function(){ ... },
> function(){ ... }
> ]);
>
> As a bit of justification for this question, the main intension of
> the Python ioloop support in tornado and asyncio is to address
> problems of network and I/O delays and to enable our code to do
> something useful during those delays. That seems like a central
> concern for users of ZeroMQ.
>
> Dave
>
>
> On Sat, Mar 12, 2016 at 03:00:58AM +0200, Doron Somech wrote:
> >Hi All,
> >During the ZeroMQ hackathon I have added the Radio/Dish pattern
> (thread
> >safe version of pubsub) and multicast support (only for Radio/Dish
> >sockets) to both libzmq and czmq.
> >Radio/Dish is very similar to pub/sub with some differences:
> > * Thread safe, you can send and receive messages from multiple
> >threads, so for publisher you now don't need internal device to
> >publish from multiple threads and with subscriber you can use it
> as
> >load balancer.
> > * Only single frame can be sent
> > * Radio/Dish matching is exact matching (vs prefix of pubsub)
> > * Group (Radio/Dish topic) is string and currently limited to 15
> >chars (might be increased in the future)
> > * Group is set as part of the zmq_msg (libzmq) or zframe (czmq).
> > * You call join or leave on the socket (with methods instead of
> >socket options)
> >
> >To learn how to use it with czmq take a loot at the following gist:
> >[1]https://gist.github.com/somdoron/9423196a228775c8f5af
> >For libzmq take a
Re: [zeromq-dev] odd error from pyzmq
Typically, with bind you want an IP address. Often a domain works, but it requires that zeromq can resolve that domain to a *local* IP address of the machine. This won’t work, for instance, if your raspberry pi is behind a router. You can listen on all IPs by using 'tcp://*:10011'. -MinRK On Fri, Feb 26, 2016 at 1:11 AM, Jerry Scharf < sch...@lagunawayconsulting.com> wrote: Hi, > > I'm working further into things. I have the raspberry pi side working > happily and am now trying to get the ubuntu master side working. I am > running python 3.4 and did the install from pip with no errors. (I am > running python2.7 on the pi side to get spi routines...) > > The socket create works fine, but wen I try to bind to my host as a > fully qualified hostname:port (in this case > "bank0pi.lt.in.finsix.com:10011" I get this stackdump: > > File "/home/jerry/finsix/marathon/master/simple/m_zmq.py", line 29, in > bank_setup > evsock.bind(evconn) >File "zmq/backend/cython/socket.pyx", line 487, in > zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5156) > File "zmq/backend/cython/checkrc.pxd", line 25, in > zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7535) > zmq.error.ZMQError: No such device > > What device is it looking for?? I check with host and the name resolves > fine. > > tia, > jerry > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Defaulting to tweetnacl?
I think tweetnacl by default makes sense. On Wed, Feb 10, 2016 at 11:40 PM, Pieter Hintjens wrote: > Hi all, > > I'd like to start moving to tweetnacl as the default when building > libzmq. This means, no separate install of libsodium, and encryption > built in by default. We can still have a --with-libsodium and > --without-curve at configure time. > > Does anyone have a problem with this? It will not change anything > significant in terms of performance nor interoperability. Just easier > builds. > > -Pieter > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] level-triggered FD
On Fri, Jan 22, 2016 at 5:30 PM, Doron Somech wrote: > the FD must be read-only, it might be possible in some OS but I won't be > portable. > > Regarding the Command FD, it must be used, otherwise the Recv/Send FD > won't work. > > So in your case you need to be add the event-loop both the command FD > (which is the regular FD) and Recv/Send FD. > > When command FD is signaled you must call zmq_process_comands, which > currently doesn't exist. > When recv/send FD is signaled you can call recv/send. > zmq_process_command it what causing the other FDs to get signaled. > > The bottom line, this is kind of syntactic sugar, it will be the > equivalent of calling has_in or has_out immediately after FD is signaled > and only then call recv/send. > Gotcha, thanks for the explanation. I think this will still be a huge improvement. -MinRK > > > > > > On Fri, Jan 22, 2016 at 5:54 PM, MinRK wrote: > >> >> >> On Fri, Jan 22, 2016 at 4:19 PM, Doron Somech wrote: >> >>> The FD today is signalled when ever a command should be processes. What >>> we can do is split it to 3 different FD: >>> >>> * Command FD : The one being used right now, this still must be used, >>> when ever signalled call process commands (which we should expose in API). >>> * Recv FD: use as level triggered to receive. >>> * Send FD: use as level triggered to send. >>> >>> Only issue with this solution, you should include in your event loop >>> minimum two FD, one for processing commands and one for send/ recv. >>> >> I think two FDs would be fine; certainly better than what we have now. It >> would eliminate the significant problem of one signal for separate events. >> Perhaps this is a naïve question: Is it not possible to have an FD signal >> writable when the socket becomes writable and readable when the socket >> becomes readable? If they both have to be read-only FDs, that seems fine, >> as long as the signaling for send and recv are separated somehow. I'm not >> sure what users would do with the Command FD. >> >> -MinRK >> >> >>> For thread safe sockets this is a little simpler as we can make one FD >>> for all sockets for processing commands. >>> On Jan 22, 2016 2:52 PM, "Pieter Hintjens" wrote: >>> >>>> Yes, the edge triggered FD in libzmq has been a constant source of >>>> annoyance. Maybe someone on this list knows how to fix it. >>>> >>>> On Fri, Jan 22, 2016 at 1:40 PM, MinRK wrote: >>>> > Hi all, >>>> > >>>> > I've implemented yet another eventloop integration in pyzmq (asyncio, >>>> this >>>> > time), and this is only nontrivial because of the edge-triggered >>>> read-only >>>> > zmq.FD. Integrating into existing eventloops would be much easier if >>>> we had >>>> > a more traditional level-triggered FD to work with. >>>> > >>>> > Is there a technical reason why we can't add a zmq.LEVEL_FD that would >>>> > behave in a more conventional manner: >>>> > >>>> > - level-triggered >>>> > - signal write when socket is writable >>>> > - signal read when socket is readable >>>> > >>>> > I would work on this myself, but unfortunately I don't think I have >>>> the >>>> > relevant expertise. >>>> > >>>> > -MinRK >>>> > >>>> > ___ >>>> > zeromq-dev mailing list >>>> > zeromq-dev@lists.zeromq.org >>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> > >>>> ___ >>>> zeromq-dev mailing list >>>> zeromq-dev@lists.zeromq.org >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>> >>> ___ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] level-triggered FD
On Fri, Jan 22, 2016 at 4:19 PM, Doron Somech wrote: > The FD today is signalled when ever a command should be processes. What we > can do is split it to 3 different FD: > > * Command FD : The one being used right now, this still must be used, when > ever signalled call process commands (which we should expose in API). > * Recv FD: use as level triggered to receive. > * Send FD: use as level triggered to send. > > Only issue with this solution, you should include in your event loop > minimum two FD, one for processing commands and one for send/ recv. > I think two FDs would be fine; certainly better than what we have now. It would eliminate the significant problem of one signal for separate events. Perhaps this is a naïve question: Is it not possible to have an FD signal writable when the socket becomes writable and readable when the socket becomes readable? If they both have to be read-only FDs, that seems fine, as long as the signaling for send and recv are separated somehow. I'm not sure what users would do with the Command FD. -MinRK > For thread safe sockets this is a little simpler as we can make one FD for > all sockets for processing commands. > On Jan 22, 2016 2:52 PM, "Pieter Hintjens" wrote: > >> Yes, the edge triggered FD in libzmq has been a constant source of >> annoyance. Maybe someone on this list knows how to fix it. >> >> On Fri, Jan 22, 2016 at 1:40 PM, MinRK wrote: >> > Hi all, >> > >> > I've implemented yet another eventloop integration in pyzmq (asyncio, >> this >> > time), and this is only nontrivial because of the edge-triggered >> read-only >> > zmq.FD. Integrating into existing eventloops would be much easier if we >> had >> > a more traditional level-triggered FD to work with. >> > >> > Is there a technical reason why we can't add a zmq.LEVEL_FD that would >> > behave in a more conventional manner: >> > >> > - level-triggered >> > - signal write when socket is writable >> > - signal read when socket is readable >> > >> > I would work on this myself, but unfortunately I don't think I have the >> > relevant expertise. >> > >> > -MinRK >> > >> > ___ >> > zeromq-dev mailing list >> > zeromq-dev@lists.zeromq.org >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] level-triggered FD
Hi all, I've implemented yet another eventloop integration in pyzmq (asyncio, this time), and this is only nontrivial because of the edge-triggered read-only zmq.FD. Integrating into existing eventloops would be much easier if we had a more traditional level-triggered FD to work with. Is there a technical reason why we can't add a zmq.LEVEL_FD that would behave in a more conventional manner: - level-triggered - signal write when socket is writable - signal read when socket is readable I would work on this myself, but unfortunately I don't think I have the relevant expertise. -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ 4.1.4 stable is now available
Am I right in understanding that this means that zeromq-4.1.5 won't support an OS version that zeromq-4.1.2 supports? Does that seem like a problem to anyone else? Seems like the `if_nametoindex` patch should be reverted on the 4.1 series. -MinRK On Wed, Jan 13, 2016 at 12:15 PM, Sergei Nikulov wrote: > 2016-01-13 13:34 GMT+03:00 James Chapman : > > Hi Pieter > > > > I've just tried to build 4.1.4 on Windows using VS2015. The build > > fails with the following compilation error: > > > > tcp_address.cpp(440): error C3861: 'if_nametoindex': identifier not found > > > > I should also add that the VS projects on 4.1 branch do not build > > without a lot of tweaking. I've taken to ignoring the packaged > > solution files and just using my own. > > > > James > > > > > > It seems that it was already been fixed on master 4.1 branch. > And it was bumped support from XP to Vista in upcoming 4.1.5 because > of this change 849e5b07d982e7c0917a13d9ff22d447817dd7fe (support for > IPv6 link local addresses). > > -- > Best Regards, > Sergei Nikulov > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ 4.1.4 stable is now available
Thanks, I only looked in the repo. On Mon, Jan 18, 2016 at 11:17 AM, Pieter Hintjens wrote: > Ah, I missed your patches. Merged it now. > > The release process for tarballs is documented here as it has for a > while: http://zeromq.org/docs:distributions > > For testing on Windows, I believe there's an appveyor CI running. > Though probably not testing the tweetnacl integration. > > > On Mon, Jan 18, 2016 at 11:10 AM, MinRK wrote: > > I added it to Makefile.am upstream, and opened a PR backporting that and > the > > Windows support to 4.1. But I wasn't able to verify that it actually > fixes > > the problem, since the release process appears to be undocumented. > > > > -MinRK > > > > On Mon, Jan 18, 2016 at 11:03 AM, Pieter Hintjens wrote: > >> > >> Not packaging tweetnacl was an oversight; someone added it to the > >> project and did not update Makefile.am. I will make this change. I > >> think it should also go into src/foreign rather than into the root > >> directory. > >> > >> On Mon, Jan 18, 2016 at 10:52 AM, MinRK wrote: > >> > I noticed that tweetnacl isn't included in the releases. Is this > >> > intentional? > >> > > >> > How are the tarballs made? I started trying to fix this, but didn't > find > >> > any > >> > documentation for how zeromq is released, and `make dist` seems to > fail > >> > due > >> > to missing Makefile sources in `build/cmake`. > >> > > >> > -MinRK > >> > > >> > On Wed, Jan 13, 2016 at 12:15 PM, Sergei Nikulov > >> > > >> > wrote: > >> >> > >> >> 2016-01-13 13:34 GMT+03:00 James Chapman : > >> >> > Hi Pieter > >> >> > > >> >> > I've just tried to build 4.1.4 on Windows using VS2015. The build > >> >> > fails with the following compilation error: > >> >> > > >> >> > tcp_address.cpp(440): error C3861: 'if_nametoindex': identifier not > >> >> > found > >> >> > > >> >> > I should also add that the VS projects on 4.1 branch do not build > >> >> > without a lot of tweaking. I've taken to ignoring the packaged > >> >> > solution files and just using my own. > >> >> > > >> >> > James > >> >> > > >> >> > > >> >> > >> >> It seems that it was already been fixed on master 4.1 branch. > >> >> And it was bumped support from XP to Vista in upcoming 4.1.5 because > >> >> of this change 849e5b07d982e7c0917a13d9ff22d447817dd7fe (support for > >> >> IPv6 link local addresses). > >> >> > >> >> -- > >> >> Best Regards, > >> >> Sergei Nikulov > >> >> ___ > >> >> zeromq-dev mailing list > >> >> zeromq-dev@lists.zeromq.org > >> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >> > > >> > > >> > > >> > ___ > >> > zeromq-dev mailing list > >> > zeromq-dev@lists.zeromq.org > >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >> > > >> ___ > >> zeromq-dev mailing list > >> zeromq-dev@lists.zeromq.org > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ 4.1.4 stable is now available
I added it to Makefile.am upstream <https://github.com/zeromq/libzmq/pull/1707>, and opened a PR <https://github.com/zeromq/zeromq4-1/pull/88> backporting that and the Windows support to 4.1. But I wasn't able to verify that it actually fixes the problem, since the release process appears to be undocumented. -MinRK On Mon, Jan 18, 2016 at 11:03 AM, Pieter Hintjens wrote: > Not packaging tweetnacl was an oversight; someone added it to the > project and did not update Makefile.am. I will make this change. I > think it should also go into src/foreign rather than into the root > directory. > > On Mon, Jan 18, 2016 at 10:52 AM, MinRK wrote: > > I noticed that tweetnacl isn't included in the releases. Is this > > intentional? > > > > How are the tarballs made? I started trying to fix this, but didn't find > any > > documentation for how zeromq is released, and `make dist` seems to fail > due > > to missing Makefile sources in `build/cmake`. > > > > -MinRK > > > > On Wed, Jan 13, 2016 at 12:15 PM, Sergei Nikulov < > sergey.niku...@gmail.com> > > wrote: > >> > >> 2016-01-13 13:34 GMT+03:00 James Chapman : > >> > Hi Pieter > >> > > >> > I've just tried to build 4.1.4 on Windows using VS2015. The build > >> > fails with the following compilation error: > >> > > >> > tcp_address.cpp(440): error C3861: 'if_nametoindex': identifier not > >> > found > >> > > >> > I should also add that the VS projects on 4.1 branch do not build > >> > without a lot of tweaking. I've taken to ignoring the packaged > >> > solution files and just using my own. > >> > > >> > James > >> > > >> > > >> > >> It seems that it was already been fixed on master 4.1 branch. > >> And it was bumped support from XP to Vista in upcoming 4.1.5 because > >> of this change 849e5b07d982e7c0917a13d9ff22d447817dd7fe (support for > >> IPv6 link local addresses). > >> > >> -- > >> Best Regards, > >> Sergei Nikulov > >> ___ > >> zeromq-dev mailing list > >> zeromq-dev@lists.zeromq.org > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ 4.1.4 stable is now available
I noticed that tweetnacl isn't included in the releases. Is this intentional? How are the tarballs made? I started trying to fix this, but didn't find any documentation for how zeromq is released, and `make dist` seems to fail due to missing Makefile sources in `build/cmake`. -MinRK On Wed, Jan 13, 2016 at 12:15 PM, Sergei Nikulov wrote: > 2016-01-13 13:34 GMT+03:00 James Chapman : > > Hi Pieter > > > > I've just tried to build 4.1.4 on Windows using VS2015. The build > > fails with the following compilation error: > > > > tcp_address.cpp(440): error C3861: 'if_nametoindex': identifier not found > > > > I should also add that the VS projects on 4.1 branch do not build > > without a lot of tweaking. I've taken to ignoring the packaged > > solution files and just using my own. > > > > James > > > > > > It seems that it was already been fixed on master 4.1 branch. > And it was bumped support from XP to Vista in upcoming 4.1.5 because > of this change 849e5b07d982e7c0917a13d9ff22d447817dd7fe (support for > IPv6 link local addresses). > > -- > Best Regards, > Sergei Nikulov > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] gssapi and pyzmq
https://github.com/zeromq/pyzmq/pull/745 should add GSSAPI to pyzmq, if you want to give it a test. On Thu, Nov 5, 2015 at 3:55 PM, Tevesz Ágnes wrote: > Hi, > > I would like to ask that do you plan to expose the Kerberos security > mechanism in PyZMQ too? I know the feature is available with czmq and > libzmq supports this security mechanism with gssapi, just the python > binding is not available yet. > > Thanks and regards, > Agnes > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] [ANN] pyzmq 15 with asyncio/tornado Future support
I've just released pyzmq 15, which adds support for basic integration with Python 3's asyncio, and Future-returning Socket/Poller APIs that can be used with coroutines in both asyncio <http://pyzmq.readthedocs.org/en/latest/api/zmq.asyncio.html#module-zmq.asyncio> and tornado <http://pyzmq.readthedocs.org/en/latest/api/zmq.eventloop.future.html>. https://pyzmq.readthedocs.org/en/latest/changelog.html Wheels are building now, and should finish uploading before too long. -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Build pyzmq on Windows with VC11 and Python 2.7
On Thu, Jul 2, 2015 at 10:55 AM, Christoph Buelter wrote: > Hey MinRK, thanks for answering. > > I don't do this on a regular basis, but for I example I just tried it with > dulwich > <https://github.com/jelmer/dulwich>and that worked fine. > > Basically what I had done for pyzmq so far is: > > python setup.py configure --zmq=bundled > > It would not build the bundled libraries, but I was able to build both > libsodium and libzeromq using the provided .sln files, by adjusting its > include and library paths. > > python setup.py build --zmq=bundled > > And that is the log for that, it fails somewhere with libsodium and I have > no idea what to do to fix it: > > http://pastebin.com/v1rQ7yF7 > > Any idea or guessing what I am doing wrong? > Afraid not, building on Windows is well outside my expertise. -MinRK > > Cheers and thanks again! > > > > Am 02.07.2015 um 00:10 schrieb MinRK: > > Can you build other Python extensions? Is PyZMQ the only one that fails? > > On Wed, Jul 1, 2015 at 2:51 PM, Christoph Buelter < > c.buel...@arcor.de> wrote: > >> Hi, >> >> has anyone yet managed to build Windows with VC11 and Python 2.7.3? >> I am having all kinds of problems. The default Python 2.7.3 has been >> compiled with VC9, but I am using a custom version that was created with >> VC11 so I need pyzmq for that version. >> Anyone ever done it? >> >> Cheers >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > > > ___ > zeromq-dev mailing > listzeromq-dev@lists.zeromq.orghttp://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Build pyzmq on Windows with VC11 and Python 2.7
Can you build other Python extensions? Is PyZMQ the only one that fails? On Wed, Jul 1, 2015 at 2:51 PM, Christoph Buelter wrote: > Hi, > > has anyone yet managed to build Windows with VC11 and Python 2.7.3? > I am having all kinds of problems. The default Python 2.7.3 has been > compiled with VC9, but I am using a custom version that was created with > VC11 so I need pyzmq for that version. > Anyone ever done it? > > Cheers > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Compile pyzmq on Windows
I believe you have to use VS2008 to build extensions for Python 2.7 or the Windows 7 (not 7.1) SDK, which is based on the same VC9. Microsoft released a special package <https://www.microsoft.com/en-us/download/details.aspx?id=44266> with a VC++ compiler for Python 2, that might be what you need. I think you may need a very recent Python 2.7 (.9 or .10, perhaps) for distutils to find it, though. -MinRK On Tue, Jun 23, 2015 at 2:37 PM, Christoph Buelter wrote: > Hello, > > I am trying to build pyzmq on *Windows*, Python 2.7.3 x64 with Visual > Studio 2012. > > On a side note: I tried building libzmq 1.40 on my own beforehand, but the > paths in the solutions seem to be outdated and the configure > --zmq=path/to/libzmq did not work properly, so I thought the step is > probably not even needed: As far as I understood, it already comes now with > the source code of its dependencies libsodium and libzmq. Please correct me > if that is wrong. > > So basically I downloaded the > > git clone git://github.com/zeromq/pyzmq.git > > repository and executed the following in a VS 2012 Command prompt: > > python setup.py configure python setup.py build_ext --inplace > > The output is as in the following link, it seems to have problems > compiling sodium: > > http://pastebin.com/YpaqwitT > > Any ideas or pointers? > > Cheers, > Christoph > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Passing a python object (PyZMQ)
On Mon, Jun 8, 2015 at 11:00 AM, Arnaud Loonstra wrote:
> A python dictionary is not thread safe, AFAIK. So if multiple
> producers(threads) write to the dictionary you'll run into trouble.
>
Depends what you mean. All of Python is threadsafe, thanks to the GIL.
Since object IDs are unique as long as the objects exist, you should have
no collisions putting different objects in the dictionary from different
threads. You could also key by UUID if for some reason you have multiple
threads sending the same exact object.
-MinRK
>
> Rg,
>
> Arnaud
>
> On June 8, 2015 6:05:08 PM GMT+02:00, Min RK wrote:
>
>>
>>
>> On Jun 8, 2015, at 01:35, Arnaud Loonstra wrote:
>>>
>>> On 06/06/2015 01:52 AM, MinRK wrote:
>>>> Without using ctypes, you could pass the objects through a namespace:
>>>>
>>>> |# shared namespace
>>>> ns = {}
>>>>
>>>> # sender
>>>> ns[id(obj)] = obj
>>>> pipe_out.send(struct.pack(b'Q',1))
>>>>
>>>> # receiver
>>>> id_bytes = pipe_in.recv()
>>>> obj_id = struct.unpack(b'Q', id_bytes)[0]
>>>> obj = ns.pop(obj_id)
>>>> |
>>>>
>>>> The ctypes cast approach doesn’t hold a reference to the object while
>>>> it’s in transit, so it’s possible for the restoration to fail if the
>>>> object has been garbage collected in between send/recv.
>>>>
>>>
>>>
>>> You're right I ended up doing something very similar to workaround early
>>> garbage collection but this is much nicer. However it only works between
>>> a producer and consumer thread, not more. So every producer thread needs
>>> its own namespace.
>>>
>>
>> I'm not sure why. You should only need one namespace per process.
>>
>> -MinRK
>>
>>
>>> Rg,
>>>
>>> Arnaud
>>> --
>>> w: http://www.sphaero.org
>>> t: http://twitter.com/sphaero
>>> g: http://github.com/sphaero
>>> i: freenode: sphaero_z25
>>> --
>>>
>>> zeromq-dev mailing list
>>> zeromq-dev@lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>> --
>>
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
> Send from my feature bloated phone.
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Passing a python object (PyZMQ)
Without using ctypes, you could pass the objects through a namespace:
# shared namespace
ns = {}
# sender
ns[id(obj)] = obj
pipe_out.send(struct.pack(b'Q', 1))
# receiver
id_bytes = pipe_in.recv()
obj_id = struct.unpack(b'Q', id_bytes)[0]
obj = ns.pop(obj_id)
The ctypes cast approach doesn’t hold a reference to the object while it’s
in transit, so it’s possible for the restoration to fail if the object has
been garbage collected in between send/recv.
-MinRK
On Thu, Jun 4, 2015 at 1:59 AM, Arnaud Loonstra wrote:
> On 2015-06-03 17:25, Arnaud Loonstra wrote:
> > On 2015-06-03 10:20, Arnaud Loonstra wrote:
> >> I read here: http://zeromq.github.io/pyzmq/serialization.html PyZMQ
> >> being able to pass an object without copying it.
> >> I guess this holds only for the serialised copy of the data..? In
> >> theory the object can be passed as a reference thus preventing a
> >> copy.
> >>
> >> Suppose I have an image as a python object which is produced in a
> >> thread. Now an other thread wants to display it. The only transport
> >> which would be able to do this is 'inproc'. However the send method
> >> of
> >> PyZMQ needs an object supporting the Buffer interface (I guess to
> >> copy
> >> the data). How would I be able to pass the image to the other thread
> >> without copying it?
> >>
> >> Rg,
> >>
> >> Arnaud
> >
> > I've found a really nasty but simple way to accomplish this
> >
> >>>> from PIL import Image
> >>>> import ctypes
> >>>> a = Image.new("RGB", (200,600), (255,255,55))
> >>>> a
> >
> >>>> id(a)
> > 140178779949936
> >>>> ctypes.cast(id(a), ctypes.py_object).value
> >
> >>>> ctypes.cast(140178779949936, ctypes.py_object).value
> >
> >>>>
> >
> > So I'm using the id() method of Python to get the pointer address
> > (integer) of the object. I send this value over the socket wher on
> > receiving I cast it bak to a python object.
> >
> > It's working. I don't if this will run into trouble with garbage
> > collection but so far so good. It probably only works in CPython.
> >
> > Rg,
> >
> > Arnaud
>
> If anybody's interested I created a proof of concept example. It kind
> of comes with a "do not try this at home" warning.
>
> https://gist.github.com/sphaero/3ff5a72e699d828306ec
>
> If anybody knows other approaches I'm very interested!
>
> Rg,
>
> Arnaud
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zproject python binding mixing with pyZMQ
On Thu, Jun 4, 2015 at 4:03 AM, Arnaud Loonstra wrote:
> On 2015-06-02 13:56, Arnaud Loonstra wrote:
> > On 2015-05-03 16:25, Arnaud Loonstra wrote:
> >> Hi all,
> >>
> >> I'm testing the new python bindings from zproject. As a test I was
> >> trying the Zyre bindings. In Zyre you can retrieve the socket using
> >> socket() in case you want to add it to a poller. However the socket
> >> method returns a type which not hashable
> >> and
> >> probably won't be understood by pyZMQ.
> >>
> >> Any ideas how to deal with this? I think it would make sense if the
> >> binding is inter operable with PyZMQ.
> >>
> >
> > To make this work the generator needs to address for the socket types
> > beginning with:
> > --- a/bindings/python/zyre.py
> > +++ b/bindings/python/zyre.py
> > @@ -45,6 +45,14 @@ zyre_p = POINTER(zyre_t)
> >
> > class zsock_t(Structure):
> > pass # Empty - only for type checking
> > +zsock_t._fields_=[
> > +("tag", c_uint),
> > +("handle", c_void_p),
> > +("endpoint", c_char_p),
> > +("cache", c_char_p),
> > +("type", c_int),
> > +("cache_size", c_size_t)
> > +]
> > zsock_p = POINTER(zsock_t)
> >
> > The handle is an actual zmq socket. We only need to get to it's file
> > descriptor to be able to use it in a select/poll.
> >
> > Perphaps the handle can be converted to a PyZMQ socket type?
>
> I might have some proof of concept code. I changed this to zyre.py:
>
> --- a/bindings/python/zyre.py
> +++ b/bindings/python/zyre.py
> @@ -44,7 +44,14 @@ class zyre_t(Structure):
> zyre_p = POINTER(zyre_t)
>
> class zsock_t(Structure):
> -pass # Empty - only for type checking
> +_fields_=[
> +("tag", c_uint),
> +("handle", c_void_p),
> +("endpoint", c_char_p),
> +("cache", c_char_p),
> +("type", c_int),
> +("cache_size", c_size_t)
> +]
> zsock_p = POINTER(zsock_t)
>
> This simple example shows getting the zyre socket as a PyZMQ socket
>
> >>> from zyre import Zyre
> >>> import zmq
> >>> z1 = Zyre('t1')
> >>> s = z1.socket()
> >>> print(s)
>
> >>> print(s.contents.endpoint)
> inproc://pipe-ee96-5a9d
> >>> sock_pointer = s.contents.handle
> >>> s2 = zmq.Socket(shadow=sock_pointer)
> >>> print(s)
>
> >>> print(s2.closed)
> False
> >>> print(s2.get_hwm())
> 1000
> >>> print(s2.underlying)
> 44483568
> >>> print(s2.get(zmq.FD))
> 9
>
> I'm not sure whether this is interchangeable with CFFI and Cython
>
Shadowing libzmq sockets with pyzmq should work fine in both CFFI and
Cython.
-MinRK
>
> Rg,
>
> Arnaud
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Building PyZmq on Mac
I think the right thing to do is to remove the support for bundling a pre-built libzmq from pyzmq. It's very complicated, rarely (if ever) used, and as you have discovered, doesn't actually work. After removing it, a bdist pointing to a specific library will leave the library alone, and delocate can take care of the rest. You can disable pyzmq's broken bundling manually right now, by making a `setup.cfg` in the pyzmq directory with: ``` [global] bundle_libzmq_dylib = False ``` At which point you can: ``` pip wheel . --build-option="--zmq=$PWD/../zeromq-4.0.4" delocate-wheel wheelhouse/pyzmq-*.whl ``` and you should be set. -MinRK On Thu, Apr 16, 2015 at 1:44 PM, Thomas Maslach wrote: > Thanks for the quick response.. > > > > Regarding bdist, you said: Does it work if you specify an absolute path? I > haven’t spent any time working on bdists other than the official ones I > use, which always build libzmq, rather than linking to an existing one. > It’s possible some args or flags should be different. > > > > It does not work with an absolute path – just tried it. To me, it looks > like the process just has a few bugs in it. Setting libzmq.dylib’s id to > @loader_path/../libzmq.dylib makes some sense, but since it is building > vers in build/temp.macosx-10.4…../scratch, it won’t find libzmq.dylib up > one level. At least, that’s my understanding on how this stuff works, but I > know I could be very wrong in this! > > > > The delocate looks like the right way to go versus running > install_name_tool many times. I will look into it, but I’m hoping that I > can get away with just using the setup script provided by pyzmq. I still > haven’t given up! > > > > Tom > > > > > > *From:* zeromq-dev-boun...@lists.zeromq.org [mailto: > zeromq-dev-boun...@lists.zeromq.org] *On Behalf Of *MinRK > *Sent:* Thursday, April 16, 2015 3:54 PM > *To:* ZeroMQ development list > *Subject:* Re: [zeromq-dev] Building PyZmq on Mac > > > > I think to properly find the libzmq.dylib during the checks when libzmq is > not installed somewhere it will be found > > On Thu, Apr 16, 2015 at 12:22 PM, Thomas Maslach wrote: > > > > I’m attempting to build pyzmq where it’ll automatically put the zeromq > libraries within in the pyzmq ‘zmq’ folder. > > > > I am using my own build of python located in my home dir. It will move > around. > > > > I first did the following: > > > > ../python27/python setup.py configure –zmq=../zeromq-4.0.4 > > ../python27/python setup.py install > > > > This installs everything, but does not put libzmq.dylib into the zmq > folder. Running otools –L libzmq.dylib also gives the id of that dylib as > the full path to zeromq-4.04 + libzmq.3.dylib. (Sorry about any confusing > terminology – my mac knowledge is slim). > > That’s correct. When you do this, you are telling pyzmq where you have > *installed* libzmq. It should be loadable from that location, and pyzmq > shouldn’t touch it. > > > > After looking on line some and at the setup.py code, I tried the following: > > > > ../python27/python setup.py configure –zmq=../zeromq-4.0.4 > > ../python27/python setup.py bdist > > > > It looks like bdist will make the bundle_libzmq_dylib property in setup.py > return True, which will copy libzmq.dylib into pyzmq’s zmq folder and call > install_name_tool to change the id of zmq. This seems to work. But, I > then get the following error in the output. It seems that it is trying to > build in the build folder, which means libzmq isn’t up a level. > > Does it work if you specify an absolute path? I haven’t spent any time > working on bdists other than the official ones I use, which always build > libzmq, rather than linking to an existing one. It’s possible some args or > flags should be different. > > > > Any idea of how I use setup.py to accomplish what I want? I just want a > pyzmq build with my zeromq linked into it, and the python folder can be > located anywhere on disk (and not necessarily on my machine). I believe I > can manually go to all the *.so files and call install_name_tool to > redirect to my libzmq. But, that doesn’t seem like a clean solution. > > There’s a great tool called delocate > <https://pypi.python.org/pypi/delocate> that automates exactly this > install_name_tool process. It may be what you are after. > > -MinRK > > > > Any help would be greatly appreciated – I’ve struggled on this for quite a > bit! > > > > Here is the output I got from running ../python27/python setup.py bdist > > > > Configure: Autodetecting ZMQ settings..
Re: [zeromq-dev] Building PyZmq on Mac
I think to properly find the libzmq.dylib during the checks when libzmq is not installed somewhere it will be found On Thu, Apr 16, 2015 at 12:22 PM, Thomas Maslach wrote: > > I’m attempting to build pyzmq where it’ll automatically put the zeromq > libraries within in the pyzmq ‘zmq’ folder. > > > > I am using my own build of python located in my home dir. It will move > around. > > > > I first did the following: > > > > ../python27/python setup.py configure –zmq=../zeromq-4.0.4 > > ../python27/python setup.py install > > > > This installs everything, but does not put libzmq.dylib into the zmq > folder. Running otools –L libzmq.dylib also gives the id of that dylib as > the full path to zeromq-4.04 + libzmq.3.dylib. (Sorry about any confusing > terminology – my mac knowledge is slim). > That’s correct. When you do this, you are telling pyzmq where you have *installed* libzmq. It should be loadable from that location, and pyzmq shouldn’t touch it. > > After looking on line some and at the setup.py code, I tried the following: > > > > ../python27/python setup.py configure –zmq=../zeromq-4.0.4 > > ../python27/python setup.py bdist > > > > It looks like bdist will make the bundle_libzmq_dylib property in setup.py > return True, which will copy libzmq.dylib into pyzmq’s zmq folder and call > install_name_tool to change the id of zmq. This seems to work. But, I > then get the following error in the output. It seems that it is trying to > build in the build folder, which means libzmq isn’t up a level. > Does it work if you specify an absolute path? I haven’t spent any time working on bdists other than the official ones I use, which always build libzmq, rather than linking to an existing one. It’s possible some args or flags should be different. > > Any idea of how I use setup.py to accomplish what I want? I just want a > pyzmq build with my zeromq linked into it, and the python folder can be > located anywhere on disk (and not necessarily on my machine). I believe I > can manually go to all the *.so files and call install_name_tool to > redirect to my libzmq. But, that doesn’t seem like a clean solution. > There’s a great tool called delocate <https://pypi.python.org/pypi/delocate> that automates exactly this install_name_tool process. It may be what you are after. -MinRK > > Any help would be greatly appreciated – I’ve struggled on this for quite a > bit! > > > > Here is the output I got from running ../python27/python setup.py bdist > > > > Configure: Autodetecting ZMQ settings... > > Custom ZMQ dir: ../zeromq-4.0.4 > > gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall > -Wstrict-prototypes -I../zeromq-4.0.4/include -Izmq/utils > -Izmq/backend/cython -Izmq/devices -c > build/temp.macosx-10.4-x86_64-2.7/scratch/vers.c -o > build/temp.macosx-10.4-x86_64-2.7/scratch/vers.o > > gcc -undefined dynamic_lookup > build/temp.macosx-10.4-x86_64-2.7/scratch/vers.o -Lzmq -lzmq -o > build/temp.macosx-10.4-x86_64-2.7/scratch/vers > > Error running version detection script: > > > > dyld: Library not loaded: @loader_path/../libzmq.dylib > > Referenced from: > /users/tom/software/ipython_mac/python/pyzmq-14.1.1/build/temp.macosx-10.4-x86_64-2.7/scratch/vers > > Reason: image not found > > > > > > error: Error running version detection script: > > > > dyld: Library not loaded: @loader_path/../libzmq.dylib > > Referenced from: > /users/tom/software/ipython_mac/python/pyzmq-14.1.1/build/temp.macosx-10.4-x86_64-2.7/scratch/vers > > Reason: image not found > > > > > > Fatal: Falling back on bundled libzmq, but setup.cfg has explicitly > prohibited building the libzmq extension. > > hwbldmac-64-02 tom /users/tom/software/ipython_mac/python/pyzmq-14.1.1 240 > > otool -L ../zeromq-4.0.4/lib/libzmq.dylib > > ../zeromq-4.0.4/lib/libzmq.dylib: > > > /users/tom/software/ipython_mac/python/zeromq-4.0.4/lib/libzmq.3.dylib > (compatibility version 5.0.0, current version 5.0.0) > > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, > current version 159.1.0) > > /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, > current version 52.0.0) > > > > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Are bindings supposed to set linger on implicit close/destroy?
(pyzmq maintainer here) I disagree with the sentiment expressed in the Guide. Either it belongs as the libzmq default behavior itself, or it doesn't. It doesn't make sense to me for language bindings to unanimously disagree with libzmq instead of changing the underlying libzmq behavior. -MinRK On Wed, Mar 18, 2015 at 9:17 PM, Dylan Cali wrote: > Hello, > > The zguide states at the end of "Making a Clean Exit": > > you need to shut down each socket that has ongoing requests. The proper >> way is to set a low LINGER value (1 second), and then close the socket. *If >> your language binding doesn't do this for you automatically when you >> destroy a context, I'd suggest sending a patch. * >> >> ... >> >> Voila! It's complex and painful enough that *any language binding author >> worth his or her salt will do this automatically and make the socket >> closing dance unnecessary.* >> > > Yet I noticed the pyzmq bindings do not seem to follow this convention and > scripts that do not explicitly set linger themselves hang. The pyzmq devs > closed this as a non-issue: > > https://github.com/zeromq/pyzmq/issues/102 > > Conversely, both the czmq and jzmq bindings do set a low linger: > > https://github.com/zeromq/czmq/issues/116 > http://git.io/hBaf > > So should this be considered a bug in pyzmq, and as a bindings author > should I follow the convention of setting a low linger? > > Thanks much, > Dylan > > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ and NORM
If you want to configure zeromq, you probably shouldn’t be using
--zmq=bundled. I would configure and install libzmq with --prefix=PREFIX,
then load it for pyzmq with setup.py install --zmq=PREFIX.
-MinRK
On Wed, Mar 11, 2015 at 11:37 AM, Michel Pelletier <
pelletier.mic...@gmail.com> wrote:
> I guess my approach doesn't help if you're looking to add special
> configure args like --with-norm, MinRK, do you know of a way to pass those
> args to the bundling configure call? A quick glance at setup.py didn't
> give me any good ideas.
>
> -Michel
>
> On Wed, Mar 11, 2015 at 10:40 AM, Michel Pelletier <
> pelletier.mic...@gmail.com> wrote:
>
>> I've found the best way to avoid these kind of problems (version mismatch
>> with existing system libzmq) is to use a virtual environment and then do a
>> bundled pyzmq build:
>>
>> $ virtualenv foo
>> $ . foo/bin/activate
>> (foo)$ pip install --install-option --zmq=bundled pyzmq
>> (foo)$ python -c 'import zmq; print zmq.zmq_version()'
>> 4.0.5
>>
>> It always makes a nice, isolated install.
>>
>> -Michel
>>
>> On Wed, Mar 11, 2015 at 10:29 AM, Brian Adamson <
>> brian.adam...@nrl.navy.mil> wrote:
>>
>>> As I think MinRF is getting at here, you need to make sure your pyzmq
>>> you are using is loading the libzmq that you built/installed with the NORM
>>> extension. On my systems, I had to download and install pyzmq from source
>>> code instead of one the prebuilt packages that assumed a dependency on an
>>> existing libzmq package instead of what you are building yourself. In a
>>> nutshell, I built (with norm) and installed the github libzmq and then
>>> separately downloaded pyzmq and used its “python setup.py install” approach
>>> to install it from source. I also had to make sure I didn’t have pyzmq
>>> installed some other way since you can end up with other packages on your
>>> systems that want to install pyzmq in a standard way since they have their
>>> own dependencies upon it.
>>>
>>> Note the current GitHub libzmq is 4.2.0. If your system is reporting
>>> version 3.2.5, it’s likely you have a conflicting (without norm and hence
>>> the error) version of libzmq installed that pyzmq is finding instead of the
>>> one you want.
>>>
>>> best regards,
>>>
>>> Brian
>>>
>>>
>>>
>>> On Mar 11, 2015, at 12:54 PM, MinRK wrote:
>>>
>>> How did you install pyzmq? What OS is this?
>>>
>>> -MinRK
>>>
>>> On Wed, Mar 11, 2015 at 7:03 AM, Adam Najman wrote:
>>>
>>>> I'm trying to establish a NORM connection using ZeroMQ as detailed here:
>>>>
>>>> http://zeromq.org/topics:norm-protocol-transport
>>>>
>>>> I've already built NORM and ZeroMQ with support for norm using
>>>> ./configure –with-norm=/path/to/norm.
>>>>
>>>> The code I'm trying is as follows:
>>>>
>>>> #Subscriberimport zmq
>>>> context = zmq.Context()
>>>> socket = context.socket(zmq.SUB)
>>>> socket.bind("norm://224.1.2.3:5556")
>>>> socket.setsockopt(zmq.SUBSCRIBE, "ZMQ-Test")while True:
>>>> string = socket.recv()
>>>> print string
>>>> #Publisherimport zmqimport time
>>>> context = zmq.Context()
>>>> socket = context.socket(zmq.PUB)
>>>> socket.connect("norm://224.1.2.3:5556")
>>>> i = 1while True:
>>>> topic = "ZMQ-Test"
>>>> message = "Hello, NORM " + str(i) + " …"
>>>> socket.send("%s %s" % (topic, message))
>>>> i += 1
>>>> time.sleep(1)
>>>>
>>>> Whenever I run either of these, I get an error message:
>>>>
>>>> Traceback (most recent call last):
>>>> File "pub.py", line 5, in
>>>> socket.connect("norm://224.1.2.3:5556")
>>>> File "zmq/backend/cython/socket.pyx", line 471, in
>>>> zmq.backend.cython.socket.Socket.connect (zmq/backend/cython/socket.c:4295)
>>>> zmq.error.ZMQError: Protocol not supported
>>>>
>>>> I've tried re-installing everything and building everything from
>>>> scratch, including NORM ZeroMQ and PyZMQ. Can anyone help with this issue?
>>>> Python Version: 2.7.9 NORM Version: 1.5b4 ZeroMQ Version: 3.2.5
>>>>
>>>> ___
>>>> zeromq-dev mailing list
>>>> zeromq-dev@lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>>
>>> ___
>>> zeromq-dev mailing list
>>> zeromq-dev@lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>>
>>> ___
>>> zeromq-dev mailing list
>>> zeromq-dev@lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ and NORM
How did you install pyzmq? What OS is this?
-MinRK
On Wed, Mar 11, 2015 at 7:03 AM, Adam Najman wrote:
> I'm trying to establish a NORM connection using ZeroMQ as detailed here:
>
> http://zeromq.org/topics:norm-protocol-transport
>
> I've already built NORM and ZeroMQ with support for norm using ./configure
> –with-norm=/path/to/norm.
>
> The code I'm trying is as follows:
>
> #Subscriberimport zmq
> context = zmq.Context()
> socket = context.socket(zmq.SUB)
> socket.bind("norm://224.1.2.3:5556")
> socket.setsockopt(zmq.SUBSCRIBE, "ZMQ-Test")while True:
> string = socket.recv()
> print string
> #Publisherimport zmqimport time
> context = zmq.Context()
> socket = context.socket(zmq.PUB)
> socket.connect("norm://224.1.2.3:5556")
> i = 1while True:
> topic = "ZMQ-Test"
> message = "Hello, NORM " + str(i) + " …"
> socket.send("%s %s" % (topic, message))
> i += 1
> time.sleep(1)
>
> Whenever I run either of these, I get an error message:
>
> Traceback (most recent call last):
> File "pub.py", line 5, in
> socket.connect("norm://224.1.2.3:5556")
> File "zmq/backend/cython/socket.pyx", line 471, in
> zmq.backend.cython.socket.Socket.connect (zmq/backend/cython/socket.c:4295)
> zmq.error.ZMQError: Protocol not supported
>
> I've tried re-installing everything and building everything from scratch,
> including NORM ZeroMQ and PyZMQ. Can anyone help with this issue? Python
> Version: 2.7.9 NORM Version: 1.5b4 ZeroMQ Version: 3.2.5
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ with Python and a basic task
Hello World is a client + server example. You need to be running the hwserver program, described above the link to hwclient in the Guide. The Python version *should* be available at http://zguide.zeromq.org/py:hwserver, but for some reason the hwserver links aren’t in the Guide. You can find it in the examples repo <https://github.com/imatix/zguide/blob/master/examples/Python/hwserver.py> in the meantime. -MinRK On Sat, Mar 7, 2015 at 8:47 PM, Michael Cuggy wrote: > Hello, > > "On your example, are you running the requisite request receiver, eg > the python script that binds to port in order that the server can > have something to connect to?" > > Yes, I think so. Can you help me with this as I seem to have made a > mistake? > > To be honest, I thought it was supposed to run by itself with no other > server: > > http://zguide.zeromq.org/py:hwclient > > The loop back address in the code snippet above made me think it could > be a standalone test. I have a pair of Linux servers. > > One server has this: > http://pastebin.com/nGc36Dq1 > > A second server has this: > http://pastebin.com/EVzCSKMn > > When I run both, I only see this: > > Connecting to hello world server... > Sending request 0 ... > > It just hangs forever. I don't think there is a security restriction > preventing the communication.* But the communication never happens, > so I don't know what is wrong. > > thanks, > > Mike > > * I ran an nmap command to test connectivity on port on both > servers. The results indicate that the port isn't being blocked. It > says "closed" and not "filtered." Filtered would indicate that the > nmap utility's probe was being dropped. A probe being dropped means > that a firewall is blocking connectivity on that port. This is not > the case. "Closed" indicates a lack of usage but not firewall > blockage. > > I therefore don't understand how to get ZeroMQ working using the > Python code on zeromoq.org. > > > On 3/1/15, Kenneth Adam Miller wrote: > > On your example, are you running the requisite request receiver, eg the > > python script that binds to port in order that the server can have > > something to connect to? Just checking. > > > > Well it depends on what all you want it to do in what scenario. If these > > are two typical user machines, you might need tcp hole punching to get > past > > firewalls to start with. There's libraries for that, and I'll probably go > > through that soon for my application. > > > > Also, you don't have to start from scratch, you could easily work from > one > > of the plethora of existing messaging clients and extend them, or at > least > > read their source to see how they work. > > > > On Mon, Mar 2, 2015 at 12:29 AM, Michael Cuggy wrote: > > > >> Hello, > >> > >> I am using RHEL v. 7.x on AWS. I installed ZeroMQ 4.0.5. I want to > >> create a basic program using Python and ZeroMQ that serves as a chat > >> service for human users using Linux on a LAN. > >> > >> I have made a significant effort. I installed the prerequisites for > >> ZeroMQ and ZeroMQ itself. I installed the Python bindings. > >> > >> The problem is that I cannot even get a Hello World program to work on > >> one server. But I used the Hello World program that was on the > >> zeromq.org page. (http://zguide.zeromq.org/py:hwclient) > >> > >> Here is what I get (after running python hello.py): > >> > >> "Connecting to hello world server > >> Sending request 0" > >> > >> It pauses indefinitely after this. I have to escape out with Ctrl-c. > >> > >> Can I get step-by-step directions on how to create a messaging program > >> using Python and Zeromq? I just want it to work between two Linux > >> machines to start with. It is just problem after problem. If no one > >> knows of such documentation, can someone please tell me why I cannot > >> see "Hello World" via the Python program that ZeroMQ.org provides? To > >> me it just > >> pauses after "Sending request 0." > >> > >> thanks, > >> > >> Mike > >> ___ > >> zeromq-dev mailing list > >> zeromq-dev@lists.zeromq.org > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >> > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ with Python and a basic task
Can you provide the complete output of installing pyzmq? And what happens when you do `python -c import zmq`? -MinRK On Sat, Mar 7, 2015 at 3:28 PM, Michael Cuggy wrote: > I am trying to run the Hello World python program taken from this link: > > http://zguide.zeromq.org/py:hwclient > > I cannot get it to run. I get an error about the import zeromq line. > I lifted the code verbatim. Maybe I installed ZeroMQ incorrectly? > > I am using Python 2.6.6 and RedHat Linux version 7. I installed ZeroMQ > by I running these commands as root: > > tar -xvf zeromq-4.0.5.tar.gz > > cd zeromq-4.0.5 > > ./configure > > make > > make install > > ldconfig > > yum install python-devel > > easy_install pyzmq > > When I try to run Hello.py I get this error: > > Traceback (most recent call last): File "Hello.py", line 7, in import > zmq ImportError: No module named zmq > > How do I get the module zmq installed? pip install pyzmq indicates > that the requirement has already been satisfied. python -v foo.py > didn't show me anything meaningful. > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
On Sun, Feb 8, 2015 at 1:14 AM, Pieter Hintjens wrote:
> On Sun, Feb 8, 2015 at 12:53 AM, MinRK wrote:
>
> > I guess the problems I identified that it solves weren't really problems,
> > then.
>
> Where in the email below are you identifying the problems that the
> experimental split of multipart messages into labels + parts was
> trying to solve?
>
Sorry, on re-reading, I guess there was more information in my head than in
my email. I shouldn't have sent that.
Status quo problems, prompting the CLIENT/SERVER proposal:
Problem: multi-part messages cannot be sent in a single atomic API call
Problem: all ROUTER messages are multi-part because the first frame is
needed for routing
Problem: using multi-part message frames for both content and routing is
confusing
Proposed CLIENT/SERVER solves the above by:
- msg.routing_id separates routing and content
- eliminate multi-part
Problems with the proposal:
Problem: single routing_id doesn't support multi-hop use cases
Problem: eliminating multi-part eliminates zero-copy framing
These are the identified problems the idea aimed to address. Of course,
that still doesn't mean it's a good idea, so I'm happy to let it die.
-MinRK
>
> -Pieter
>
> > If I recall correctly, libzmq-3.0 (or was it the old 4.0 experimental
> > branch?) preserved multi-hop while separating routing from content by
> > making the routing id a sequence rather than a single value. That way,
> > it seems that the same push/pop behavior that happens on a ROUTER-DEALER
> > device today could still work on CLIENT-SERVER. The push/pop would be
> > happening on the routing stack instead of message content.
> >
> > If a message were something like:
> >
> > {
> > frame_t frames[];
> >int nframes;
> >routing_id_t route[];
> >int nroutes;
> > }
> >
> > it would seem that you could have multi-part content (selfishly, in the
> > way that affects me - messages composed of non-contiguous memory),
> > multi-hop routing, and send it all with a single `zmq_send_newfangled`
> > call, allowing future attempts at making `zmq_send_newfangled` a
> > threadsafe call.
> >
> > There may be reasons this would be super gross and horrible, but it's an
> > idea, anyway.
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
On Sat, Feb 7, 2015 at 12:02 AM, Pieter Hintjens wrote: > On Fri, Feb 6, 2015 at 9:28 PM, MinRK wrote: > > > There may be reasons this would be super gross and horrible, but it's an > > idea, anyway. > > It didn't solve any identifiable problem, and forced every proxy loop > to become two loops. If multipart is nasty because it makes the > concept of "message" more complex, then this was doubly horrid. > I guess the problems I identified that it solves weren't really problems, then. > > Separating the routing stack from the message would probably be neat > if the result was an atomic message, and proper semantics for using > that routing stack (a "reply" method, for instance). > > -Pieter > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
On Fri, Feb 6, 2015 at 1:08 PM, Thomas Rodgers
wrote:
> libzmq has had io_vec like send/receive for a while (zmq_sendiov,
> zmq_recviov), there is no documentation for them however.
>
Yes, though with it's current implementation, using sendiov is the same as
several calls to zmq_send(..., SNDMORE), making it no better or worse than
making those several calls yourself.
-MinRK
>
> On Fri, Feb 6, 2015 at 2:40 PM, Michel Pelletier <
> pelletier.mic...@gmail.com> wrote:
>
>> Something else that occurred to me today, perhaps incorrectly, is that
>> there was mention that nanomsg doesn't have multipart messages, but it does
>> still preserve multipart messages in a way, but supporting scatter and
>> gather io vectors. Perhaps emulating this well known API pattern is the
>> right approach to moving forward that still keep everyone happy and brings
>> us closer to the ability to have thread safety.
>>
>> -Michel
>>
>> On Fri, Feb 6, 2015 at 12:28 PM, MinRK wrote:
>>
>>> If I recall correctly, libzmq-3.0 (or was it the old 4.0 experimental
>>> branch?) preserved multi-hop while separating routing from content by
>>> making the routing id a sequence rather than a single value. That way, it
>>> seems that the same push/pop behavior that happens on a ROUTER-DEALER
>>> device today could still work on CLIENT-SERVER. The push/pop would be
>>> happening on the routing stack instead of message content.
>>>
>>> If a message were something like:
>>>
>>> {
>>>frame_t frames[];
>>>int nframes;
>>>routing_id_t route[];
>>>int nroutes;
>>> }
>>>
>>> it would seem that you could have multi-part content (selfishly, in the
>>> way that affects me - messages composed of non-contiguous memory),
>>> multi-hop routing, and send it all with a single `zmq_send_newfangled`
>>> call, allowing future attempts at making `zmq_send_newfangled` a threadsafe
>>> call.
>>>
>>> There may be reasons this would be super gross and horrible, but it's an
>>> idea, anyway.
>>>
>>> -MinRK
>>>
>>>
>>> On Fri, Feb 6, 2015 at 9:02 AM, Thomas Rodgers
>>> wrote:
>>>
>>>> Adding a mutex, even one that is never contended, to the socket will
>>>>> essentially triple this (one atomic CAS to acquire the mutex, one atomic
>>>>> CAS to put the message on the pipe, one atomic CAS to release the mutex).
>>>>
>>>>
>>>> This is a bit of a "blue sky" view of the cost of acquiring a mutex.
>>>> For the adventurous of spirit, chase down the call path of pthread_mutex
>>>> sometime in GLIBC. It is substantially more involved than a single pair of
>>>> 'lock; cmpxchg' instructions, but it tries really hard to make that the
>>>> rough cost of the happy path.
>>>>
>>>> On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers
>>>> wrote:
>>>>
>>>>> Having thought about this for a couple of more days, I want to at
>>>>> least take a stab at arguing against "threadsafe" sockets -
>>>>>
>>>>> libzmq's thread safety guarantees, to me anyway, are very clear,
>>>>> unsurprising and non-controversial - I cannot share a socket with another
>>>>> thread without a full fence.
>>>>>
>>>>> The kinds of systems I generally build have very strict requirements
>>>>> on overall latency, to the point that most of my networking IO is done
>>>>> through kernel-bypass libraries and NICs that support this, for raw TCP
>>>>> and
>>>>> UDP multicast. The latency sensitive code that does IO is in it's own
>>>>> thread, with exclusive access to the NICs which are accessed via kernel
>>>>> bypass. Coordination with other threads in the same process is done via
>>>>> inproc pair sockets. Pair sockets + very small messages (small enough that
>>>>> libzmq does not need to perform allocation) provide a very nice interface
>>>>> to a lock free queue with low overhead using a single atomic CAS
>>>>> operation.
>>>>> Atomic operations are cheap, but they are not free (~30 clocks on x86).
>>>>> Adding a mutex, even one that is never contended, to the socket will
>>>>> essentially triple this (one atomic CAS to acquire the mutex, one atomic
>>>>> CAS to put the message on the p
Re: [zeromq-dev] Notes from a hackathon
If I recall correctly, libzmq-3.0 (or was it the old 4.0 experimental
branch?) preserved multi-hop while separating routing from content by
making the routing id a sequence rather than a single value. That way, it
seems that the same push/pop behavior that happens on a ROUTER-DEALER
device today could still work on CLIENT-SERVER. The push/pop would be
happening on the routing stack instead of message content.
If a message were something like:
{
frame_t frames[];
int nframes;
routing_id_t route[];
int nroutes;
}
it would seem that you could have multi-part content (selfishly, in the way
that affects me - messages composed of non-contiguous memory), multi-hop
routing, and send it all with a single `zmq_send_newfangled` call, allowing
future attempts at making `zmq_send_newfangled` a threadsafe call.
There may be reasons this would be super gross and horrible, but it's an
idea, anyway.
-MinRK
On Fri, Feb 6, 2015 at 9:02 AM, Thomas Rodgers
wrote:
> Adding a mutex, even one that is never contended, to the socket will
>> essentially triple this (one atomic CAS to acquire the mutex, one atomic
>> CAS to put the message on the pipe, one atomic CAS to release the mutex).
>
>
> This is a bit of a "blue sky" view of the cost of acquiring a mutex. For
> the adventurous of spirit, chase down the call path of pthread_mutex
> sometime in GLIBC. It is substantially more involved than a single pair of
> 'lock; cmpxchg' instructions, but it tries really hard to make that the
> rough cost of the happy path.
>
> On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers
> wrote:
>
>> Having thought about this for a couple of more days, I want to at least
>> take a stab at arguing against "threadsafe" sockets -
>>
>> libzmq's thread safety guarantees, to me anyway, are very clear,
>> unsurprising and non-controversial - I cannot share a socket with another
>> thread without a full fence.
>>
>> The kinds of systems I generally build have very strict requirements on
>> overall latency, to the point that most of my networking IO is done through
>> kernel-bypass libraries and NICs that support this, for raw TCP and UDP
>> multicast. The latency sensitive code that does IO is in it's own thread,
>> with exclusive access to the NICs which are accessed via kernel bypass.
>> Coordination with other threads in the same process is done via inproc pair
>> sockets. Pair sockets + very small messages (small enough that libzmq does
>> not need to perform allocation) provide a very nice interface to a lock
>> free queue with low overhead using a single atomic CAS operation. Atomic
>> operations are cheap, but they are not free (~30 clocks on x86). Adding a
>> mutex, even one that is never contended, to the socket will essentially
>> triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the
>> message on the pipe, one atomic CAS to release the mutex). I would like to
>> have the option to avoid this.
>>
>> If a wrapper wants thread safe sockets to enable certain use-cases that
>> may be more idiomatic for the language in question, it can provide the full
>> fence. AZMQ <https://github.com/zeromq/azmq> does exactly this by
>> default, but you have the option to opt out of it. It does this because
>> Boost Asio by default allows it's sockets to be used from multiple threads
>> for async IO and I need to guard more than just exclusive access to the
>> ZeroMQ socket a the fence in this case. Putting a mutex inside of the
>> libzmq socket, essentially doubles the overhead for no gain in useful
>> functionality and runs completely counter to one of C and C++'s overarching
>> principles: "don't pay for what you don't use".
>>
>> If a class of apps really demands short lived exclusive access to a
>> socket, provide a pool abstraction. The pool is thread safe, obtain a
>> socket, perform I/O in a single thread, return the socket to the pool.
>>
>> On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier <
>> pelletier.mic...@gmail.com> wrote:
>>
>>>
>>>
>>> On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens wrote:
>>>
>>>> The discussion about thread safety was quite short iirc, though that
>>>> contributor did discuss other things... at length. I merged his
>>>> "thread safe socket" change rapidly, then we reverted it after a few
>>>> days, and he disappeared. It was rather brute force and I suspect did
>>>> not work at all, it simply wrapped all accesses to the socket
>>>> structure in mutexes. No discussion at the time of multipart data and
>
Re: [zeromq-dev] Notes from a hackathon
On Tue, Feb 3, 2015 at 2:04 PM, Pieter Hintjens wrote:
> Thanks for the questions. I'd try to flatter by saying how great these
> questions are, except I know you're all way too smart to fall for
> that. :)
>
> Let me answer a bunch of questions in one go:
>
> * Sorry for not posting this here. I've written a more detailed
> analysis of the motivation for this thread (and experiments):
> http://hintjens.com/blog:84.
>
> * The main goal with this is simplicity and accuracy. I'm tired of
> teaching people ZeroMQ constructs that are clearly over-designed and
> disconnected from reality in subtle ways. Each apology for friction
> ("sorry, but don't use REQ-REP, they are kind of weird", "sorry,
> IDENTITY is not really identity", "sorry, message sometimes means
> frame and sometimes message") really signals costs up and down the
> stacks.
>
> * Sessions require protocol commands that are invisible to "normal"
> use. That means either protocol support (e.g. command frames) or some
> layering on top of ZeroMQ blobs.
>
> * Picture sending solves a cross-language problem, namely how to
> construct zproto-style commands without using dedicated codecs. While
> the API is C-ish, the functionality is not. I'd expect that to be
> wrapped in bindings.
>
> * Per-language serialization is an anti-pattern. It's common, and fine
> as such, yet ZeroMQ should IMO always strive to be cross-language.
>
> * REQ-REP between threads may make sense. We certainly use it (ZAP,
> for authentication).
>
> * Is this discussion C4 compatible? Yes, no, this is chatter, not a
> plan or roadmap. Until pull requests hit the road, it's all vapour.
> However the problems I restated in my blog article are real, to many
> people, and should be aired.
>
> * Multipart does easy serialization, yes. Not in C though. We've
> learned how to do variable list serialization with zproto-compatible
> picture sending in CZMQ. Is this a sufficient alternative? I think so.
> zsock_send() was essential to making actors cheap enough to use
> widely. egrep zsock_send *.c in malamute-core/src.
>
Perhaps it is because I spend my days in a higher level language like
Python, but zproto is not an attractive option. I will trust you that it
makes more sense when writing C or C++ libraries. I care a lot more about
multi-part content than routing frames being part of content, which I
acknowledge as a source of this confusion. I don't see anything in this
proposal that makes removing multipart *content* make any sense. Its
removal doesn't seem to provide any benefit. The new server socket removes
the *need* for it in a simple case, which I agree is a benefit, but there
seems to be no benefit to removing the capability itself.
>From a brief twitter discussion, Pieter pointed out that I might be able to
keep `send/recv_multipart` in pyzmq, and reimplement zero-copy framing with
`zmq_send`. I'm not sure how that works without SNDMORE, and I'm not sure
that I can get that and preserve access to `zmq_free_fn`, which I need to
synchronize garbage collection, but it's an interesting idea.
>
> * Locking on socketsL: we did in fact have a pull request for this
> once. It was... nasty. We reverted it. We breathed a sigh of relief.
> However looking back, the problem was valid and has never been solved
> since.
>
> * If you're doing multipart a lot in an application, you can replace
> that with zproto. You already have contracts then, why not make them
> explicit and documented? That's always a win in my experience.
>
> * Multi-hop request reply is a strange beast and I'm keen to
> understand those architectures that use it. My suspicion is that we've
> shifted our vision of what ZeroMQ "is" over the years. My own
> experience seems plausible, yet I distrust it. So, the stories in my
> article are please for falsification, not attempts to convince.
>
I use multi-hop (never more than 2-3, not generic comlicated hops), but I
had the same code working when zeromq-3.0 split routing information from
content (I think they were called 'command frames' or something?). I think
the main difference between this and the earlier attempt was the earlier
attempt used the same mechanism for subscriptions and routing frames, as
well as allowing the routing information itself to be multipart. My
multi-hop architecture is simple enough that I don't expect that I would
have difficulty moving the routing information to the application level,
instead of letting zeromq handle it.
-MinRK
>
> * I think we have learned to not break existing code. It's going to be
> interesting to run multiple models at once. We do this in CZMQ and it
> works OK. The
Re: [zeromq-dev] zmqstream with req socket only sends the first message
On Tue, Feb 3, 2015 at 7:00 AM, drebs wrote:
Thanks a lot for your answer, now it works like a charm.
>
> I couldn't find documentation about this, do you think it's a good idea to
> add
> a paragraph to zmqstream.send docstring (or anywhere else) about threaded
> use?
>
> If you think it's a good idea, I have a suggestion based on the info you
> gave
> me. It's in the patch below, but feel free to modify in the way you think
> it
> is more appropriate.
>
Since this is an issue generic to any use of tornado IOLoop, I would be a
bit more brief, and link to the tornado docs
<http://tornado.readthedocs.org/en/latest/ioloop.html#tornado.ioloop.IOLoop.add_callback>
.
-MinRK
> --
>
> From cf9418dc14b2dc2e1aca5be1b79a105982dcd877 Mon Sep 17 00:00:00 2001
> From: drebs
> Date: Tue, 3 Feb 2015 12:50:59 -0200
> Subject: [PATCH] Add doc for threaded zmqstream.send.
>
> ---
> zmq/eventloop/zmqstream.py |8
> 1 file changed, 8 insertions(+)
>
> diff --git a/zmq/eventloop/zmqstream.py b/zmq/eventloop/zmqstream.py
> index 86a97e4..59149ec 100644
> --- a/zmq/eventloop/zmqstream.py
> +++ b/zmq/eventloop/zmqstream.py
> @@ -242,6 +242,14 @@ class ZMQStream(object):
> def send(self, msg, flags=0, copy=True, track=False, callback=None):
> """Send a message, optionally also register a new callback for
> sends.
> See zmq.socket.send for details.
> +
> +When using threads, calling send on the zmqstream from another
> thread
> +doesn’t properly tell the IOLoop thread that there’s an event to
> +process. This could cause small delays if the IOLoop is already
> +processing lots of events, but it can also cause the message to
> never
> +be sent if the zmq socket is the only one it’s handling. In this
> case,
> +you should hand off the stream.send to the IOLoop’s thread via
> +IOLoop.add_callback.
> """
> return self.send_multipart([msg], flags=flags, copy=copy,
> track=track, callback=callback)
>
A
> --
> 1.7.10.4
>
>
>
> --
>
>
> MinRK codificou 9.4K bytes:
> >Calling send on the zmqstream from another thread doesn’t properly
> tell
> >the IOLoop thread that there’s an event to process. This could call
> small
> >delays if the IOLoop is already processing lots of events, but it can
> >cause the message to never send if the zmq socket is the only one it’s
> >handling.
> >
> >You want your ZmqREQConnection.send to hand off the stream.send to the
> >IOLoop’s thread via IOLoop.add_callback:
> >
> > def send(self, *args, **kwargs):
> > print("Sending message to backend: (%s, %s)" % (args, kwargs))
> > self._ioloop.add_callback(lambda : self._stream.send(*args,
> **kwargs))
> >
> >-MinRK
> >
> >On Mon, Feb 2, 2015 at 12:58 PM, drebs [1]dr...@riseup.net wrote:
> >>
> >
> > Hello, zmq community.
> >
> > I’m trying to use a pyzmq’s zmqstream with a REQ socket to talk to a
> > txzmq REP
> > socket, but only the first message is ever sent and received back.
> >
> > REQ pyzmq endpoint code: [2]http://paste.debian.net/143615/
> > REP txzmq endpoint code: [3]http://paste.debian.net/143617/
> >
> > I would expect that all the messages would be sent. Am I missing
> > something
> > here?
> >
> > If I let some requests accumulate, and then start the server, all
> the
> > accumulated requests are replied, but the new ones are not sent.
> >
> > If I replace the REQ endpoint by a pure pyzmq version with no
> zmqstream
> > or
> > ioloop, it works fine ([4]http://paste.debian.net/143619/).
> >
> > Anyone can give some light on this?
> >
> > Thanks a lot!
> > drebs.
> >
> > —
> >
> >
> --
> >
> > zeromq-dev mailing list
> > [5]zeromq-dev@lists.zeromq.org
> > [6]http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
> >
> > References
> >
> >Visible links
> >1. http://mailto:dr...@riseup.net/
> >2. http://paste.debian.net/143615/
> >3. http://paste.debian.net/143617/
> >4. http://paste.debian.net/143619/
> >5. mailto:zeromq-dev@lists.zeromq.org
> >6. http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
I happily use multipart rather a lot, to the point that I recommend that pyzmq users exclusively use send/recv_multipart, and never single-frame send/recv. I think it enables a really nice API. I find it very useful to be able to send a large buffer plus a header of message information without having to copy both blocks into a new buffer before handing off to zeromq. I find this particularly valuable in Python, where building that new buffer can be quite expensive. In Python, and I suspect other higher level wrappers, where most messages are bytes or buffer objects (not wrapped in Message APIs), moving the routing information to message properties is not really an improvement, because I'm not aware of any pyzmq users accessing message properties, so the standard way to use pyzmq is to discard the `zmq_msg_t` object and only deal with bytes. But that's more of an API question for pyzmq than libzmq. As far as I can tell, I also use the multihop case that Pieter's blog post suggests "does not seem to happen in practice" every day, so maybe my use of zeromq is not typical. >From the "never break things" stance of libzmq and the discussion above, it seems that there's no plan for multipart to ever be removed. Deprecated even seems like too strong a word for a useful feature that will continue to be supported indefinitely. Or is anyone proposing the eventual removal of multipart? -MinRK On Tue, Feb 3, 2015 at 9:37 AM, Arnaud Kapp wrote: > Hello everyone, > > I have read this thread and the blog post from Pieter. > I am looking forward to some of those things, but I have some reserve > about other points. I'll try to explain what I think, and why. > > I agree that ZMQ_IDENTITY is a bit confusing. Allowing remote client > to set routing information is a bit counterintuitive. Also there is > risk for impersonation. If a peer disconnect, an other one could > "steal" it's identity. I believe that relying on the socket identity > to identify peer is not a robust enough approach, even when using > CURVE. > Storing the peer routing id in the message's properties seems like a good > idea. > > I think thread-safe socket would be cool. As in great to have in some > particular situation, but most of the case it shouldn't be needed. > > > Aiming for a cleaner and simpler ZMQ internal code is good. We must be > careful however that this will not make the end-user's life harder. I > believe I see 2 points that would make my life harder as a ZMQ user > today if we had what we is discussed here: > + The removal (or depreciation) of the REQ-REP socket. While > fragile over the network, I find them very useful for sending > synchronous messages between threads. > + Removal / depreciation of multipart messages. Read below. > > The next paragraph is about multipart messages. Let me be honest and > state that I have no idea of the real performance penalty induced by > multipart messages. What I want to defend here is their usefulness. > I *really* like multipart messages. To be honest, this is one of the > many features that made me try ZMQ. This is also the feature that > helps me most: I my current project, I mainly pass messages around > between thread, so I have less use for other awesome features like > auto reconnect or security. > > IMO framing is very good because it allows you to send (and receive) > various kind of data *easily*. I like doing this kind of stuff: > > zmqpp::message msg; > msg << "HELLO WORLD" << 42 << 13; > s.send(msg); > > And on the receiving end: >std::string cmd; >int value_1, value_2; >msg >> cmd >> value_1 >> value_2; > > Sure, there is no verification of the message content, the application > would even crash if a message was badly formatted. However, this is > not a concern when messages are flowing on inproc:// : I know that I > send and I know what to receive. I believe that the picture, which is > great in CZMQ, is not really appropriate for C++ and maybe other > language. My point is (for c++) that stream operator are a more > apprioriate / natural way of doing what the picture does. > > ZProto is mentioned as a possible replacement to framing. Disclaimer: > I've taken a look at zproto, but I have never used it. > I think it's a great project with some strong advantages and real > world use case. However, it comes at the cost of more complexity. This > is what I meant when talking about making the end-user's life harder. > Do I want to use zproto to design complex, client-server protocol? > Yes, I probably should give it a try. > Do I really want to use zproto when sending 3-4 frames on an inproc > socket? I'd rather
Re: [zeromq-dev] zmqstream with req socket only sends the first message
Calling send on the zmqstream from another thread doesn’t properly tell the
IOLoop thread that there’s an event to process. This could call small
delays if the IOLoop is already processing lots of events, but it can cause
the message to never send if the zmq socket is the only one it’s handling.
You want your ZmqREQConnection.send to hand off the stream.send to the
IOLoop’s thread via IOLoop.add_callback:
def send(self, *args, **kwargs):
print("Sending message to backend: (%s, %s)" % (args, kwargs))
self._ioloop.add_callback(lambda : self._stream.send(*args, **kwargs))
-MinRK
On Mon, Feb 2, 2015 at 12:58 PM, drebs dr...@riseup.net
<http://mailto:dr...@riseup.net> wrote:
>
Hello, zmq community.
I’m trying to use a pyzmq’s zmqstream with a REQ socket to talk to a txzmq
REP
socket, but only the first message is ever sent and received back.
REQ pyzmq endpoint code: http://paste.debian.net/143615/
REP txzmq endpoint code: http://paste.debian.net/143617/
I would expect that all the messages would be sent. Am I missing something
here?
If I let some requests accumulate, and then start the server, all the
accumulated requests are replied, but the new ones are not sent.
If I replace the REQ endpoint by a pure pyzmq version with no zmqstream or
ioloop, it works fine (http://paste.debian.net/143619/).
Anyone can give some light on this?
Thanks a lot!
drebs.
—
--
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Implementing zeromq security
The inproc://zeromq.zap.01 url is hardcoded in libzmq <https://github.com/zeromq/zeromq4-x/blob/v4.0.5/src/session_base.cpp#L285>. The session internally creates a client connection to that endpoint. -MinRK On Fri, Dec 12, 2014 at 12:56 PM, Riskybiz wrote: > > I’m trying to implement zeromq security for the first time and WITHOUT > using CZMQ. Reading through http://hintjens.com/blog:49 & ZAP protocol > info http://rfc.zeromq.org/spec:27 & code at > zeromq-4.0.4\tests\tests_security_curve.cpp . I have a question and hope > someone might be able to enlighten me > > > > At http://hintjens.com/blog:49 ; > > > > ‘Internally, the authenticator talks to libzmq across a protocol called > ZAP <http://rfc.zeromq.org/spec:27>, aka RFC 27. Every single time a > client tries to connect to a server, libzmq asks the ZAP handler (the > authenticator), if there is one installed, to allow or deny the connection.’ > > > > In zeromq-4.0.4\tests\tests_security_curve.cpp ; > > > > // Spawn ZAP handler > > // We create and bind ZAP socket in main thread to avoid case > > // where child thread does not start up fast enough. > > void *handler = zmq_socket (ctx, ZMQ_REP); > > assert (handler); > > rc = zmq_bind (handler, "inproc://zeromq.zap.01"); > > assert (rc == 0); > > void *zap_thread = zmq_threadstart (&zap_handler, handler); > > > > Question is; > > > > I see that the zap_handler is provided with the pointer to the REP socket > bound to "inproc://zeromq.zap.01" however I don’t understand how libzmq is > made aware to direct authentication requests to the particular handler > socket. I don’t see any further references in the code to > "inproc://zeromq.zap.01"? How does libzmq know where to send the > authentication requests? > > > > One other thing; is there any api documentation for zmq_threadstart & > zmq_threadclose? > > > > Thanks, > > > > Riskybiz. > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PyZMQ with Libsodium on Windows?
PyZMQ builds libzmq (and libsodium) as Python extensions, using the compiler associated with Python (hence the need for VC9, the compiler associated with Python.org 2.7). They are built with `setup.py build_ext`. -MinRK On Sat, Oct 11, 2014 at 1:15 PM, André Caron wrote: > Hi again! > > OK, so I managed building libsodium on Windows with MSVC 2008 to match > Python 2.7. Simply required creating a VS solution and include all the > source files. The only problems I ran into are: > 1) is not provided with MSVC 2008, so I added a stub for it > within the solution in order to build; and > 2) there are some "static inline" function declarations which are rejected > by the compiler, so I removed the "inline" (which is optional). > > So I now have a "libsodium.lib" and a "libsodium.dll". > > Once I have a proof of concept that's working inside PyZMQ, I'll see if I > can get this MSVC 2008 solution included upstream in libsodium. > > However, I'm not familiar with Python bundling process. How should I > provide the libsodium headers, lib & dll files to PyZMQ? What commands do > you run to create the PyZMQ egg and whl bundles once I've made libsodium > available to PyZMQ? > > Thanks, > > André > > On Fri, Oct 10, 2014 at 5:36 AM, Frank Hartmann wrote: > >> Hi, >> >> Tweetnacl linux integration has been added to libzmq some month ago. >> >> You could try compiling with tweetnacl and VS2008. One(only?) thing >> missing is randombytes() function on windows. >> >> Probably MS has a crypto API somewhere which should make this simple, if >> you trust them. And you can check how libsodium does it. >> >> regards >> Frank >> >> Min RK writes: >> >> > For pyzmq, it must work with VC9 (VS2008), for Python 2.7. >> > >> > -MinRK >> > >> >> On Oct 9, 2014, at 18:08, Steven McCoy wrote: >> >> >> >> I think it is easier now as they have more support than just MSVC2013 >> (C99 compat) when crypto was added to ZeroMQ. >> >> >> >>> On 9 October 2014 17:47, MinRK wrote: >> >>> It's not bundled simply because I couldn't build it on Windows. If >> you can come up with a simple fix for building bundled libsodium, then I >> would bundle libsodium on Windows. >> >>> >> >>> -MinRK >> >>> >> >>>> On Thu, Oct 9, 2014 at 1:48 PM, André Caron >> wrote: >> >>>> Hi there! >> >>>> >> >>>> I'm trying to secure some PyZMQ communications with the curve >> security features. The code seems straightforward, but when I call >> `zmq.auth.create_certificates()` to generate some keys, I get the following >> exception: >> >>>> >> >>>> zmq.error.ZMQError: Not supported >> >>>> >> >>>> This seems to be due to the absence of libsodium. >> >>>> >> >>>> In the release notes for PyZMQ 14.1.0, I can see that libsodium is >> not bundled with PyZMQ on Windows. >> >>>> >> >>>> First, I'm curious to know why it's not bundled on Windows. If it's >> simply a matter of finding a volunteer, maybe I can pitch in? >> >>>> >> >>>> Second, I'd like to know if there's any known procedure to get it >> running on Windows? >> >>>> >> >>>> Thanks! >> >>>> >> >>>> André >> >>>> >> >>>> ___ >> >>>> zeromq-dev mailing list >> >>>> zeromq-dev@lists.zeromq.org >> >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >>> >> >>> >> >>> ___ >> >>> zeromq-dev mailing list >> >>> zeromq-dev@lists.zeromq.org >> >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >> >> ___ >> >> zeromq-dev mailing list >> >> zeromq-dev@lists.zeromq.org >> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > >> > ___ >> > zeromq-dev mailing list >> > zeromq-dev@lists.zeromq.org >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZMQError: Resource temporarily unavailable
On Wed, Oct 8, 2014 at 1:11 PM, Karthik Sharma
wrote:
> I have an zmq function as below
>
> def recieve_messages(self):
> string = self.sub_socket.recv(flags=zmq.NOBLOCK)
> print('flow mod messages recieved {}'.format(string))
>
>
> When I run the program however I get the following error.
>
> string = self.sub_socket.recv(flags=zmq.NOBLOCK)
> File "socket.pyx", line 616, in zmq.core.socket.Socket.recv
> (zmq/core/socket.c:5961)
> File "socket.pyx", line 650, in zmq.core.socket.Socket.recv
> (zmq/core/socket.c:5832)
> File "socket.pyx", line 119, in zmq.core.socket._recv_copy
> (zmq/core/socket.c:1669)
> ZMQError: Resource temporarily unavailable
>
>
> Can someone explain what is likely causing this error.
>
There is no message to receive. That's what NOBLOCK means – it will always
finish immediately, either returning a message if there is one, or raising
ZMQError(EAGAIN) if there is not.
-MinRK
>
> Regards,
> Karthik.
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PyZMQ with Libsodium on Windows?
It's not bundled simply because I couldn't build it on Windows. If you can come up with a simple fix for building bundled libsodium, then I would bundle libsodium on Windows. -MinRK On Thu, Oct 9, 2014 at 1:48 PM, André Caron wrote: > Hi there! > > I'm trying to secure some PyZMQ communications with the curve security > features. The code seems straightforward, but when I call > `zmq.auth.create_certificates()` to generate some keys, I get the following > exception: > > zmq.error.ZMQError: Not supported > > This seems to be due to the absence of libsodium. > > In the release notes for PyZMQ 14.1.0, I can see that libsodium is not > bundled with PyZMQ on Windows. > > First, I'm curious to know why it's not bundled on Windows. If it's > simply a matter of finding a volunteer, maybe I can pitch in? > > Second, I'd like to know if there's any known procedure to get it running > on Windows? > > Thanks! > > André > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [ask] MonitoredQueue Prefix
On Thu, Aug 28, 2014 at 8:57 PM, bino oetomo wrote: > Dear All ... > > I'm learning about monitor device based on > > http://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/pyzmqdevices/monitorqueue.html > > My script is at http://pastebin.com/3HvdCsHC > > Basically it have 4 Process : > 1. Gate : This is the one that use MonitoredQueue. It have zmq.PULL at the > frontend, zmq.PUSH at the backend, and zmq.PUB at the monitor port > 2. monitor : this is the one that 'listen' the monitoring port of 'gate', > and just print out what it heard > 3. puller : this one just PULL data from 'gate' backend port, and just > print out what it got > 4. sender : This one just a simple payload pusher that push data to 'gate' > frontend. > > --result snipet-- > MONITOR STARTED ! > PULLER STARTED > Sender said : 1 > Monitor : ipc:///tmp/msock GOT : in > PULLER Got : 1 > Monitor : ipc:///tmp/msock GOT : 1 > Sender said : 2 > PULLER Got : 2 > Monitor : ipc:///tmp/msock GOT : in > Monitor : ipc:///tmp/msock GOT : 2 > Sender said : 3 > PULLER Got : 3 > Monitor : ipc:///tmp/msock GOT : in > Monitor : ipc:///tmp/msock GOT : 3 > - > Longer result at http://pastebin.com/7WZ1PW0U > > Looks like that monitor client always got 2 messages ... thats is : > a. 'in prefix' > b. real payload > It's not two messages, it is an extra frame on the front of a single multipart message, similar to the IDENTITY frame on a ROUTER socket, or the optional topic frame on PUB. > > My question is : Where and how to set so that monitor client only got 'the > real payload' and get rid of 'prefixes' ? > I think you might want to use the vanilla Proxy instead of MonitoredQueue. MQ was added to pyzmq a while before zmq_proxy was added to libzmq, and it does a few things that the later proxy doesn't: 1. it adds a prefix frame, so monitors can identify which socket in the device received the message 2. it supports ROUTER-ROUTER devices, which can be used as multiplexers If you aren't interested in either one of those, then the plain proxy that ships with libzmq is probably what you are after. -MinRK > > Note : my pyzmq version is '14.1.1' > > Sincerely > -bino- > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq 14.3.1 test failure
Can you run any/all of the scripts in examples/security?
What do you get from `zmq.curve_keypair()`?
On Tue, Jul 29, 2014 at 6:52 AM, Greg Ward wrote:
> Hey folks --
>
> I'm trying to build pyzmq 14.3.1, but getting a couple of test
> failures. One is intermittent, but the other seems to happen every
> time (except when the tests crash early because of the intermittent
> failure).
>
> This is on Scientific Linux 6.5, 64-bit Intel, using the OS RPM
> python-2.6.6-52.el6.x86_64 and a locally built zeromq 4.0.4 RPM.
>
> I don't need a complex build environment to reproduce this, just:
>
> $ tar -xzf pyzmq-14.3.1.tar.gz
> $ cd pyzmq-14.3.1
> $ python setup.py build_ext --inplace
> $ python setup.py test
>
> Here is the failure that happens every time:
>
> '''
> threaded auth - NULL ... FAIL
>
> ==
> FAIL: threaded auth - NULL
> --
> Traceback (most recent call last):
> File "/tmp/pyzmq-14.3.1/zmq/tests/test_auth.py", line 127, in test_null
> self.assertTrue(self.can_connect(server, client))
> AssertionError:
> >> begin captured logging <<
> zmq.auth: DEBUG: auth received API command 'TERMINATE'
> - >> end captured logging << -
> '''
>
> Here is the intermittent one (seems to happen ~20% of the time):
>
> '''
> test_blacklist (zmq.tests.test_auth.TestIOLoopAuthentication) ... ERROR
>
> ==
> ERROR: test_blacklist (zmq.tests.test_auth.TestIOLoopAuthentication)
> --
> Traceback (most recent call last):
> File "/tmp/pyzmq-14.3.1/zmq/tests/test_auth.py", line 292, in tearDown
> super(TestIOLoopAuthentication, self).tearDown()
> File "/tmp/pyzmq-14.3.1/zmq/tests/test_auth.py", line 42, in tearDown
> super(BaseAuthTestCase, self).tearDown()
> File "/tmp/pyzmq-14.3.1/zmq/tests/__init__.py", line 103, in tearDown
> raise RuntimeError("context could not terminate, open sockets likely
> remain in test")
> RuntimeError: context could not terminate, open sockets likely remain in
> test
> >> begin captured logging <<
> zmq.auth: DEBUG: version: '1.0', request_id: '1', domain: u'global',
> address: u'127.0.0.1', identity: '', mechanism: 'NULL'
> zmq.auth: DEBUG: DENIED (blacklist) address=127.0.0.1
> zmq.auth: DEBUG: ZAP reply code=400 text=Address is blacklisted
> - >> end captured logging << -
> '''
>
> Any ideas what's going wrong here? Thanks --
>
> Greg
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PeriodicCallback timeout?
Why not just queue sends directly? What is the goal of the Queue? For threadsafety, you should use the `add_callback` functionality of IOLoop: loop = IOLoop.instance() loop.add_callback(lambda : stream.send_multipart(msg)) Then you don't need any polling, it will just send messages when there are messages to be sent. -MinRK On Thu, Jul 3, 2014 at 7:07 AM, KIU Shueng Chuan wrote: > How about replacing the Queue with zmq sockets? Then you could just add > the "consumer" socket to the event loop. > On 3 Jul 2014 16:15, "Indradhanush Gupta" > wrote: > >> Hello, >> >> I am using a PeriodicCallback in pyzmq, that checks a python Queue for >> data that needs to be sent. >> >> It looks like this - >> >> self.to_client = Queue.Queue() >> >> def check_to_client(self): >> while not self.to_client.empty(): >> data = self.to_client.get() >> self.frontend.send(data) >> >> So, going by the current callback definition, it sends each element of >> the Queue until it is empty. I am wondering if this is good. It appears, >> that this callback might block if the Queue is very large. >> >> I have two approaches in mind. >> >> 1. Send n items, then return, where n = a number that will not block. So >> is there an idea on what might be a good number? >> >> 2. Use a timeout on check_to_client, such that it runs for n milliseconds >> and returns. Again what would be a good enough number for this? But also, >> is this even possible? >> >> Any other solution that someone has used before? This appears to be a >> very common problem. >> >> Thanks, >> -- >> Indradhanush Gupta >> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_ctx_term vs zmq_ctx_destroy vs zmq_ctx_shutdown
On Sun, Jun 15, 2014 at 11:55 AM, Indradhanush Gupta < indradhanush.gu...@gmail.com> wrote: > > > > On Mon, Jun 16, 2014 at 12:17 AM, MinRK wrote: > >> To add further confusion, PyZMQ distinguishes term from destroy (pyzmq >> used the term ‘destroy’ before zmq did). >> >> In pyzmq, ctx.term just calls the underlying libzmq zmq_ctx_term (née >> zmq_term) function, which blocks until all sockets are closed. >> ctx.destroy, on the other hand, closes all sockets prior to calling >> term. czmq has a similar behavior for destroy. After these destroy >> behaviors were established, libzmq added a zmq_ctx_destroy function, but >> just as a rename of zmq_term, which causes confusion like this, hence >> the change replacing zmq_ctx_destroy with the less confusing zmq_ctx_term >> . >> > But I still don't get why zmq.Context.term() blocks while > zmq.Context.destroy() returns immediately. I am stopping my reactor loop > and closing down my open sockets first. > If term blocks and destroy doesn't, that means that you still have open sockets or unsent messages with LINGER=-1. How are you closing your sockets? Can you provide a code sample that reproduces the behavior you are seeing? -MinRK > > >> -MinRK >> >> >> >> On Sun, Jun 15, 2014 at 6:03 AM, Pieter Hintjens wrote: >> >>> zmq_term/zmq_init are an older deprecated API. We switched to a more >>> consistent model for the API in 3.2, so zmq_ctx_xxx for all methods >>> that work with contexts, like zmq_msg_xxx for all methods that work on >>> messages. >>> >>> zmq_ctx_destroy was the initial choice for the termination method. >>> However people pointed out that the context isn't actually destroyed, >>> it's terminated, so we added _term() as a synonym. To be honest I'm >>> not keen on _term() as it seems inconsistent for no benefit. We >>> destroy sockets and contexts asynchronously... I'd prefer _destroy(). >>> >>> More usefully, we later added _shutdown() method that stops the >>> context but leaves it in existence; this allows a two-stage shutdown, >>> with signals being sent to all sockets waiting on blocking operations, >>> and then allowing the app to call zmq_ctx_destroy/term when wanted. >>> >>> -Pieter >>> >>> On Sat, Jun 14, 2014 at 3:34 PM, Indradhanush Gupta >>> wrote: >>> > Hello, >>> > >>> > I consulted the docs for zmq_ctx_term and zmq_ctx_destroy, in the API >>> > version 4.0 and it appears to be both have the same description. What >>> is the >>> > difference between the two? >>> > I'm using pyzmq, by the way. >>> > >>> > When I called zmq.Context.term() the call blocks indefinitely, while >>> > zmq.Context.destroy() returns immediately. I am shutting down my >>> IOLoop, >>> > then closing all open sockets by hand and only then calling one of the >>> > above. >>> > >>> > What is the difference between the two calls? Also, it appears >>> > zmq_ctx_destroy is going to be deprecated according to the 4.1 dev API >>> docs. >>> > Why is term() blocking while destroy() doesn't? >>> > >>> > I'm also confused as to when should I call shutdown, destroy or term? >>> > >>> > If it helps, I have not set any LINGER option on any of the sockets. >>> > >>> > Thanks, >>> > -- >>> > Indradhanush Gupta >>> > (dhanush on irc) >>> > >>> > >>> > ___ >>> > zeromq-dev mailing list >>> > zeromq-dev@lists.zeromq.org >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> > >>> ___ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > > -- > Indradhanush Gupta > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmq_ctx_term vs zmq_ctx_destroy vs zmq_ctx_shutdown
To add further confusion, PyZMQ distinguishes term from destroy (pyzmq used the term ‘destroy’ before zmq did). In pyzmq, ctx.term just calls the underlying libzmq zmq_ctx_term (née zmq_term) function, which blocks until all sockets are closed. ctx.destroy, on the other hand, closes all sockets prior to calling term. czmq has a similar behavior for destroy. After these destroy behaviors were established, libzmq added a zmq_ctx_destroy function, but just as a rename of zmq_term, which causes confusion like this, hence the change replacing zmq_ctx_destroy with the less confusing zmq_ctx_term. -MinRK On Sun, Jun 15, 2014 at 6:03 AM, Pieter Hintjens wrote: > zmq_term/zmq_init are an older deprecated API. We switched to a more > consistent model for the API in 3.2, so zmq_ctx_xxx for all methods > that work with contexts, like zmq_msg_xxx for all methods that work on > messages. > > zmq_ctx_destroy was the initial choice for the termination method. > However people pointed out that the context isn't actually destroyed, > it's terminated, so we added _term() as a synonym. To be honest I'm > not keen on _term() as it seems inconsistent for no benefit. We > destroy sockets and contexts asynchronously... I'd prefer _destroy(). > > More usefully, we later added _shutdown() method that stops the > context but leaves it in existence; this allows a two-stage shutdown, > with signals being sent to all sockets waiting on blocking operations, > and then allowing the app to call zmq_ctx_destroy/term when wanted. > > -Pieter > > On Sat, Jun 14, 2014 at 3:34 PM, Indradhanush Gupta > wrote: > > Hello, > > > > I consulted the docs for zmq_ctx_term and zmq_ctx_destroy, in the API > > version 4.0 and it appears to be both have the same description. What is > the > > difference between the two? > > I'm using pyzmq, by the way. > > > > When I called zmq.Context.term() the call blocks indefinitely, while > > zmq.Context.destroy() returns immediately. I am shutting down my IOLoop, > > then closing all open sockets by hand and only then calling one of the > > above. > > > > What is the difference between the two calls? Also, it appears > > zmq_ctx_destroy is going to be deprecated according to the 4.1 dev API > docs. > > Why is term() blocking while destroy() doesn't? > > > > I'm also confused as to when should I call shutdown, destroy or term? > > > > If it helps, I have not set any LINGER option on any of the sockets. > > > > Thanks, > > -- > > Indradhanush Gupta > > (dhanush on irc) > > > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Reply-Request Pattern: Accessing a specific server method with a client request message?
On Mon, Apr 28, 2014 at 10:16 AM, Michel Pelletier < pelletier.mic...@gmail.com> wrote: > 0mq only works with messages. If you want to messages to indicate > functions to be called or arguments to be passed, you have to do that work > yourself. 0mq doesn't know what a Python function is, or how to serialize > a list of arguments. Thankfully it is relatively straightforward in Python > and in many other languages to accomplish this and has already been done. > Check out ZeroRPC, it's a Remote Procedure Call library that works over > zeromq: > > http://zerorpc.dotcloud.com/ > > zerorpc isn't your only option, there are many rpc libraries for Python, > most of which don't use 0mq. > > > http://stackoverflow.com/questions/1879971/what-is-the-current-choice-for-doing-rpc-in-python > And there are many that do: https://pypi.python.org/pypi?%3Aaction=search&term=zeromq+rpc&submit=search -MinRK > > > -Michel > > > On Mon, Apr 28, 2014 at 9:15 AM, Jess Updegrove wrote: > >> Hello Community! >> >> I'm a new programmer and I'm trying to build a server that has a bunch of >> different functions. I want the architecture to be server-client, in which >> a client can connect to the server, send it a list of parameters and the >> function it wants the result from, and get a reply. However, I just cannot >> seem to figure out how to do this in ZMQ and I'm looking for guidance. I'm >> using Python (2.7) and the pyzmq library with the latest version of ZMQ. >> I've read the Guide multiple times and looked through a bunch of examples. >> Essentially, I want the convo between the client and server to go something >> like this: >> >> C: Server, give me the current temperature in Seattle, Washington using >> the function get_weather() >> S: Accessing get_weather() with argument "Seattle" [goes to it's function >> and gets the result] >> S: Here's the result! >> C: Server, change the temperature (in a database) for Seattle from 50 to >> 70. >> S: Accessing change_value() with key temperature.Seattle. >> S: Value updated! >> >> However, I can only find examples that allow me to send a message but >> not specify a list of arguments. I also just can't seem to figure out the >> best way to send a message from the client and have the server pick it >> apart and figure out which function to call without using a ton of if >> statements checking the message content. Is ZMQ suitable for what I'm >> doing? Am I approaching it incorrectly? Also note that I already built a >> similar architecture like this is WAMPV2 using RPC calls and that's >> essentially what I'm trying to replicate. I'm new with the terminology used >> by ZMQ (sockets, threads, etc.) so it's very likely I'm just confused about >> how it all works. Any help would be appreciated! >> >> Thanks >> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] [ANNOUNCE] pyzmq 14.2.0 fixes memory leak
I just released pyzmq 14.2, which fixes a memory leak introduced in 14.0 when using `copy=False`. https://pypi.python.org/pypi/pyzmq/14.2.0 -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [pyzmq] Cython vs CFFI backends
On Sun, Apr 13, 2014 at 7:12 PM, Matt Connolly wrote: > I had a similar issue with the ruby gem “rbczmq” which wraps CZMQ. > > CZMQ itself wraps the ZMQ context and does its own socket close (with > linger set) before asking ZMQ context terminate. This process is not thread > safe and I needed to use a mutex to ensure that no other sockets were > closed during this shutdown. (In ruby this could be triggered by a garbage > collect of a socket object.). > pyzmq has the same mechanism - ctx.destroy([linger]) closes sockets before calling term. It is not at all threadsafe, so should only ever be used in single-threaded applications, or with extreme caution. It's easy to segfault with this if you are using sockets in threads. -MinRK > Using a mutex for socket close would have a penalty if many sockets need > to be opened and closed quickly, however, I understand this isn’t a > recommended pattern anyway. > > Best, > Matt. > > > On 14 Apr 2014, at 8:41 am, Michi Henning wrote: > > I just went through a similar experience. In my case, the problem was that > close() > wasn't called from the same thread that created the socket. > > You may find the explanations here useful: > > http://zeromq.org/whitepapers:0mq-termination > > Debugging these hands is a real pain. Is there anything that could be > added to the Zmq API > to make this easier? For example, getting a list of unclosed sockets would > be ultra-useful, > as would be warnings if a socket is closed from a thread other than the > one that created it. > > Cheers, > > Michi. > > > On 14 Apr 2014, at 8:29 , Alexander S. wrote: > > Hey all. > > I am debugging a hang in my application: the ctx.linger option is set > before creating any sockets, but it still hangs on ctx.term(). > > While scanning the pyzmq code I noticed the CFFI backend supports passing > a linger parameter to the term() call whereas the Cython backend does not. > This does not appear to be documented. Is there a reason for this > difference? Or can I simply code up a pull request to add the linger > parameter? > > More generally, is PyPy the only reason one would use the CFFI backend? > That is, can I compile pyzmq to use CFFI with CPython? And do we have a > published list of differences/features supported by the two backends? > > Lot's of questions. :) Thanks for the help! > > Alexander Sideropoulos > alexan...@thequery.net > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [pyzmq] Cython vs CFFI backends
On Sun, Apr 13, 2014 at 3:29 PM, Alexander S. wrote: > Hey all. > > I am debugging a hang in my application: the ctx.linger option is set > before creating any sockets, but it still hangs on ctx.term(). > > While scanning the pyzmq code I noticed the CFFI backend supports passing > a linger parameter to the term() call whereas the Cython backend does not. > This does not appear to be documented. Is there a reason for this > difference? Or can I simply code up a pull request to add the linger > parameter? > Any deviation of the CFFI backend from the Cython backend should be considered a bug (zero-copy support being the one exception). It doesn't make much sense to set LINGER in term because: - linger can only be set on sockets before closing them - the only way for term to be called with open sockets and not block forever is if those sockets are active in other threads - if sockets are active in other threads, they should not be touched because sockets are not threadsafe I have [removed](https://github.com/zeromq/pyzmq/pull/528) the inappropriate `linger` arg from Context.term in the cffi backend. > > More generally, is PyPy the only reason one would use the CFFI backend? > That is, can I compile pyzmq to use CFFI with CPython? > PyPy support is the reason the CFFI backend exists, though technically you can use the CFFI backend with CPython. It is not officially supported, nor would I recommend anyone use this combination. > And do we have a published list of differences/features supported by the > two backends? > The only known difference is a lack of support for zero-copy support in the CFFI backend. Anything else is probably a bug. -MinRK > > Lot's of questions. :) Thanks for the help! > > Alexander Sideropoulos > alexan...@thequery.net > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Hilarious(?) weird error when using PyZMQ 14.1.1 + zmq.auth.create_certificates()
That is indeed an interesting failure. How did you install libzmq and libsodium? What is the output of a simple: python -c 'import zmq; print(zmq.curve_keypair())' -MinRK On Fri, Mar 28, 2014 at 2:23 AM, Jonas Thiem wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hi *, > > Traceback (most recent call last): > File "./keycertgen.py", line 44, in > zmq.auth.create_certificates(path, "key") > File "/usr/lib64/python3.3/site-packages/zmq/auth/certs.py", line 67, > in create_certificates > public_key, secret_key = zmq.curve_keypair() > File "utils.pyx", line 51, in zmq.backend.cython.utils.curve_keypair > (zmq/backend/cython/utils.c:801) > File "/usr/lib64/python3.3/site-packages/zmq/error.py", line 127, in > _check_rc > raise ZMQError(errno) > zmq.error.ZMQError: Success > > It is quite amusing in a way, but still I would prefer if it actually > worked. > Same problem when calling zmq.curve_keypair() directly (not really > surprising since that is also what zmq.auth.create_certificates > appears to use from the callback). > > Any ideas what I need to do so zmq.curve_keypair() works properly? > > Regards, > Jonas Thiem > -BEGIN PGP SIGNATURE- > Version: GnuPG v1 > > iQIcBAEBAgAGBQJTNT+RAAoJEBIDTbbx8Yke7iIP/1l4SK2CUW7aUJmuTKC9/14L > WN88amkOjPr74uK2Kjxx1+aDHQQq81A6bMYdUzOz2hGFDvm0NFAORR+reO1f0qob > zW0U3LSyxp2ef+5IYqGKGzkXaorbHO/pLp6rEVvQnCIRISqvLg1vfinkY7ewRw1m > MTcq2olxUFfDMDlSqIaADY0ODtbZQWRHg29hQKYXR/i5/tlRlPfJeAR3uKnrgLKn > LEjt4Yd/bG32KQG9BFjFQpshmzoths+S5/1I40DO7K0JKhgC5jR2uuvICSFvh56U > Ai3QCG4PlQ8yw+geuExiT+FrrL+cKL6h/DJCjhK7pxcNK8hkLsPGmSygVFzwHyW3 > VJSPMubwWe74F1WdiIECBm6V3535+Om31mDTl3sauKfHFqiic3ym+VG2LSRGtB2d > bRkzWwWF0G1SQI6acjs/si4I/YF+/sh3wPm4d2Thf+jjYnLdX8ZJLPExFix9DK6N > 0hgOP7y3Ov4Xi+VeAYIzNlP5LTcjfglKJVOxnzWe3nnq/B6vtzausbEtTaGSuFoa > L+S7zqvqsKGcTtKGn2ZVNanK040SrvACW7MhjwHnwtQjcP/zFkzn0v9t02wRJSP6 > 5XlJkXu8xp4f0QpN6kGxXsRrwzebjvPWKpW0Lf8BJqyF+rQ5K//X4I6H3hPtPucg > fEmJDI5GsoHcoLbj89T9 > =MEGJ > -END PGP SIGNATURE- > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Pattern for clean shutdown of a proxy loop
On Wed, Mar 19, 2014 at 12:56 AM, Pieter Hintjens wrote: > Hmm... this would be more confusing than helpful IMO. 4.1.x should > normally go through the usual 'release candidate' -> 'stable' process. > This would suggest we can randomly backport new functionality so long > as we update the version number. That isn't our process at all... > If adding zmq_proxy_steerable can't be justified as fixing a bug, then > we can't backport it. > My view is that the old zmq_proxy is pretty unusable, and not much > used. The fact that the code sat unmodified for years kind of proves > it. I'm going to merge that pull request. > We may need to modify our rules to allow backports of changes that > seem exceptionally useful, or where the cost to e.g. bindings of not > backporting seems too high (as here). > Amending the rules is fine, I just wanted to point out that you can't backport new features without updating the minor version number within the current definitions of libzmq minor and patch versions. As an author and user of the pyzmq bindings, there is no cost to me in failing to backport the steerable function. I have used zmq_proxy daily (since it was called zmq_device), with no issue. I don't actually have any plan to expose the steerable version in pyzmq, because it doesn't offer any real benefit in that context. I don't think the steerable version of the function belongs in libzmq at all, so backporting it seems a bit silly to me. -MinRK > > > On Tue, Mar 18, 2014 at 8:21 PM, MinRK wrote: > > I think backporting the function is okay, but that would mean that > zeromq4-x > > should become 4.1.x, and libzmq should be bumped to 4.2. > > > > > > On Mon, Mar 17, 2014 at 1:04 AM, Pieter Hintjens wrote: > >> > >> We don't usually backport new functionality to existing stable > >> releases, because it's been troublesome in the past (i.e. breaking > >> things in bad ways). > >> > >> However, this is fairly safe and if you want to make a pull request to > >> zeromq4-x with the new zmq_proxy code, I'll merge it. > >> > >> I've made an issue https://github.com/zeromq/libzmq/issues/929 that > >> you can refer to in the pull request. > >> > >> -Pieter > >> > >> On Mon, Mar 17, 2014 at 5:16 AM, Cosmo Harrigan > >> wrote: > >> > Thanks. Is it acceptable to backport zmq_proxy_steerable to libzmq > >> > 4.0.4? Or > >> > if not, when do you estimate the next stable release that would > contain > >> > it? > >> > > >> > Best, > >> > Cosmo > >> > > >> > > >> > On Tue, Mar 11, 2014 at 11:18 PM, Pieter Hintjens > wrote: > >> >> > >> >> CZMQ has its own steerable proxy so that it can run on older versions > >> >> of libzmq. Also, some optimizations, but that's secondary. Also, > >> >> wrapping libzmq's method is extra work, it turns out. > >> >> > >> >> The zmq_steerable_proxy method is quite recent so hasn't been wrapped > >> >> by other bindings yet. > >> >> > >> >> On Tue, Mar 11, 2014 at 11:58 PM, Cosmo Harrigan > >> >> wrote: > >> >> > Thanks for the reply. I took a look, and my understanding is that > the > >> >> > CZMQ > >> >> > zproxy class waits for messages from both a backend and a pipe > (which > >> >> > is > >> >> > a > >> >> > PAIR socket to expose the API) and updates a local variable > 'stopped' > >> >> > which > >> >> > ends the loop when the pipe receives the 'STOP' command. > >> >> > > >> >> > And it refers to that as a 'steerable proxy'. > >> >> > > >> >> > Then, I found that there is a class in libzmq called > >> >> > zmq_proxy_steerable > >> >> > (http://api.zeromq.org/4-1:zmq-proxy-steerable), which seems to > offer > >> >> > exactly what I was asking about. > >> >> > > >> >> > Why isn't CZMQ zproxy using zmq_proxy_steerable? > >> >> > > >> >> > For C++ programming, cppzmq doesn't wrap either of those classes. > >> >> > Niether > >> >> > does zmqpp. > >> >> > > >> >> > So, I'm wondering, what pattern do other people who are using > ZeroMQ > >
Re: [zeromq-dev] Pattern for clean shutdown of a proxy loop
I think backporting the function is okay, but that would mean that zeromq4-x should become 4.1.x, and libzmq should be bumped to 4.2. On Mon, Mar 17, 2014 at 1:04 AM, Pieter Hintjens wrote: > We don't usually backport new functionality to existing stable > releases, because it's been troublesome in the past (i.e. breaking > things in bad ways). > > However, this is fairly safe and if you want to make a pull request to > zeromq4-x with the new zmq_proxy code, I'll merge it. > > I've made an issue https://github.com/zeromq/libzmq/issues/929 that > you can refer to in the pull request. > > -Pieter > > On Mon, Mar 17, 2014 at 5:16 AM, Cosmo Harrigan > wrote: > > Thanks. Is it acceptable to backport zmq_proxy_steerable to libzmq > 4.0.4? Or > > if not, when do you estimate the next stable release that would contain > it? > > > > Best, > > Cosmo > > > > > > On Tue, Mar 11, 2014 at 11:18 PM, Pieter Hintjens wrote: > >> > >> CZMQ has its own steerable proxy so that it can run on older versions > >> of libzmq. Also, some optimizations, but that's secondary. Also, > >> wrapping libzmq's method is extra work, it turns out. > >> > >> The zmq_steerable_proxy method is quite recent so hasn't been wrapped > >> by other bindings yet. > >> > >> On Tue, Mar 11, 2014 at 11:58 PM, Cosmo Harrigan > >> wrote: > >> > Thanks for the reply. I took a look, and my understanding is that the > >> > CZMQ > >> > zproxy class waits for messages from both a backend and a pipe (which > is > >> > a > >> > PAIR socket to expose the API) and updates a local variable 'stopped' > >> > which > >> > ends the loop when the pipe receives the 'STOP' command. > >> > > >> > And it refers to that as a 'steerable proxy'. > >> > > >> > Then, I found that there is a class in libzmq called > zmq_proxy_steerable > >> > (http://api.zeromq.org/4-1:zmq-proxy-steerable), which seems to offer > >> > exactly what I was asking about. > >> > > >> > Why isn't CZMQ zproxy using zmq_proxy_steerable? > >> > > >> > For C++ programming, cppzmq doesn't wrap either of those classes. > >> > Niether > >> > does zmqpp. > >> > > >> > So, I'm wondering, what pattern do other people who are using ZeroMQ > in > >> > C++ > >> > currently follow in this case to control their proxy loops? I suppose > >> > one > >> > solution is the one mentioned in the other thread, namely, wrapping > CZMQ > >> > in > >> > your C++ application. Any suggestions? > >> > > >> > Thanks, > >> > Cosmo > >> > > >> > > >> > > >> > > >> > > >> > On Tue, Mar 11, 2014 at 2:02 PM, Pieter Hintjens > wrote: > >> >> > >> >> You can look at the CZMQ zproxy class to see one way to do this. > >> >> > >> >> On Tue, Mar 11, 2014 at 9:35 PM, Cosmo Harrigan > >> >> wrote: > >> >> > Hi, > >> >> > > >> >> > What is the best practice when you are creating a loop that > functions > >> >> > like > >> >> > the zmq_proxy device, to bind a PULL socket that is fed by a > backend > >> >> > ventilator with a frontend PUB socket, in order to properly handle > a > >> >> > clean > >> >> > shutdown? > >> >> > > >> >> > Most of the examples on the zguide just show "while (1)" with a > loop > >> >> > that > >> >> > never terminates. What's the suggested way of signaling to the loop > >> >> > which is > >> >> > running in its own thread that it is time to shut down? > >> >> > > >> >> > Thanks, > >> >> > Cosmo > >> >> > > >> >> > > >> >> > ___ > >> >> > zeromq-dev mailing list > >> >> > zeromq-dev@lists.zeromq.org > >> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >> >> > > >> >> ___ > >> >> zeromq-dev mailing list > >> >> zeromq-dev@lists.zeromq.org > >> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >> > > >> > > >> > > >> > ___ > >> > zeromq-dev mailing list > >> > zeromq-dev@lists.zeromq.org > >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > >> > > >> ___ > >> zeromq-dev mailing list > >> zeromq-dev@lists.zeromq.org > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] [ANNOUNCE] pyzmq-14.1.0
I just released pyzmq-14.1.0, which adds implementations of authenticators in zmq.auth, and bundled libsodium in bdists, so people are more likely to have pyzmq with security available. https://github.com/zeromq/pyzmq/releases/tag/v14.1.0 I will build the Windows bdists tomorrow. -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] [python] setuptools, pyzmq and sodium
On Thu, Mar 6, 2014 at 11:24 AM, Drew Crawford wrote: > Hi folks, > > >1. I’m writing a package >2. That I want to distribute with setuptools >3. That requires pyzmq >4. …to be built with libsodium > > > I’m wondering what is the best way to achieve this. If I simply declare > in my setup.py > > install_requires=['pyzmq>=14.0.1’] > > > This creates a dependency on pyzmq, however it does not check if it is > sodium-aware. And if no pyzmq is present on the system, it installs a > non-sodium version, so the install is incorrect. > > I see there is some work going on in pyzmq 14.1 to change how libsodium > packaging works internally but ideally I am looking for a solution that > works with 14.0.1. > If it helps, I'm trying to release pyzmq-14.1 in the next week or so. I was planning to make an RC today or tomorrow. 14.1 will bundle libsodium when it bundles libzmq. That still doesn't cover the case entirely, because if pyzmq can find a libzmq on the system, it will link it, even if it doesn't link libsodium. Quite simply, setuptools does not have the capacity to express dependencies on non-Python things. I see two main choices: 1. check for libsodium in setup.py (zmq.curve_keypair() will fail in the absence of libsodium), and fail (or warn) manually with a message saying that they need to rebuild libzmq and/or pyzmq. 2. make the same check at import time. This will be the most visible to users. I would recommend #2 because #1 *will not* work if you ever decide to build wheels, because wheels do not allow you to express any nontrivial logic at install time. If you want a command that will guarantee pyzmq with libzmq and libsodium to include in this message (once I release it), it would be: pip install pyzmq>=14.1.0 --install-option="--zmq=bundled" -MinRK > > Drew > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq curve example code
The weird errno should be fixed in
master<https://github.com/zeromq/pyzmq/pull/483>.
But this should only affect the case when libzmq is not linked against
libsodium. Are you sure that it is?
What is the output of ldd /path/to/libzmq.so?
-MinRK
On Tue, Feb 25, 2014 at 2:09 PM, Greg Ward wrote:
> On 25 February 2014, To ZeroMQ development list said:
> > But that seems to have decayed somewhat. I noticed an error from
> > pyzmq's setup script about libsodium in the umpteenth build yesterday.
> > I think maybe I'll start from scratch with a fresh prefix dir and see
> > if that clarifies things.
>
> OK I've updated libsodium, libzmq, and pyzmq from git master, rebuilt
> and reinstalled everything to a fresh prefix (/usr/local/zmq4).
> Building pyzmq ("python setup.py build --zmq=/usr/local/zmq4") printed
>
> Warning: Detected ZMQ version: 4.1.0. pyzmq's support for libzmq-dev is
> experimental.
>
> but what the hell, I'm gonna barge on past that and take a chance. I
> saw no warnings about curve or libsodium.
>
> Now I have three problems in examples/security:
>
> 1) strawhouse succeeds but logs an error:
>
> $ python strawhouse.py
> [ERROR] Failed to deny [u'127.0.0.1']
> Traceback (most recent call last):
>File
> "/usr/local/zmq4/lib/python2.7/site-packages/zmq/auth/thread.py", line 97,
> in _handle_pipe
> self.authenticator.deny(*addresses)
>File
> "/usr/local/zmq4/lib/python2.7/site-packages/zmq/auth/base.py", line 85, in
> deny
> raise ValueError("Only use a whitelist or a blacklist, not both")
> ValueError: Only use a whitelist or a blacklist, not both
> [INFO] Strawhouse test OK
>
>(I didn't see this yesterday; unclear what has changed -- except
>I'm now linking with latest upstream libzmq)
>
> 2) stonehouse.py refers to the wrong script:
>
> $ python stonehouse.py
> [CRITICAL] Certificates are missing - run generate_certificates
> script first
>
>(https://github.com/zeromq/pyzmq/pull/480)
>(I saw this yesterday, so today I thought I'd fix it)
>
> 3) generate_keys.py still crashes mysteriously:
>
> $ python generate_keys.py
> Traceback (most recent call last):
>File "generate_keys.py", line 49, in
> generate_certificates(os.path.dirname(__file__))
>File "generate_keys.py", line 30, in generate_certificates
> server_public_file, server_secret_file =
> zmq.auth.create_certificates(keys_dir, "server")
>File
> "/usr/local/zmq4/lib/python2.7/site-packages/zmq/auth/certs.py", line 67,
> in create_certificates
> public_key, secret_key = zmq.curve_keypair()
>File "utils.pyx", line 51, in
> zmq.backend.cython.utils.curve_keypair (zmq/backend/cython/utils.c:762)
>File "/usr/local/zmq4/lib/python2.7/site-packages/zmq/error.py",
> line 128, in _check_rc
> raise ZMQError(errno)
> zmq.error.ZMQError: No such file or directory
>
>(same problem as yesterday)
>
> So I've dug in a little bit, and it looks like something is
> incorrectly reusing errno. Evidence: I hacked
> zmq/backend/cython/utils.py as follows:
>
> --- a/zmq/backend/cython/utils.pyx
> +++ b/zmq/backend/cython/utils.pyx
> @@ -44,10 +44,16 @@ def curve_keypair():
> (public, secret) : two bytestrings
> The public and private keypair as 40 byte z85-encoded bytestrings.
> """
> +import os, signal
> cdef int rc
> cdef char[64] public_key
> cdef char[64] secret_key
> +print('calling zmq_curve_keypair(%r, %r)' % (public_key, secret_key))
> +pid = os.getpid()
> +print('sudo strace -f -tt -p %d' % pid)
> +os.kill(pid, signal.SIGSTOP)
> rc = zmq_curve_keypair (public_key, secret_key)
> +print('zmq_curve_keypair() = %r' % rc)
> _check_rc(rc)
> return public_key, secret_key
>
>
> This gives me:
>
> * the opportunity to strace (in another terminal window) starting
> from just before the mysterious error
> * clear indicators (write() to stdout) before and after the error
>
> So I run generate_keys.py with the hacked utils.pyx:
>
> $ python generate_keys.py
> calling zmq_curve_keypair('P\xc4X', '')
> sudo strace -f -tt -p 14453
> zsh: suspended (signal) python generate_keys.py
>
> In another window, I run the suggested strace command. Then I "fg"
> generate_keys.py. Here's what strace reports:
>
> $ sudo strace -f -tt -p 143
Re: [zeromq-dev] pyzmq curve example code
How did you build libzmq and/or install pyzmq? It could be that you don't have libsodium linked, in which case the curve_keypair would fail. If that's the case, obviously the error message should be better. -MinRK On Mon, Feb 24, 2014 at 3:47 PM, Greg Ward wrote: > Hey all -- > > I'm playing around with the security examples in latest master of > pyzmq. Looks promising, but not perfect. In particular, I'm stuck by > this failure: > > $ python generate_keys.py > Traceback (most recent call last): > File "generate_keys.py", line 50, in > generate_certificates(os.path.dirname(__file__)) > File "generate_keys.py", line 31, in generate_certificates > server_public_file, server_secret_file = > zmq.auth.create_certificates(keys_dir, "server") > File "/data/src/pyzmq/zmq/auth.py", line 74, in create_certificates > public_key, secret_key = zmq.curve_keypair() > File "utils.pyx", line 53, in zmq.backend.cython.utils.curve_keypair > (zmq/backend/cython/utils.c:820) > File "/data/src/pyzmq/zmq/error.py", line 128, in _check_rc > raise ZMQError(errno) > zmq.error.ZMQError: No such file or directory > > I'm digging through the code now to figure out 1) which zmq call > failed, 2) what file or directory doesn't exist, and 3) how to improve > that exception to report the guilty filename. > > (N.B. line numbers in that stack trace don't match upstream: I've > hacked in some print statements to figure out what's going on.) > > Answers: > > 1) It's almost certain that zmq_curve_keypair() is failing, being called >from curve_keypair() in zmq/backend/cython/utils.pyx. > > 2) I don't see any file I/O in pyzmq's curve_keypair(), or in ZMQ itself. >No idea where the "No such file or directory" error is coming from. > > 3) Even less of an idea. > > Any clues? Hints are welcome. I'm happy to submit a pull request if > only I can figure out what's going on. > > Thanks -- > > Greg > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Using pyczmq for zcert/zauth only
On Tue, Feb 4, 2014 at 2:10 AM, Goswin von Brederlow wrote: On Mon, Feb 03, 2014 at 11:06:32AM -0800, MinRK wrote: > > On Mon, Feb 3, 2014 at 10:57 AM, Michel Pelletier < > > pelletier.mic...@gmail.com> wrote: > > > > > On Mon, Feb 3, 2014 at 9:30 AM, Greg Ward wrote: > > > > > >> Hi all -- > > >> > > >> > > >> But that doesn't work: > > >> > > >> $ cat shadow-ctx.py > > >> import zmq > > >> from pyczmq import zctx > > >> > > >> ctx1 = zmq.Context() > > >> ctx2 = zctx.shadow_zmq_ctx(ctx1) > > >> > > >> > > >> Arghghh. Impedance mismatch. Any idea how to make that wrapper work? > > >> > > > > > > Hmm, I thought I had wrapped shadow_zmq_ctx, but you're right I missed > it > > > and I misspoke. Thanks for the diff I'll add it in. > > > > > > When I made my statement they were interoperable I assumed the > existence > > > of method for zmq to expose the raw zeromq context object as a cffi > cdata > > > pointer. I guess what I should have said was *if* zmq exposes a cdata > > > pointer they can be interoperable via shadow_zmq_ctx. > > > > > > > > > > > >> The problem is right in the docstring for zctx.underlying(): > > >> > > >> def underlying(ctx): > > >> """ > > >> Return low-level 0MQ context object, will be NULL before first > socket > > >> is created. Use with care. > > >> """ > > >> > > >> Well, I'm stumped. Has anyone successfully used pyzmq and pyczmq in > > >> the same script? Care to show the code? > > >> > > >> > > > You could create a dummy socket, then you'll get the underlying cdata > > > pointer. Note that I don't know if pyzmq provides a way to use that > > > pointer. Same problem as above. > > > > > > I may have misinterpreted it, but I thought Min has said the latest > pyzmq > > > supports authentication and encryption? If not, I think sending a PR > to > > > pyzmq adding it would be easier than trying to get the libraries > compatible > > > in-process. Then everybody wins, the libraries can cross-talk (via > zmq) > > > and you get the features you want in one library. > > > > > > > pyzmq does support auth in zmq.auth, but I haven't cut a release since it > > was added. Will be 14.1.0, I presume. > > > > -MinRK > > Do you have an example source for that? And what's the ETA on the next > release? > Some examples are here<https://github.com/zeromq/pyzmq/tree/master/examples/security>. I need to do some cleanup, and fix some intermittently failing tests before release. I am in the midst of trying to release IPython right now, so it will probably be at least a few weeks. -MinRK > MfG > Goswin > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Using pyczmq for zcert/zauth only
On Mon, Feb 3, 2014 at 10:57 AM, Michel Pelletier < pelletier.mic...@gmail.com> wrote: > On Mon, Feb 3, 2014 at 9:30 AM, Greg Ward wrote: > >> Hi all -- >> >> >> But that doesn't work: >> >> $ cat shadow-ctx.py >> import zmq >> from pyczmq import zctx >> >> ctx1 = zmq.Context() >> ctx2 = zctx.shadow_zmq_ctx(ctx1) >> >> >> Arghghh. Impedance mismatch. Any idea how to make that wrapper work? >> > > Hmm, I thought I had wrapped shadow_zmq_ctx, but you're right I missed it > and I misspoke. Thanks for the diff I'll add it in. > > When I made my statement they were interoperable I assumed the existence > of method for zmq to expose the raw zeromq context object as a cffi cdata > pointer. I guess what I should have said was *if* zmq exposes a cdata > pointer they can be interoperable via shadow_zmq_ctx. > > > >> The problem is right in the docstring for zctx.underlying(): >> >> def underlying(ctx): >> """ >> Return low-level 0MQ context object, will be NULL before first socket >> is created. Use with care. >> """ >> >> Well, I'm stumped. Has anyone successfully used pyzmq and pyczmq in >> the same script? Care to show the code? >> >> > You could create a dummy socket, then you'll get the underlying cdata > pointer. Note that I don't know if pyzmq provides a way to use that > pointer. Same problem as above. > > I may have misinterpreted it, but I thought Min has said the latest pyzmq > supports authentication and encryption? If not, I think sending a PR to > pyzmq adding it would be easier than trying to get the libraries compatible > in-process. Then everybody wins, the libraries can cross-talk (via zmq) > and you get the features you want in one library. > pyzmq does support auth in zmq.auth, but I haven't cut a release since it was added. Will be 14.1.0, I presume. -MinRK > > -Michel > > > >> Thanks -- >> >>Greg >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyczmq and ctx.set_linger()
On Sat, Jan 18, 2014 at 2:29 AM, Goswin von Brederlow wrote: > On Fri, Jan 17, 2014 at 09:34:29AM -0500, Greg Ward wrote: > > On 17 January 2014, Pieter Hintjens said: > > > On Thu, Jan 16, 2014 at 9:01 PM, Michel Pelletier > > > wrote: > > > > > > > Yes, pyczmq is very much a literal 1 to 1 wrapper around the czmq C > > > > interface. > > > > > > > > Note that there is also an "object oriented" interface: > > > > > > CZMQ was designed to allow an OO interface on top; it may not succeed > > > in all classes. Where it's possible I'd probably not expose the 1-1 > > > wrapper at all, and only provide the proper OO interface. (Less > > > confusing for users, maybe?) > > > > Having played around with pyczmq a little bit for a few hours, I'm > > starting to wonder if that's really the right way to wrap CZMQ. My two > > concerns are 1) import-time overhead and 2) the C-style interface (not > > OO, un-Pythonic). > > > > I note that pyzmq already solves both of these problems: it's much > > faster to import than pyczmq, and it exposes a nice Pythonic interface > > to the 0mq core API. > > > > Would it make sense for pyzmq to wrap CZMQ as well? > > > >Greg > > I've been using pyzmq and didn't even know pyczmq (or czmq) existed > because nothing was missing in pyzmq. > > But with zeromq 4.0 and CURVE there are parts that pyzmq lacks. > Notably the authentication and certificate functions. I think it would > be greate to have that added to pyzmq without requiring yet another > module. Either via czmq or simply reimplement it in native python. > Current (14.0) pyzmq supports all the security features in libzmq, but the extra convenience added by czmq (ZAP handlers, certificates etc.) are in pyzmq master via `zmq.auth`. -MinRK > > MfG > Goswin > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyczmq and ctx.set_linger()
On Fri, Jan 17, 2014 at 8:12 AM, Michel Pelletier < pelletier.mic...@gmail.com> wrote: > On Fri, Jan 17, 2014 at 6:34 AM, Greg Ward wrote: > >> On 17 January 2014, Pieter Hintjens said: >> > On Thu, Jan 16, 2014 at 9:01 PM, Michel Pelletier >> > wrote: >> > >> > > Yes, pyczmq is very much a literal 1 to 1 wrapper around the czmq C >> > > interface. >> > > >> > > Note that there is also an "object oriented" interface: >> > >> > CZMQ was designed to allow an OO interface on top; it may not succeed >> > in all classes. Where it's possible I'd probably not expose the 1-1 >> > wrapper at all, and only provide the proper OO interface. (Less >> > confusing for users, maybe?) >> >> Having played around with pyczmq a little bit for a few hours, I'm >> starting to wonder if that's really the right way to wrap CZMQ. My two >> concerns are 1) import-time overhead > > > This seems like an un-problem to me. cffi parses the function > declarations on import, that takes a small amount time. You pay that small > price once. > > >> and 2) the C-style interface (not >> OO, un-Pythonic). >> > > As I pointed out there is an "OO" interface. Maybe it's the Lisper in me > talking, but... they're just functions. They take arguments. You call > them. It's seems pretty straightforward and "Pythonic" to me. > > >> I note that pyzmq already solves both of these problems: it's much >> faster to import than pyczmq, and it exposes a nice Pythonic interface >> to the 0mq core API. >> > > It certainly does and I'm quite fond of pyzmq, I use it in all my existing > zmq production code. There's no reason why you shouldn't use one or the > other, or both at the same time. pyczmq is going to be slower, guaranteed. > CFFI does things in Python interpreter executed code that pyzmq does in > compiled C code, compiling away the interpreter and there is no doubt in my > mind that pyzmq is faster. pyczmq is not an attempt by me to replace pyzmq > or supersede it in any way, my primary goals were to 1) learn CFFI, 2) > evaluate the czmq API (it's quite nice, and ironically, "Pythonic" from the > C point of view, passing the "instance" consistently as the first argument). > > Would it make sense for pyzmq to wrap CZMQ as well? >> > > I would say absolutely but I see MinRK doesn't agree. But, the point is > kind of moot, as I said you can use both at the same time. There is no > difference between czmq sockets and zmq sockets, and you can access the > inner context object in czmq via zctx_underlying (exposed as > pyczmq.zctx.underlying). > I think making sure the low-level pointers are available in such a way that you can comfortably pass the zmq sockets and contexts in between the two libraries would be useful, and should be fairly easy. > > -Michel > > > >> >>Greg >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyczmq import overhead
On Thu, Jan 16, 2014 at 12:12 PM, Michel Pelletier < pelletier.mic...@gmail.com> wrote: > Hmm, I don't have any strong feelings either way. All python imports > seems slow to me on almost all useful libraries, I use a lot of numpy, > scipy, pandas, and scikit-learn for my job and I'm used to multi-second > startup times. Even ipython takes a second or two on my new macbook. For > long running jobs and apps it doesn't seem to be an issue. > > If you're running very short scripts that need to fire and exit quickly, > then I can see that being a problem, but 0.5 isn't a huge win to me. But > like I said, I don't have strong feelings, and I think you could count > pyczmq's users on one hand at this point. Anyone else have an opinion? > > Also, this might be some slowness related to using dlopen instead of > verify in cffi. verify is *much* slower the first time due to compilation, > but then is much faster for subsequent imports. Maybe investigating that > approach first would be fruitful. > If you use cffi.verify, you can also do the compilation at build time, as recommended by cffi<http://cffi.readthedocs.org/en/latest/index.html?highlight=get_extension#distributing-modules-using-cffi>. PyZMQ does this<https://github.com/zeromq/pyzmq/blob/f58bb80b3b7583fb45824d0a1037128f7f29d44a/setup.py#L1003> for its CFFI backend on PyPy, and it seems to work reasonably well. -MinRK > > -Michel > > > On Thu, Jan 16, 2014 at 9:53 AM, Greg Ward wrote: > >> Speaking of pyczmq, it is *very* slow to import: >> >> $ time python -c 'import pyczmq' >> python -c 'import pyczmq' 1.11s user 0.05s system 100% cpu 1.152 total >> >> Turns out this is because pyczmq/__init__.py imports every submodule, >> as well as some identifiers in a couple of submodules. I'm pretty sure >> this is considered an anti-pattern. (I dimly recall seeing Guido van >> Rossum disparage the practice on python-dev many years ago.) >> >> I've come to agree with him, if only on performance grounds. E.g. >> here's the best case: >> >> $ echo -n > pyczmq/__init__.py >> $ time python -c 'import pyczmq' >> python -c 'import pyczmq' 0.03s user 0.01s system 99% cpu 0.039 total >> >> A more realistic test: >> >> $ time python -c 'from pyczmq import zmq, zctx, zsocket' >> python -c 'from pyczmq import zmq, zctx, zsocket' 0.46s user 0.05s >> system 100% cpu 0.508 total >> >> Or, for all the submodules needed to implement Pieter's "ironhouse" >> example in Python: >> >> $ time python -c 'from pyczmq import zmq, zctx, zsocket, zcert, zauth, >> zstr' >> python -c 'from pyczmq import zmq, zctx, zsocket, zcert, zauth, zstr' >> 0.61s user 0.03s system 100% cpu 0.640 total >> >> That's not great, but 0.6 sec is still better than 1.1 sec. >> >> Is pyczmq young enough that we can consider breaking backwards >> compatibility like this? Anyone currently doing >> >> import pyczmq >> >> ctx = pyczmq.zctx.new() >> sock = pyczmq.zsocket.new(...) >> >> would have to change their imports to: >> >> import pyczmq.zctx >> import pyczmq.zsocket >> >> If that's OK, I'll work up a patch and send it. >> >>Greg >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyczmq and ctx.set_linger()
On Fri, Jan 17, 2014 at 6:34 AM, Greg Ward wrote: > On 17 January 2014, Pieter Hintjens said: > > On Thu, Jan 16, 2014 at 9:01 PM, Michel Pelletier > > wrote: > > > > > Yes, pyczmq is very much a literal 1 to 1 wrapper around the czmq C > > > interface. > > > > > > Note that there is also an "object oriented" interface: > > > > CZMQ was designed to allow an OO interface on top; it may not succeed > > in all classes. Where it's possible I'd probably not expose the 1-1 > > wrapper at all, and only provide the proper OO interface. (Less > > confusing for users, maybe?) > > Having played around with pyczmq a little bit for a few hours, I'm > starting to wonder if that's really the right way to wrap CZMQ. My two > concerns are 1) import-time overhead and 2) the C-style interface (not > OO, un-Pythonic). > > I note that pyzmq already solves both of these problems: it's much > faster to import than pyczmq, and it exposes a nice Pythonic interface > to the 0mq core API. > > Would it make sense for pyzmq to wrap CZMQ as well? > I don't think it would. Most of the things czmq adds to libzmq are more logically reimplemented in Python, rather than exposed by linking anothing library. The added C dependency would also be too much of a pain, given the minimal benefit it provides at the Python level. -MinRK > >Greg > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PyZMQ recv() acts crazy on one machine
And how did you install libzmq? It seems like you have a borked install.
Perhaps rebuilding libzmq and/or pyzmq is the answer.
On Sun, Dec 29, 2013 at 6:27 PM, Thomas Johnson
wrote:
> Thanks, that was the problem. The broken one had 4.0.3 and the working one
> had 2.2.0
>
>
> On Sun, Dec 29, 2013 at 12:09 PM, Min RK wrote:
>
>> what is zmq.zmq_version() on each?
>>
>> -MinRK
>>
>> On Dec 29, 2013, at 0:01, Thomas Johnson
>> wrote:
>>
>> Consider the following simple program:
>> #CUT HERE
>> #!/usr/bin/python
>> import zmq
>>
>> context_push = zmq.Context(1)
>> socket_push = context_push.socket(zmq.PUSH)
>>
>> context_pull = zmq.Context(1)
>> socket_pull = context_pull.socket(zmq.PULL)
>>
>> socket_pull.bind("tcp://127.0.0.1:2000")
>> socket_push.connect("tcp://127.0.0.1:2000")
>>
>> socket_push.send("Hello")
>> message = socket_pull.recv()
>> print "Got message: '%s'" % message
>> #CUT HERE
>>
>>
>> On machine A, as expected, it prints: Got message: 'Hello'
>> On machine B, it prints: Got message: ''
>>
>> Separately from this program, Machine B also immediately raises a
>> "Resource temporarily unavailable" if I create a PULL socket and do a
>> (blocking) socket.recv() call. Strangely, it seems to hang if I do a
>> socket.recv(flags=zmq.NOBLOCK), almost as if the flag was being flipped.
>>
>> Both A and B are Ubuntu precise, using pyzmq 14.0.1. B happens to have
>> its native language set as Russian, if that matters.
>>
>> Any idea what is going on or what I'm doing wrong?
>>
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Problem installing python language binding on openwrt
what C compiler do you have available? You may need to:
export CC=actual_c_compiler
because Python tries to compile extensions with the compiler used for
Python itself by default (often not available if cross compiled)
On Mon, Dec 9, 2013 at 11:15 AM, Roman wrote:
> Here is the output of running "DISTUTILS_DEBUG=1 python setup.py install"
> on my device
>
> root@OpenWrt:/home# cd pyzmq-14.0.1/
> root@OpenWrt:/home/pyzmq-14.0.1# ls
> AUTHORS.md PKG-INFOdocssetup.py
> CONTRIBUTING.md README.md examplessetupegg.py
> COPYING.BSD build perftox.ini
> COPYING.LESSER buildutils setup.cfg.android zmq
> MANIFEST.in bundled setup.cfg.template zmqversion.py
> root@OpenWrt:/home/pyzmq-14.0.1# DISTUTILS_DEBUG=1 python setup.py install
> options (after parsing config files):
> options (after parsing command line):
> option dict for 'install' command:
> {}
> running install
> Distribution.get_command_obj(): creating 'install' command object
> pre-finalize_{unix,other}:
> prefix: None
> exec_prefix: None
> home: None
> user: 0
> install_base: None
> install_platbase: None
> root: None
> install_purelib: None
> install_platlib: None
> install_lib: None
> install_headers: None
> install_scripts: None
> install_data: None
> compile: None
> compile: True
> optimize: None
> force: None
> skip_build: 0
> record: None
> post-finalize_{unix,other}():
> prefix: /usr
> exec_prefix: /usr
> home: None
> user: 0
> install_base: /usr
> install_platbase: /usr
> root: None
> install_purelib: $base/lib/python$py_version_short/site-packages
> install_platlib: $platbase/lib/python$py_version_short/site-packages
> install_lib: None
> install_headers: $base/include/python$py_version_short/$dist_name
> install_scripts: $base/bin
> install_data: $base
> compile: None
> compile: True
> optimize: None
> force: None
> skip_build: 0
> record: None
> post-expand_basedirs():
> prefix: /usr
> exec_prefix: /usr
> home: None
> user: 0
> install_base: /usr
> install_platbase: /usr
> root: None
> install_purelib: $base/lib/python$py_version_short/site-packages
> install_platlib: $platbase/lib/python$py_version_short/site-packages
> install_lib: None
> install_headers: $base/include/python$py_version_short/$dist_name
> install_scripts: $base/bin
> install_data: $base
> compile: None
> compile: True
> optimize: None
> force: None
> skip_build: 0
> record: None
> config vars:
> {'base': '/usr',
> 'dist_fullname': 'pyzmq-14.0.1',
> 'dist_name': 'pyzmq',
> 'dist_version': '14.0.1',
> 'exec_prefix': '/usr',
> 'platbase': '/usr',
> 'prefix': '/usr',
> 'py_version': '2.7.3',
> 'py_version_nodot': '27',
> 'py_version_short': '2.7',
> 'sys_exec_prefix': '/usr',
> 'sys_prefix': '/usr',
> 'userbase': '/root/.local',
> 'usersite': '/root/.local/lib/python2.7/site-packages'}
> post-expand_dirs():
> prefix: /usr
> exec_prefix: /usr
> home: None
> user: 0
> install_base: /usr
> install_platbase: /usr
> root: None
> install_purelib: /usr/lib/python2.7/site-packages
> install_platlib: /usr/lib/python2.7/site-packages
> install_lib: None
> install_headers: /usr/include/python2.7/pyzmq
> install_scripts: /usr/bin
> install_data: /usr
> compile: None
> compile: True
> optimize: None
> force: None
> skip_build: 0
> record: None
> after prepending root:
> prefix: /usr
> exec_prefix: /usr
> home: None
> user: 0
> install_base: /usr
> install_platbase: /usr
> root: None
> install_purelib: /usr/lib/python2.7/site-packages
> install_platlib: /usr/lib/python2.7/site-packages
> install_lib: /usr/lib/python2.7/site-packages/
> install_headers: /usr/include/python2.7/pyzmq
> install_scripts: /usr/bin
> install_data: /usr
> compile: None
> compile: True
> optimize: None
> force: None
> skip_build: 0
> record: None
> Distribution.get_command_obj(): creating 'build' command object
> running build
> running build_py
> Distribution.get_command_obj(): creating 'build_py' command object
> running build_ext
> Distribution.get_command_obj(): creating 'build_ext' command object
> running configure
> Distribution.get_command_obj(): creating 'configure' command object
>
> Configure: Autodetecting ZMQ settings...
> Custom ZMQ dir: /usr/local
> creating build/temp.linux-mips-2.7/scratch/tmp
> cc -c /tmp/timer_create4S7pHV.c -o
> build/temp.linux-mips-2.7/scratch/tmp/timer_create4S7pHV.o
> unable to execute cc: No such file or directory
> ccache_cc -fno-strict-aliasing -Os -pipe -mips32r2 -mtune=mips32r2
> -fno-caller-saves -fhonour-copts -Wno-error=unused-but-set-variable
> -msoft-float -DNDEBUG -Os -pipe -mips32r2 -mtune=mips32r2 -fno-caller-saves
> -fhonour-copts -Wno-error=unused-but-set-variable
Re: [zeromq-dev] Migrating to 4.0
In general, you can use 0MQ 4 exactly the same as 3.2, there should be no backward-incompatible changes, only new stuff. That's the goal, anyway, obviously you have to give it a try to see how it goes in practice, but it's been pretty solid in my experience thus far. -MinRK On Fri, Dec 6, 2013 at 2:48 PM, Greg Ward wrote: > Hi all -- > > I have an app running just fine on 0MQ 3.2.3 and pyzmq 13.1.0. > Unfortunately it includes a homebrew cryptosystem that makes me > nervous. I'd like to move to 0MQ 4.0 and use the builtin crypto there. > I don't see any docs on porting to 0MQ 4.0... am I missing something? > Or is it trivial -- just upgrade the package and continue as if > nothing happened? > > Thanks -- > >Greg > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread affinity
On Mon, Dec 2, 2013 at 3:15 PM, Greg Ward wrote: > On 30 November 2013, Pieter Hintjens said: > > Hi Justin, > > > > This is an area of some debate. We've had patches to libzmq that made > > sockets thread safe, and removed those patches again. Sharing sockets > > between threads for the most part is bad for design and performance. > > However there are languages where this just essential, for the reasons > > you describe. > > > > My advice would be to solve this in the language binding. Simply > > create socket access routines (send/recv) that do the necessary > > locking. > > But doesn't that mean holding a lock during a potentially blocking > operation? E.g. if I send() to a REQ socket that has hit its > high-water mark, send() will block. If I'm holding a mutex during that > operation, than other threads can block on that mutex for a long time. > That sort of nasty cascading lock situation can go very bad very > quickly in my experience. > You are holding a lock on a resource while it's not safe for another thread to use it. If you have significant contention on one of these locks, it means you are trying to have significant concurrent access on a single socket, and the lock will effectively serialize those threads. This is really why sharing sockets across threads is ill advised. You can use locks to move a socket from one thread to another, but it is only useful if that transaction is sufficiently rare, such as when blocking zmq calls are a minority of your app's run time. -MinRK > >Greg > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Announce: libzmq/4.0.2 stable released
I also cut pyzmq-14.0.1 yesterday, which bundles 4.0.3. On Sun, Nov 24, 2013 at 11:50 PM, Pieter Hintjens wrote: > Hi all, > > There was a failing test case in 4.0.2 (error in a new test case, not > the library), so we made a 4.0.3 release to fix that. Same place as > usual. Sorry for the hiccup. > > -Pieter > > On Sun, Nov 24, 2013 at 3:27 PM, Pieter Hintjens wrote: > > Hi all, > > > > I've pushed a stable libzmq/4.0.2 release to the usual download site. > > > > You can get it from http://zeromq.org/intro:get-the-software. > > > > The tagged release is: > > https://github.com/zeromq/zeromq4-x/tree/v4.0.2 > > > > -Pieter > > > > > > 0MQ version 4.0.2 stable, released on 2013/11/24 > > > > > > Bug Fixes > > - > > > > * Fixed LIBZMQ-583 - improved low-res timer for Windows > > * Fixed LIBZMQ-578 - z85_decode was extremely slow > > * Fixed LIBZMQ-577 - fault in man pages. > > * Fixed LIBZMQ-574 - assertion failure when ran out of system file > handles > > * Fixed LIBZMQ-571 - test_stream failing in some cases > > * Fixed LIBZMQ-569 - Socket server crashes with random client data and > when > > talking to 2.2 versions > > * Fixed LIBZMQ-39 - Bad file descriptor during shutdown > > * Pulled expected failing test_linger.cpp from release > > * Reduced pause time in tests to allow "make check" to run faster > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq static build
On Tue, Nov 19, 2013 at 3:54 PM, Nikola wrote: > Any pointers on how to make it static? > not that I am aware of, but it shouldn't be too hard to search for. > > > On Wed, Nov 20, 2013 at 12:33 AM, MinRK wrote: > >> Yes and no - compiled Python modules are themselves dynamic libraries, >> so the best result is N-1 dynamic loads if you make libzmq static. >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > > -- > God is Real, unless declared Integer. > J. Allan Toogood, FORTRAN programmer > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq static build
Yes and no - compiled Python modules are themselves dynamic libraries, so the best result is N-1 dynamic loads if you make libzmq static. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq static build
> Question is , can I make it so much portable to put it in my project directory and just call import pyzmq and it will work? yes, bdist_wheel or bdist_egg already work this way. Even bdist_dumb should work, I think. > I have version 4.0.1 compiled,isn't there some kinda options to add like -static and compile static version of zmq inside pyzmq. Then I suppose I could just put module in same dir and call it (or not?). It is not static, it is still a dynamic library, just with a relative rpath so it's all shipped together. On Tue, Nov 19, 2013 at 2:56 PM, Nikola wrote: > I have version 4.0.1 compiled,isn't there some kinda options to add like > -static and compile static version of zmq inside pyzmq. > Then I suppose I could just put module in same dir and call it (or not?). > > > On Tue, Nov 19, 2013 at 11:44 PM, Nikola wrote: > >> First tnx for helping out. >> Question is , can I make it so much portable to put it in my project >> directory and just call import pyzmq and it will work? >> >> >> On Tue, Nov 19, 2013 at 7:05 PM, MinRK wrote: >> >>> Add the flag `--zmq=bundled`, and pyzmq will compile libzmq as a Python >>> extension, and ship it. So for a fully portable pyzmq: >>> >>> python setupegg.py bdist_wheel --zmq=bundled >>> >>> should do it. >>> >>> >>> On Tue, Nov 19, 2013 at 1:08 AM, Nikola wrote: >>> >>>> Hi, >>>> >>>> >>>> Is there option to make static build so that pyzmq doesn't need >>>> external zmq lib. >>>> >>>> Any pointers on the options. >>>> >>>> I see that you can specify harcoded rpath but I would like to put >>>> everything in one package and ship it together with app. >>>> >>>> Tnx for help in advance, >>>> N. >>>> >>>> ___ >>>> zeromq-dev mailing list >>>> zeromq-dev@lists.zeromq.org >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>>> >>> >>> ___ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >> >> >> -- >> God is Real, unless declared Integer. >> J. Allan Toogood, FORTRAN programmer >> > > > > -- > God is Real, unless declared Integer. > J. Allan Toogood, FORTRAN programmer > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq static build
Add the flag `--zmq=bundled`, and pyzmq will compile libzmq as a Python extension, and ship it. So for a fully portable pyzmq: python setupegg.py bdist_wheel --zmq=bundled should do it. On Tue, Nov 19, 2013 at 1:08 AM, Nikola wrote: > Hi, > > > Is there option to make static build so that pyzmq doesn't need external > zmq lib. > > Any pointers on the options. > > I see that you can specify harcoded rpath but I would like to put > everything in one package and ship it together with app. > > Tnx for help in advance, > N. > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Assertion failed: buffer_size == header_size (stream_engine.cpp:484)
I have someone reporting: Assertion failed: buffer_size == header_size (stream_engine.cpp:484) when using libzmq 4 in concert with instances of 2.2 and 3.2. I haven't yet been able to reproduce this, but does anyone know a likely culprit for a failure in the v1 handshake in libzmq-4.0? -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Bad file descriptor in rm_fd()
125 consecutive test runs without failure and counting. Thanks guys! On Wed, Nov 6, 2013 at 8:52 AM, Pieter Hintjens wrote: > Ric, this is great! I'll backport the fix to 3.2 and 4.0 once MinRK > confirms it. > On Nov 6, 2013 4:55 PM, wrote: > >> Well, hopefully I haven't broken anything, it looks OK to me. >> >> For the record, a way I found to make this trigger every time is to put a >> 1 second sleep between the unlock and the relock, and a 0.5 second sleep at >> the start of reaper_t::process_reaped (without the delay in >> process_repeaped the second stop gets sent but the mailbox gets closed >> before its processed). >> >> Ric. >> >> >> [image: Inactive hide details for "Pieter Hintjens" ---06/11/2013 >> 03:45:28 PM---On Wed, Nov 6, 2013 at 4:31 PM, > Hintjens" ---06/11/2013 03:45:28 PM---On Wed, Nov 6, 2013 at 4:31 PM, < >> richard_new...@waters.com> wrote: > >> >> From: "Pieter Hintjens" >> To: "ZeroMQ development list" , >> Date: 06/11/2013 03:45 PM >> Subject: Re: [zeromq-dev] Bad file descriptor in rm_fd() >> Sent by: zeromq-dev-boun...@lists.zeromq.org >> -- >> >> >> >> On Wed, Nov 6, 2013 at 4:31 PM, wrote: >> > >> > OK, so investigating this, I think >> https://github.com/zeromq/libzmq/pull/738 may solve the issue. >> > >> > What I think is happening is ctx_t::terminate, we set the state to >> terminating then immediately unlock and relock the slot_sync lock. >> > >> > If the last destroy_socket gets in while we are brief unlocked, both >> destroy_socket and terminate will issue a reaper->stop (), so we will call >> process_stop twice. >> > >> > Anyone know why we do the unlock/relock dance? >> >> >> I'd guess this was an attempt by Sustrik to make the shutdown work >> properly. It's always been a difficult part of the design. >> >> -Pieter >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >> === >> The information in this email is confidential, and is intended solely for >> the addressee(s). >> Access to this email by anyone else is unauthorized and therefore >> prohibited. If you are >> not the intended recipient you are notified that disclosing, copying, >> distributing or taking >> any action in reliance on the contents of this information is strictly >> prohibited and may be unlawful. >> === >> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > <>___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Bad file descriptor in rm_fd()
Once in a while, when running either the IPython or PyZMQ test suite, I still get this error: Bad file descriptor (kqueue.cpp:77) or Bad file descriptor (epoll.cpp:81) Stack trace suggests that this happens when destroying a context: Thread 0: 1 libzmq.3.dylib 0x00010f26b170 zmq::signaler_t::send() + 52 2 libzmq.3.dylib 0x00010f261b2f zmq::object_t::send_stop() + 35 3 libzmq.3.dylib 0x00010f2534a7 zmq::ctx_t::~ctx_t() + 59 4 libzmq.3.dylib 0x00010f253a29 zmq::ctx_t::terminate() + 439 5 libzmq.3.dylib 0x00010f27c071 zmq_ctx_term + 35 Thread 6 Crashed: 0 libsystem_kernel.dylib 0x7fff94a4d866 __pthread_kill + 10 1 libsystem_pthread.dylib 0x7fff8cac835c pthread_kill + 92 2 libsystem_c.dylib 0x7fff97570bba abort + 125 3 libzmq.3.dylib 0x00010f25a9e1 zmq::zmq_abort(char const*) + 9 4 libzmq.3.dylib 0x00010f25d0fe zmq::kqueue_t::kevent_delete(int, short) + 142 5 libzmq.3.dylib 0x00010f25d1b0 zmq::kqueue_t::rm_fd(void*) + 42 6 libzmq.3.dylib 0x00010f2687a3 zmq::reaper_t::process_stop() + 59 7 libzmq.3.dylib 0x00010f26862b zmq::reaper_t::in_event() + 161 8 libzmq.3.dylib 0x00010f25d40c zmq::kqueue_t::loop() + 362 I am still seeing this error once in a while with libzmq-master as of today. I don't think it's a recent regression. A minimal test case is difficult, since it only seems to raise after at least a hundred tests, and only a small fraction of the time even then. Given that it is always late in the process that the assert is hit, I have always assumed that it is FD exhaustion that is causing the problem, but I am not actually sure, and I am fairly careful about cleaning up sockets. Properties of the test suite that sees the issue: - create and destroy many contexts and sockets - the previous test's context should always be destroyed before the next test starts - it is not reliably the same test where the assert is hit I'm afraid I don't know enough about the internals to really tell what's going on here, or figure out why the deleted FD is invalid (maybe it was already closed, and the error should be ignored?). Anyone have insight on what might be causing the problem, or how I might dig deeper into more useful information? -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] shutting down proxy device cleanly with pyzmq
This is fixed in pyzmq master, so nothing more to do but get around to a release. On Fri, Oct 18, 2013 at 9:46 PM, Sideropoulos, Alexander < alexan...@thequery.net> wrote: > 13.1.0 > > I have attached an interactive session log which shows the issue as > clearly as possible. I might just be doing something dumb, though... > > Thanks for looking into it. > --ap > > > On Wed, Oct 16, 2013 at 12:52 AM, MinRK wrote: > >> What version of pyzmq? Can you provide a complete failing example? I >> can't reproduce this in pyzmq master, so maybe I have fixed it since 13.0. >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] shutting down proxy device cleanly with pyzmq
What version of pyzmq? Can you provide a complete failing example? I can't reproduce this in pyzmq master, so maybe I have fixed it since 13.0. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Generating Curve certs in pyzmq
zcert is probably useful for dealing with metadata *about* keypairs, but the main thing that I think is helpful for bindings is the simple wrap of `crypto_box_keypair` so that keypairs can be created without having to link libsodium separately. For instance, if I were to make a cert like czmq's with metadata in Python, I would just use the keypair generation and a Python dict, then serialize to whatever common format (yaml, json, etc.). -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Generating Curve certs in pyzmq
zcert and zauth are in czmq, not libzmq, and pyzmq doesn't wrap czmq. But all of these things can be implemented with socket options, etc., which are up to date with master in pyzmq. Once some of the auth APIs stabilize, I will work out what helpful utilities belong in pyzmq (e.g. easy ZAP threads, etc.). I'm also working on a PR to put curve_keygen into zmq_utils, so it is available to all bindings as a library function, without having to wrap libsodium separately. -MinRK On Sat, Sep 28, 2013 at 2:20 PM, Thomas S Hatch wrote: > I imagine I am missing something basic here, I can see where the certs are > being generated in the examples but I am having a hard time tracking down a > few more things... > > 1. In the Ironhouse example the integrity of the client is verified, but I > am missing where the integrity of the server is verified, can I just add a > zauth_configure_curve call to the client end and store the server's public > cert? > 2. I am having trouble finding the routines in pyzmq to replicate all of > the routines in the blog post. Are the zcert_new routines in pyzmq's git > yet? If not, can someone please point me in the right direction to know > where I can help get them added? > > Thanks again for ZeroMQ and for the new Curve stuff, this is going to help > out immensely! > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] multiprocessing forked children kill zeromq server
On Fri, Aug 30, 2013 at 3:12 PM, Matt Connolly wrote: > I've been looking at a similar scenario using ruby. > > As far as I can tell you cannot even touch anything relating to the > parents context in the child, including closing it. I haven't fully tested > it out but it looks to me like the pipes used for internal communication in > the context will cause messages within he context to go to the wrong place > causing asserts in both child and parent. > That's correct - the Python bindings protect you a little bit, because Context.term and Socket.close are no-ops if the process has been forked (no libzmq API will be called). But no 'real' methods, where you are asking for something to actually happen, are protected for performance reasons. -MinRK > I'm experimenting with a way of terminating the inherited context in the > child process (closing all sockets and pipes and terminating all zmq > threads) but haven't managed to do it with out triggering an assert > somewhere. > > The only thought I had was to use a parent with no zmq context and only > create a context in a child so that there is never any inherited resources. > This may not easily fit your use case though. > > Best, > Matt > > > On 31 Aug 2013, at 4:36 am, MinRK wrote: > > > > You cannot continue to use zmq sockets after a fork - you have to take > care in your application that no sockets created before the fork will be > used by any calls in the child process. > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Is ZMQ_PLAIN authentication supposed to do anything?
On Fri, Aug 30, 2013 at 1:37 PM, Pieter Hintjens wrote: > On Thu, Aug 29, 2013 at 1:32 AM, MinRK wrote: > > > Thanks. By closed, you mean the connecting peer (client) should be > closed, > > or the inner pipe on the server side? What should be the user-visible > > symptoms of failed authentication, both on the client side and the server > > side, if any? I'm looking to add a failed-auth test to test_security, > but it > > is unclear to me what the expected behavior is. Is the symptom only that > > messages sent do not arrive, or should sending a message not succeed in > the > > first place? > > I think the net result of a failed authentication should be the same > as if there was no network connection; no pipes created on either side > of the connection, and no route to or from the unauthenticated client. > Thanks - so as far as the connecter is concerned, it is as if the peer is unavailable, and for the binder, it is as if nobody connected. > > -Pieter > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] multiprocessing forked children kill zeromq server
On Fri, Aug 30, 2013 at 11:50 AM, Townsend, Scott E. (GRC-RTM0)[Vantage Partners, LLC] wrote: MinRK noted that: > > You cannot continue to use zmq sockets after a fork - you have to > take care in your application that no sockets created before the fork will > be [not be] used by any calls in the child process. > > Thanks, that makes sense. But can anyone provide guidance in how to do > that? I'm quite new to zmq, but somewhat versed in Python multiprocessing. > I think there's only one zmq context in use by the main server, and in the > child process I context.term() as soon as I can. That did not fix the > issue (but then I don't know that it should). > > It's entirely possible that there are zmq messages in flight while the > multiprocessing fork is going on, so I can imagine that when a child 'wakes > up' it might think it should process a message that it shouldn't. Is it > possible to 'suspend' zmq while the fork is occurring so I can somehow have > the child cleanup things it shouldn't be using, and then the parent resumes > zmq processing? Or is there some way to mark the sockets such that they > aren't inherited by the child? Or something else I can try? > It depends on how your app works, and what events trigger zmq calls. Without knowing anything about your app or what tasks might access zmq from forked processes, one thing you can do is just make sure certain tasks only happen in the main process: if multiprocessing.current_process().name != 'MainProcess': # just skip this in forked process return zmq_api_call() -MinRK > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] multiprocessing forked children kill zeromq server
You cannot continue to use zmq sockets after a fork - you have to take care in your application that no sockets created before the fork will be used by any calls in the child process. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Is ZMQ_PLAIN authentication supposed to do anything?
On Sun, Aug 25, 2013 at 1:53 PM, Pieter Hintjens wrote: > Failed authentication should cause the socket to be closed. We'll take > a look at this. Thanks for catching it. > Thanks. By closed, you mean the connecting peer (client) should be closed, or the inner pipe on the server side? What should be the user-visible symptoms of failed authentication, both on the client side and the server side, if any? I'm looking to add a failed-auth test to test_security, but it is unclear to me what the expected behavior is. Is the symptom only that messages sent do not arrive, or should sending a message not succeed in the first place? -MinRK > On Sun, Aug 25, 2013 at 8:51 PM, MinRK wrote: > > Hello, > > > > I'm working on [adding support](https://github.com/zeromq/pyzmq/pull/401 > ) > > for 3.3 bits in pyzmq, and I'm testing the authentication mechanisms. I > > translated the [security > > test]( > https://github.com/zeromq/libzmq/blob/master/tests/test_security.cpp) > > to Python and it ran just fine. However, when I checked to confirm that > it > > actually did something, I changed the password to be incorrect - and the > > test *still* ran fine. This means that ZMQ_PLAIN authentication actually > > has no effect, and failed authentication doesn't result in any errors, > and > > messages still send and receive as normal. I made the same changes to > the C > > test with the same result: **failed authentication has no consequence**. > I > > confirmed that `receive_and_process_zap_reply` is indeed returning > `rc=-1` > > and setting `errno=EACCES`, but this does not seem to have any effect on > the > > behavior of the sockets. > > > > I assume this is not intended. Is the implementation supposed to be > complete > > at this point? And what precisely should be the effect of a failed > > authentication (i.e. which calls should raise, block, etc.). > > > > Thanks, > > -MinRK > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Is ZMQ_PLAIN authentication supposed to do anything?
Hello, I'm working on [adding support](https://github.com/zeromq/pyzmq/pull/401) for 3.3 bits in pyzmq, and I'm testing the authentication mechanisms. I translated the [security test]( https://github.com/zeromq/libzmq/blob/master/tests/test_security.cpp) to Python and it ran just fine. However, when I checked to confirm that it actually did something, I changed the password to be incorrect - and the test *still* ran fine. This means that ZMQ_PLAIN authentication actually has no effect, and failed authentication doesn't result in any errors, and messages still send and receive as normal. I made the same changes to the C test with the same result: **failed authentication has no consequence**. I confirmed that `receive_and_process_zap_reply` is indeed returning `rc=-1` and setting `errno=EACCES`, but this does not seem to have any effect on the behavior of the sockets. I assume this is not intended. Is the implementation supposed to be complete at this point? And what precisely should be the effect of a failed authentication (i.e. which calls should raise, block, etc.). Thanks, -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq: using libzmq functions from cython
On Fri, Aug 9, 2013 at 12:00 PM, Michael Haberler wrote: > Tom, > > Am 09.08.2013 um 20:34 schrieb Tom Farnbauer : > > > I'm pretty certain that I'm importing zmq first and it loads just fine. > Then once I try to import my stuff, it fails and tells me that: > > > > ImportError: libzmq.so: cannot open shared object file: No such file or > directory > > > > One interesting thing I noticed is that when I run `ldd pyzmq backend extensions>` libzmq.so is not in the output. It is however > listed when I run it on my own extension module. This happens even if I > link specifically against the libzmq instance that is bundled with zmq. > I've tried playing around with different dlopen settings to no avail (based > on this old info here: > http://muttley.hates-software.com/2006/01/25/c37456e6.html). It appears > that python doesn't use RTLD_GLOBAL by default, but trying to force it > (through sys.setdlopenflags) didn't help... > > > > If I add the zmq package directory to LD_LIBRARY_PATH, it works just > fine. I'm trying not to have to gdb it but I just may have to. > > you might want to look into the rpath linker option - in essence it gives > you a way to bundle a library path with a binary > Ah yes, I forgot that I do this in pyzmq when I ship libzmq with it. The distutils setting is `runtime_library_dirs`, and can include relative paths (relative to the compiled extension, if I recall). -MinRK > > - Michael > > > > > > The code in question is here: > https://github.com/SleepingPills/splice.io/blob/master/splice/receiver.pyx(warning: > it is fairly hacky at the moment) > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq: using libzmq functions from cython
Depending on the nature of your code, it should be okay as long as pyzmq is imported first, which loads the bundled libzmq into RTLD_GLOBAL if there is one. It is meant to support use from Cython, but I confess that I am only aware of one such project (gevent-zeromq), which has since been absorbed into pyzmq itself as zmq.green (and it no longer uses Cython). Can you perhaps share code where you are trying to do this? Perhaps there are things I should cleanup in pyzmq to make this easier. On Fri, Aug 9, 2013 at 1:45 AM, Tom Farnbauer wrote: > Hello, > > I have a somewhat peculiar problem. To get around the GIL, I have a small > cython module that uses libzmq functions to set up some preliminary message > processing before passing on messages to pure python code. The pure python > code uses pyzmq as well. This obviously means that I need to use the zmq > headers and libraries to compile and link the cython module, but at the > same time I have to have pyzmq installed as well. All works fine as long as > libzmq is available on LD_LIBRARY_PATH at runtime, but this is normally not > the case when libzmq is bundled with pyzmq (since it sits in the directory > of pyzmq). > > Does anyone have any advice on what would be the best solution to make the > above thing work? Pyzmq seems to expose libzmq through the libzmq.pxd file, > but I still need the libzmq headers and libraries to be able to build my > python module. If, however, I link against a separate zmq prefix, > libzmq.so.1 will be missing at runtime. > > Will I need to hook into the bundling process somehow to make sure that my > extension module is built specifically against the bundled libzmq? > > The docs mention that pyzmq makes it simple to use its own extension > modules, but also the core libzmq functions from cython (by having the > modular pxd files I presume), but there is little detail in achieving that > (unfortunately). > > Cheers, > > Tom > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] What's the easiest way to install PyZMQ on a Windows box running Python 2.5?
On Wed, Aug 7, 2013 at 3:57 PM, Bryan Richardson wrote: > Thanks for the response MinRK. So are you saying I could build Python from > source on a Windows machine using VS 2008, and then build the pyzmq MSI > from source against the pre-compiled ZeroMQ library for Windows > (libzmq-v90-mt-3_2_3 for the case of VS 2008) and that MSI would install > and run OK on my target machine, which has a Python that was built with VS > 2003? > I'm not sure, but it's worth a try. > > Bottom line is I'm not even able to build a v71 version of libzmq... :-( > > > On Wed, Aug 7, 2013 at 4:41 PM, MinRK wrote: > >> I never made an MSI for 2.5, since it was deprecated before pyzmq had any >> binary installers. >> >> Note that you don't have to build pyzmq on the machine itself, you can >> run `python setupegg.py bdist_msi` (or bdist_egg) on your own Windows >> machine or VM, then pass that MSI to the target machine. >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] What's the easiest way to install PyZMQ on a Windows box running Python 2.5?
I never made an MSI for 2.5, since it was deprecated before pyzmq had any binary installers. Note that you don't have to build pyzmq on the machine itself, you can run `python setupegg.py bdist_msi` (or bdist_egg) on your own Windows machine or VM, then pass that MSI to the target machine. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq poller performance
>From my tests this evening, it looks like zmq_poll does have an appreciable cost (plus the overhead of being wrapped in Python), and will be notably slower than a simple `while True: s.recv()`. But I did find some inefficiencies in the Poller implementation, which should be addressed by PR #381 <https://github.com/zeromq/pyzmq/pull/381>, bringing the difference closer to 4x from 10x. Your tests on my laptop: pure recv: mps: 241121 mps: 419644 mps: 413874 mps: 434421 mps: 428623 mps: 404847 poll (master): mps: 48544 mps: 54143 mps: 51642 mps: 4 mps: 47591 mps: 54218 poll (after 381): mps: 110436 mps: 110876 mps: 104220 mps: 110690 mps: 100544 mps: 110922 -MinRK On Mon, Jun 24, 2013 at 12:18 PM, Brian Knox wrote: > Thanks Min - additionally if I'm incorrect in my assumption that the poll > look in my example code should perform better, or if there's a better way > to do what I'm doing just let me know. > > The example code polls only one socket (which isn't that useful, hah) but > I wanted to keep the test case dirt simple. The actual thing I was working > on has one socket that should receive data at a relatively high rate (say > 50k to 100k msgs/s) , and one socket that receives data at a much lower > rate (say 1 msg/s) (for heartbeating, command and control, etc). > > > Brian > > > On Mon, Jun 24, 2013 at 12:12 PM, Min RK wrote: > >> Thanks for the report, I will look into whether I have introduced a >> performance degradation in the last few iterations. >> >> -MinRK >> >> On Jun 24, 2013, at 11:27, Brian Knox wrote: >> >> > It's been awhile since I've used pyzmq, and I'm running into a >> performance issue using Poller.poll(). >> > >> > With a simple blocking recv() in a while True loop, I get ~ 300k >> messages a second. >> > >> > With the simplest case of the same code using a poll on the socket, I >> get ~ 30k messages a second. >> > >> > (pyzeromq)taotetek@Moya:~/src/performance_example$ python ./receiver.py >> > mps: 226049 >> > mps: 348582 >> > mps: 271728 >> > mps: 300389 >> > mps: 411059 >> > mps: 276749 >> > >> > (pyzeromq)taotetek@Moya:~/src/performance_example$ python >> ./poll_receiver.py >> > mps: 28066 >> > mps: 28116 >> > mps: 29745 >> > mps: 28912 >> > mps: 28679 >> > >> > I don't remember a 10x performance degradation last time I used a >> poller - but perhaps I'm remembering something wrong, or making a painfully >> obvious error in my code .. I've been using czmq from C mostly for the last >> while. >> > >> > I'm using pyzmq from git master, and the latest Cython (0.19.1) with >> libzmq 3.2.3. >> > >> > I've attached the test cases I'm using. >> > >> > >> > ___ >> > zeromq-dev mailing list >> > zeromq-dev@lists.zeromq.org >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] striking difference in performance among ZeroMQ bindings.
I ran your tests on my Linux machine (amd64 Ubuntu 12.04, 12 GB RAM, i7 930), in case you are interested in more numbers (everything from git master) jzmq: It took 2.574 seconds to process 200 messages of size 10 Byte(s) in a batch 777000.7770007771 messages/second It took 1.856 seconds to process 50 messages of size 1.0 KiloByte(s) in a batch 269396.5517241379 messages/second It took 2.673 seconds to process 125000 messages of size 10.0 KiloByte(s) in a batch 46763.93565282454 messages/second jeromq: It took 1.33 seconds to process 200 messages of size 10 Byte(s) in a batch 1503759.3984962406 messages/second It took 2.129 seconds to process 50 messages of size 1.0 KiloByte(s) in a batch 234852.04321277596 messages/second It took 3.014 seconds to process 125000 messages of size 10.0 KiloByte(s) in a batch 41473.12541473126 messages/second PyZMQ (CPython 2.7 + Cython) It took 1.382134 seconds to process 15 messages of size 10 Byte(s) in a batch 108527.827258 messages/second It took 1.627566 seconds to process 15 messages of size 1.0 Kilobyte(s) in a batch 92162.1611658 messages/second It took 2.618599 seconds to process 15 messages of size 10.0 Kilobyte(s) in a batch 57282.539251 messages/second PyZMQ (PyPy 2.0b + CFFI) It took 1.614755 seconds to process 15 messages of size 10 Byte(s) in a batch 92893.3491458 messages/second It took 1.674885 seconds to process 15 messages of size 1.0 Kilobyte(s) in a batch 89558.3875908 messages/second It took 4.030202 seconds to process 15 messages of size 10.0 Kilobyte(s) in a batch 37218.9781058 messages/second Making a few runs, most of these numbers fluctuate by about 10-20% (I couldn't run labview) -Min RK On Thu, May 2, 2013 at 10:08 PM, crocket wrote: > Anyhow, jzmq is somewhat slower than JeroMQ when processing small messages. > The slowness that accompnies 10K messages was partly due to using swap > space on HDD. > > > On Fri, May 3, 2013 at 1:38 PM, crocket wrote: > >> There is a DEALER that sends messages and receives identical messages in >> a batch. >> >> There is the ROUTER that just sends back what it receives to the sender. >> >> I tested JeroMQ, labview-zmq, jzmq, and PyZMQ. >> >> The test source code is at >> https://github.com/crocket/ZeroMQThroughputTest >> >> To summarize the test results on a windows xp sp3 32bit system with 3GB >> RAM, >> for messages of 10 bytes, JeroMQ >> jzmq >>> PyZMQ >>> labview-zmq >> for messages of 1 kilobytes, JeroMQ = jzmq = PyZMQ >>> labview-zmq >> for messages of 10 kilobytes, JeroMQ = jzmq = PyZMQ = labview-zmq >> >> Below are the test results. >> >> Why is labview-zmq so slow when sending small messages? >> It's just a wrapper around libzmq like jzmq and PyZMQ. >> >> 1) JeroMQ >> >> It took 1.59 seconds to process 60 messages of size 10 Byte(s) in a >> batch >> 376411.54 messages/second >> >> It took 35.81 seconds to process 60 messages of size 1.0 KiloByte(s) >> in a batch >> 16754.16 messages/second >> >> It took 304.89 seconds to process 60 messages of size 10.0 >> KiloByte(s) in a batch >> 1967.92 messages/second >> >> 2) jzmq >> >> It took 4.63 seconds to process 60 messages of size 10 Byte(s) in a >> batch >> 129729.73 messages/second >> >> It took 34.30 seconds to process 60 messages of size 1.0 KiloByte(s) >> in a batch >> 17494.75 messages/second >> >> It took 48.36 seconds to process 10 messages of size 10.0 KiloByte(s) >> in a batch >> 2067.82 messages/second >> (If I try to send 600,000 messages of 10 kilobytes in a batch, it says >> "not enough space".) >> >> 3) PyZMQ >> >> It took 6.51 seconds to process 15 messages of size 10Byte(s) in a >> batch >> 23021.58 messages/second >> >> It took 8.78 seconds to process 15 messages of size 1.0Kilobyte(s) in >> a batch >> 17081.85 messages/second >> >> It took 73.78 seconds to process 15 messages of size 10.0Kilobyte(s) >> in a batch >> 2033.03 messages/second >> >> 4) labview-zmq >> >> It took 11.79 seconds to process 4 messages of size 10Byte(s) in a >> batch >> 3390.73 messages/second >> >> It took 13.70 seconds to process 4 messages of size 1.0Kilobyte(s) in >> a batch >> 2919.04 messages/second >> >> It took 20.67 seconds to process 4 messages of size 10.0Kilobyte(s) >> in a batch >> 1935 messages/second >> > > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] "cython inline checkrc" change in PyZMQ 13.0.0 changes handling of EINTR?
To further clarify, there are two cases:
1. no sigint handler registered - ^C absolutely should result in
KeyboardInterrupt. This behavior is intended, and has not changed since
2.1.9.
2. sigint handler registered - in this case, KeyboardInterrupt should *not*
be raised, and the regular ZMQError(EINTR) should happen. This was broken
in 13.0.0, but is fixed in master.
13.0 was a big release, and there are unsurprisingly a few bugs. There
will be a bugfix release around 30 days after the original (about two weeks
from now).
-MinRK
On Mon, Mar 11, 2013 at 2:16 PM, Min RK wrote:
>
>
> On Mar 11, 2013, at 13:13, Jonathan Kamens wrote:
>
> Greetings,
>
> With PyZMQ versions prior to 13.0.0, we were running into problems with
> certain PyZMQ calls getting interrupted by restartable signals (e.g.,
> SIGALRM) used by our application. We fixed this problem like this:
>
> while True:
> try:
> self.context.term()
> except zmq.ZMQError as exc:
> if exc.errno == EINTR:
> log.info('zmq_term interrupted by signal,
> restarting')
> else:
> log.exception('Error terminating ZMQ context')
> raise
> except BaseException as exc:
> log.exception('Error terminating ZMQ context')
> raise
> else:
> break
>
> Note that 0MQ allows term() to be restarted when it gets
> EINTR<http://api.zeromq.org/2-1:zmq-term#toc4>
> .
>
> This worked just fine, but has stopped working in PyZMQ 13.0.0. Now,
> instead of a ZMQError with errno set to EINTR, we are getting a
> KeyboardInterrupt exception. I think this commit to
> PyZMQ<https://github.com/zeromq/pyzmq/commit/3959e4515f86db11f660738df5d36f62478c39e2>is
> the cause.
>
> Questions:
>
>1. Was this change in behavior intentional? If so, it probably should
>be documented in the release notes.
>2. Am I understanding correctly that what needs to be done to fix the
>problem in my app is to do "except KeyboardInterrupt" instead of "except
>zmq.ZMQError as exc" and not bother to check errno?
>
> Thanks in advance for any help you can provide.
>
>
> This was not intentional, and is fixed in master. There should only be a
> difference in behavior if you have non-default signal handler registered.
>
>
> Regards,
>
> Jonathan Kamens
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] pyzmq build fails ?
Yes, that would be appropriate. Since I keep Cython up to date, I simply
didn't notice that I had introduced this dependency. I will investigate,
and make sure the message is better.
On Fri, Feb 22, 2013 at 10:42 AM, xantares 09 wrote:
>
> Yep, that was it, mine was 0.15, now with 0.18.
> The build system should warn about the required version of cython.
> Do you know which one is required ?
>
> So we could do throw something around here:
> try:
> from Cython.Distutils import build_ext as build_ext_c
> cython=True
> except ImportError:
> cython=False
>
> Cheers!
>
>
> --
> From: benjami...@gmail.com
> Date: Fri, 22 Feb 2013 08:54:38 -0800
> To: zeromq-dev@lists.zeromq.org
> Subject: Re: [zeromq-dev] pyzmq build fails ?
>
>
> What Cython version? try updating it.
>
> -MinRK
>
> On Feb 22, 2013, at 0:55, xantares 09 wrote:
>
> Hi
>
> I'm trying to build pyzmq from git.
>
> I installed libzmq from git successfully.
>
> The configure in pyzmq went OK after my patch:
> pyzmq $ python ./setup.py configure --zmq=../libzmq/build/install
> running configure
> cc -c /tmp/timer_create0hukf0.c -o tmp/timer_create0hukf0.o
> cc tmp/timer_create0hukf0.o -o a.out
> tmp/timer_create0hukf0.o: In function `main':
> timer_create0hukf0.c:(.text+0x15): undefined reference to `timer_create'
> collect2: ld returned 1 exit status
> cc -c /tmp/timer_createy_kGTs.c -o tmp/timer_createy_kGTs.o
> cc tmp/timer_createy_kGTs.o -lrt -o a.out
>
> Configure: Autodetecting ZMQ settings...
> Custom ZMQ dir: ../libzmq/build/install
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -I../libzmq/build/install/include -Izmq/utils
> -Izmq/core -Izmq/devices -c build/temp.linux-x86_64-2.7/scratch/vers.c -o
> build/temp.linux-x86_64-2.7/scratch/vers.o
> gcc -pthread build/temp.linux-x86_64-2.7/scratch/vers.o
> -L../libzmq/build/install/lib
> -Wl,-R/home/schueller/projects/libzmq/build/install/lib -lzmq -lrt -o
> build/temp.linux-x86_64-2.7/scratch/vers
> ZMQ version detected: 3.3.0
>
>
> Then the cython build fails:
> pyzmq $ python ./setup.py build
> running build
> running build_py
> running build_ext
> running configure
> cc -c /tmp/timer_createnanPIx.c -o tmp/timer_createnanPIx.o
> cc tmp/timer_createnanPIx.o -o a.out
> tmp/timer_createnanPIx.o: In function `main':
> timer_createnanPIx.c:(.text+0x15): undefined reference to `timer_create'
> collect2: ld returned 1 exit status
> cc -c /tmp/timer_createolhYyX.c -o tmp/timer_createolhYyX.o
> cc tmp/timer_createolhYyX.o -lrt -o a.out
>
> Configure: Autodetecting ZMQ settings...
> Custom ZMQ dir: ../libzmq/build/install
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -I../libzmq/build/install/include -Izmq/utils
> -Izmq/core -Izmq/devices -c build/temp.linux-x86_64-2.7/scratch/vers.c -o
> build/temp.linux-x86_64-2.7/scratch/vers.o
> gcc -pthread build/temp.linux-x86_64-2.7/scratch/vers.o
> -L../libzmq/build/install/lib
> -Wl,-R/home/schueller/projects/libzmq/build/install/lib -lzmq -lrt -o
> build/temp.linux-x86_64-2.7/scratch/vers
> ZMQ version detected: 3.3.0
>
> skipping 'zmq/core/_device.c' Cython extension (up-to-date)
> skipping 'zmq/core/_poll.c' Cython extension (up-to-date)
> skipping 'zmq/core/_version.c' Cython extension (up-to-date)
> cythoning zmq/core/constants.pyx to zmq/core/constants.c
>
> Error compiling Cython file:
>
> ...
>
>
> #-
> # Symbols to export
>
> #-
>
> __all__ = [ key for key in locals().keys() if not key.startswith('_') ]
> ^
>
>
> zmq/core/constants.pyx:190:33: Cannot convert 'size_t (zmq_msg_t *) nogil'
> to Python object
>
> Error compiling Cython file:
>
> ...
> [[[ The same kind of error repeated a bunch of times ]]]
> ...
> zmq/core/constants.pyx:190:33: Cannot convert 'void *(void) nogil' to
> Python object
> building 'zmq.core.constants' extension
> gcc -
Re: [zeromq-dev] [ANN] PyZMQ-13.0.0
On Thu, Feb 21, 2013 at 4:48 PM, Brian Knox wrote: > Nice! > > Does this also include the zmq_proxy accomodations? > I'm not sure what you mean by 'accommodations', but zmq_proxy is available as `zmq.proxy`, and for use in threads / subprocesses via `zmq.devices.ThreadProxy/ProcessProxy`. > > Brian > > > On Thu, Feb 21, 2013 at 7:27 PM, MinRK wrote: > >> pyzmq 13.0.0 released, with bdists bundling libzmq-3.2.2. >> >> Main new feature is support for pypy via added CFFI backend. >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] [ANN] PyZMQ-13.0.0
pyzmq 13.0.0 released, with bdists bundling libzmq-3.2.2. Main new feature is support for pypy via added CFFI backend. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PyZMQ 13.0.0rc1
Minor Cython tweaks for rc3: pip install https://dl.dropbox.com/sh/nsww1t3adru9p3o/OD_MslRnkB/pyzmq-13.0.0-rc3.tar.gz All RCs here: https://www.dropbox.com/sh/nsww1t3adru9p3o/JvkcvlOcxA I expect this to be the last one before 13.0 release this week, unless I hear about issues. -MinRK On Sat, Feb 9, 2013 at 1:28 PM, MinRK wrote: > Some more build fixes for RC2: > > pip install > https://dl.dropbox.com/sh/nsww1t3adru9p3o/E1p1gyK8eG/pyzmq-13.0.0-rc2.tar.gz > > All RCs here: https://www.dropbox.com/sh/nsww1t3adru9p3o/JvkcvlOcxA > > -MinRK > > > On Tue, Feb 5, 2013 at 1:01 PM, Brian Knox wrote: > >> That did the trick! >> >> >> On Tue, Feb 5, 2013 at 4:00 PM, Brian Knox wrote: >> >>> Excellent ,will give it a shot when I get a few moments. Thanks Min! >>> >>> Brian >>> >>> >>> On Tue, Feb 5, 2013 at 1:25 PM, MinRK wrote: >>> >>>> >>>> >>>> On Tue, Feb 5, 2013 at 9:56 AM, Brian Knox wrote: >>>> >>>>> Min - I did not use pip, I tried installing by "hand" via pypy >>>>> setup.py install - but I'll go back and try reinstalling the deps with pip >>>>> - if I still have an issue I'll send more feedback, thanks! >>>>> >>>> >>>> Gotcha, 'by hand' doesn't invoke setuptools, so dependencies are >>>> ignored. That's to be expected. `python setupegg.py install` will run >>>> setup with setuptools, invoking the dependency stuff. >>>> >>>> >>>>> >>>>> >>>>> Brian >>>>> >>>>> >>>>> On Tue, Feb 5, 2013 at 12:34 PM, MinRK wrote: >>>>> >>>>>> >>>>>> >>>>>> On Tue, Feb 5, 2013 at 8:54 AM, Brian Knox wrote: >>>>>> >>>>>>> >>>>>>> Min - I've got it installed now under both python 2 and pypy. The >>>>>>> only issue I'm having is some test code I wrote is not working under >>>>>>> pypy, >>>>>>> saying I'm missing the "py" module (which the pyzmq changelog does say >>>>>>> is >>>>>>> required). I'm not that familiar with pypy at this point .. and >>>>>>> googling >>>>>>> for "pypy py" is a bit of a nightmare (hah). Is "py" something that >>>>>>> should >>>>>>> have built when I built pypy, or is it an external module I need to get >>>>>>> from somewhere? >>>>>>> >>>>>> >>>>>> How did you install the RC? >>>>>> >>>>>> the CFFI backend has a few dependencies: >>>>>> >>>>>> pip install py ctypes-configure cffi >>>>>> >>>>>> They should be pulled in if you install with pip/easy_install. Let >>>>>> me check if I messed that bit up. >>>>>> >>>>>> -MinRK >>>>>> >>>>>> >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> Brian >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 4, 2013 at 8:49 PM, MinRK wrote: >>>>>>> >>>>>>>> I've cut an RC for the next release of pyzmq. >>>>>>>> >>>>>>>> I'll be putting RCs up here: >>>>>>>> https://www.dropbox.com/sh/nsww1t3adru9p3o/JvkcvlOcxA >>>>>>>> >>>>>>>> or you can grab and test this one, with: >>>>>>>> >>>>>>>> pip install >>>>>>>> https://dl.dropbox.com/sh/nsww1t3adru9p3o/7PY0WgxESA/pyzmq-13.0.0-rc1.tar.gz >>>>>>>> >>>>>>>> Highlights: >>>>>>>> >>>>>>>> - PyPy support (via cffi) >>>>>>>> - some build changes that should make it a little more sane (fixes >>>>>>>> issues on HP-UX, and cross-compiling to Android) >>>>>>>> - Updates to zeromq-3.2 support (adds proxy, Context and Frame >>>>>>>> get/set, disconnect, bind, etc.). >>>>>>>> - bundled libzmq will be 3.2.2 (when installed from bdist or libzmq >>>>>>>> is not found at build time) >>>>>>>> >>>>>>>> Change notes: http://zeromq.github.com/pyzmq/changelog.html >>>>>>>> >>>>>>>> Please do let me know if you have any build issues, because that >>>>>>>> part of the code is far from awesome. >>>>>>>> >>>>>>>> -MinRK >>>>>>>> >>>>>>>> ___ >>>>>>>> zeromq-dev mailing list >>>>>>>> zeromq-dev@lists.zeromq.org >>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> ___ >>>>>>> zeromq-dev mailing list >>>>>>> zeromq-dev@lists.zeromq.org >>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>>> >>>>>>> >>>>>> >>>>>> ___ >>>>>> zeromq-dev mailing list >>>>>> zeromq-dev@lists.zeromq.org >>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> >>>>>> >>>>> >>>>> ___ >>>>> zeromq-dev mailing list >>>>> zeromq-dev@lists.zeromq.org >>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>> >>>>> >>>> >>>> ___ >>>> zeromq-dev mailing list >>>> zeromq-dev@lists.zeromq.org >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>>> >>> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PyZMQ 13.0.0rc1
Some more build fixes for RC2: pip install https://dl.dropbox.com/sh/nsww1t3adru9p3o/E1p1gyK8eG/pyzmq-13.0.0-rc2.tar.gz All RCs here: https://www.dropbox.com/sh/nsww1t3adru9p3o/JvkcvlOcxA -MinRK On Tue, Feb 5, 2013 at 1:01 PM, Brian Knox wrote: > That did the trick! > > > On Tue, Feb 5, 2013 at 4:00 PM, Brian Knox wrote: > >> Excellent ,will give it a shot when I get a few moments. Thanks Min! >> >> Brian >> >> >> On Tue, Feb 5, 2013 at 1:25 PM, MinRK wrote: >> >>> >>> >>> On Tue, Feb 5, 2013 at 9:56 AM, Brian Knox wrote: >>> >>>> Min - I did not use pip, I tried installing by "hand" via pypy setup.py >>>> install - but I'll go back and try reinstalling the deps with pip - if I >>>> still have an issue I'll send more feedback, thanks! >>>> >>> >>> Gotcha, 'by hand' doesn't invoke setuptools, so dependencies are >>> ignored. That's to be expected. `python setupegg.py install` will run >>> setup with setuptools, invoking the dependency stuff. >>> >>> >>>> >>>> >>>> Brian >>>> >>>> >>>> On Tue, Feb 5, 2013 at 12:34 PM, MinRK wrote: >>>> >>>>> >>>>> >>>>> On Tue, Feb 5, 2013 at 8:54 AM, Brian Knox wrote: >>>>> >>>>>> >>>>>> Min - I've got it installed now under both python 2 and pypy. The >>>>>> only issue I'm having is some test code I wrote is not working under >>>>>> pypy, >>>>>> saying I'm missing the "py" module (which the pyzmq changelog does say is >>>>>> required). I'm not that familiar with pypy at this point .. and googling >>>>>> for "pypy py" is a bit of a nightmare (hah). Is "py" something that >>>>>> should >>>>>> have built when I built pypy, or is it an external module I need to get >>>>>> from somewhere? >>>>>> >>>>> >>>>> How did you install the RC? >>>>> >>>>> the CFFI backend has a few dependencies: >>>>> >>>>> pip install py ctypes-configure cffi >>>>> >>>>> They should be pulled in if you install with pip/easy_install. Let >>>>> me check if I messed that bit up. >>>>> >>>>> -MinRK >>>>> >>>>> >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Brian >>>>>> >>>>>> >>>>>> On Mon, Feb 4, 2013 at 8:49 PM, MinRK wrote: >>>>>> >>>>>>> I've cut an RC for the next release of pyzmq. >>>>>>> >>>>>>> I'll be putting RCs up here: >>>>>>> https://www.dropbox.com/sh/nsww1t3adru9p3o/JvkcvlOcxA >>>>>>> >>>>>>> or you can grab and test this one, with: >>>>>>> >>>>>>> pip install >>>>>>> https://dl.dropbox.com/sh/nsww1t3adru9p3o/7PY0WgxESA/pyzmq-13.0.0-rc1.tar.gz >>>>>>> >>>>>>> Highlights: >>>>>>> >>>>>>> - PyPy support (via cffi) >>>>>>> - some build changes that should make it a little more sane (fixes >>>>>>> issues on HP-UX, and cross-compiling to Android) >>>>>>> - Updates to zeromq-3.2 support (adds proxy, Context and Frame >>>>>>> get/set, disconnect, bind, etc.). >>>>>>> - bundled libzmq will be 3.2.2 (when installed from bdist or libzmq >>>>>>> is not found at build time) >>>>>>> >>>>>>> Change notes: http://zeromq.github.com/pyzmq/changelog.html >>>>>>> >>>>>>> Please do let me know if you have any build issues, because that >>>>>>> part of the code is far from awesome. >>>>>>> >>>>>>> -MinRK >>>>>>> >>>>>>> ___ >>>>>>> zeromq-dev mailing list >>>>>>> zeromq-dev@lists.zeromq.org >>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>>> >>>>>>> >>>>>> >>>>>> ___ >>>>>> zeromq-dev mailing list >>>>>> zeromq-dev@lists.zeromq.org >>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> >>>>>> >>>>> >>>>> ___ >>>>> zeromq-dev mailing list >>>>> zeromq-dev@lists.zeromq.org >>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>> >>>>> >>>> >>>> ___ >>>> zeromq-dev mailing list >>>> zeromq-dev@lists.zeromq.org >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>>> >>> >>> ___ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PyZMQ 13.0.0rc1
On Tue, Feb 5, 2013 at 9:56 AM, Brian Knox wrote: > Min - I did not use pip, I tried installing by "hand" via pypy setup.py > install - but I'll go back and try reinstalling the deps with pip - if I > still have an issue I'll send more feedback, thanks! > Gotcha, 'by hand' doesn't invoke setuptools, so dependencies are ignored. That's to be expected. `python setupegg.py install` will run setup with setuptools, invoking the dependency stuff. > > > Brian > > > On Tue, Feb 5, 2013 at 12:34 PM, MinRK wrote: > >> >> >> On Tue, Feb 5, 2013 at 8:54 AM, Brian Knox wrote: >> >>> >>> Min - I've got it installed now under both python 2 and pypy. The only >>> issue I'm having is some test code I wrote is not working under pypy, >>> saying I'm missing the "py" module (which the pyzmq changelog does say is >>> required). I'm not that familiar with pypy at this point .. and googling >>> for "pypy py" is a bit of a nightmare (hah). Is "py" something that should >>> have built when I built pypy, or is it an external module I need to get >>> from somewhere? >>> >> >> How did you install the RC? >> >> the CFFI backend has a few dependencies: >> >> pip install py ctypes-configure cffi >> >> They should be pulled in if you install with pip/easy_install. Let me >> check if I messed that bit up. >> >> -MinRK >> >> >>> >>> Thanks! >>> >>> Brian >>> >>> >>> On Mon, Feb 4, 2013 at 8:49 PM, MinRK wrote: >>> >>>> I've cut an RC for the next release of pyzmq. >>>> >>>> I'll be putting RCs up here: >>>> https://www.dropbox.com/sh/nsww1t3adru9p3o/JvkcvlOcxA >>>> >>>> or you can grab and test this one, with: >>>> >>>> pip install >>>> https://dl.dropbox.com/sh/nsww1t3adru9p3o/7PY0WgxESA/pyzmq-13.0.0-rc1.tar.gz >>>> >>>> Highlights: >>>> >>>> - PyPy support (via cffi) >>>> - some build changes that should make it a little more sane (fixes >>>> issues on HP-UX, and cross-compiling to Android) >>>> - Updates to zeromq-3.2 support (adds proxy, Context and Frame get/set, >>>> disconnect, bind, etc.). >>>> - bundled libzmq will be 3.2.2 (when installed from bdist or libzmq is >>>> not found at build time) >>>> >>>> Change notes: http://zeromq.github.com/pyzmq/changelog.html >>>> >>>> Please do let me know if you have any build issues, because that part >>>> of the code is far from awesome. >>>> >>>> -MinRK >>>> >>>> ___ >>>> zeromq-dev mailing list >>>> zeromq-dev@lists.zeromq.org >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>>> >>> >>> ___ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] PyZMQ 13.0.0rc1
On Tue, Feb 5, 2013 at 8:54 AM, Brian Knox wrote: > > Min - I've got it installed now under both python 2 and pypy. The only > issue I'm having is some test code I wrote is not working under pypy, > saying I'm missing the "py" module (which the pyzmq changelog does say is > required). I'm not that familiar with pypy at this point .. and googling > for "pypy py" is a bit of a nightmare (hah). Is "py" something that should > have built when I built pypy, or is it an external module I need to get > from somewhere? > How did you install the RC? the CFFI backend has a few dependencies: pip install py ctypes-configure cffi They should be pulled in if you install with pip/easy_install. Let me check if I messed that bit up. -MinRK > > Thanks! > > Brian > > > On Mon, Feb 4, 2013 at 8:49 PM, MinRK wrote: > >> I've cut an RC for the next release of pyzmq. >> >> I'll be putting RCs up here: >> https://www.dropbox.com/sh/nsww1t3adru9p3o/JvkcvlOcxA >> >> or you can grab and test this one, with: >> >> pip install >> https://dl.dropbox.com/sh/nsww1t3adru9p3o/7PY0WgxESA/pyzmq-13.0.0-rc1.tar.gz >> >> Highlights: >> >> - PyPy support (via cffi) >> - some build changes that should make it a little more sane (fixes issues >> on HP-UX, and cross-compiling to Android) >> - Updates to zeromq-3.2 support (adds proxy, Context and Frame get/set, >> disconnect, bind, etc.). >> - bundled libzmq will be 3.2.2 (when installed from bdist or libzmq is >> not found at build time) >> >> Change notes: http://zeromq.github.com/pyzmq/changelog.html >> >> Please do let me know if you have any build issues, because that part of >> the code is far from awesome. >> >> -MinRK >> >> ___ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] PyZMQ 13.0.0rc1
I've cut an RC for the next release of pyzmq. I'll be putting RCs up here: https://www.dropbox.com/sh/nsww1t3adru9p3o/JvkcvlOcxA or you can grab and test this one, with: pip install https://dl.dropbox.com/sh/nsww1t3adru9p3o/7PY0WgxESA/pyzmq-13.0.0-rc1.tar.gz Highlights: - PyPy support (via cffi) - some build changes that should make it a little more sane (fixes issues on HP-UX, and cross-compiling to Android) - Updates to zeromq-3.2 support (adds proxy, Context and Frame get/set, disconnect, bind, etc.). - bundled libzmq will be 3.2.2 (when installed from bdist or libzmq is not found at build time) Change notes: http://zeromq.github.com/pyzmq/changelog.html Please do let me know if you have any build issues, because that part of the code is far from awesome. -MinRK ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] new libzmq API
On Fri, Jan 18, 2013 at 12:01 AM, Pieter Hintjens wrote: > On Fri, Jan 18, 2013 at 1:16 AM, MinRK wrote: > > > For instance, what would we do if there ever needs to be a context option > > that isn't an int? > > In theory, using macros and a single function makes it easier to write > generic code; in practice since the property types are not consistent, > it's not worth much (and is actually nasty since it hides different > function signatures under a single one). > > So I'd probably go with one function pair (get/set) per option, with > explicit arguments and return types, as we do in CZMQ. It's not much > harder to build the API and it's significantly nicer to use. > > We started this experiment with the zmq_msg_more() function. This is > what I'd do for context properties that aren't ints. > Hm. As a binding maintainer, a proliferation of functions to wrap sounds pretty bad to me. But if zmq_FOO_get/set only deal with ints, and non-int options remain as fairly rare as they have been, maybe them getting their own functions isn't so bad, but it wouldn't be super consistent But if you are talking about a zmq_socket_set_FOO for every single socket option, I am -(uint)(-1) on that. In all, I'm no longer convinced there really is a better situation on this particular issue than what we currently have, because every approach has its drawbacks. > > -Pieter > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
