Re: [zeromq-dev] Python Bindings for Zyre
The API is going to be very similar, yes. Any differences will be minor. On Sun, Feb 22, 2015 at 7:44 AM, Utsav Drolia utsavdro...@gmail.com wrote: Would the API be the same for both projects? That way I can start with pyre and move to python bindings for zyre when they mature. On Feb 21, 2015 7:15 PM, Pieter Hintjens p...@imatix.com wrote: Zyre has (since about a day) Python bindings via the zproject generator that Zyre uses. I'm not sure they are 100% complete yet. See latest commits to zproject. Pyre is an interesting alternative. On Sat, Feb 21, 2015 at 11:26 PM, Utsav Drolia utsavdro...@gmail.com wrote: Hi, There is a project - pyre (https://github.com/zeromq/pyre) which is a python implementation of the ZRE protocol. Will zyre directly support python bindings or should one use pyre instead? Thanks! Utsav ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Python Bindings for Zyre
Zyre has (since about a day) Python bindings via the zproject generator that Zyre uses. I'm not sure they are 100% complete yet. See latest commits to zproject. Pyre is an interesting alternative. On Sat, Feb 21, 2015 at 11:26 PM, Utsav Drolia utsavdro...@gmail.com wrote: Hi, There is a project - pyre (https://github.com/zeromq/pyre) which is a python implementation of the ZRE protocol. Will zyre directly support python bindings or should one use pyre instead? Thanks! Utsav ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Python Bindings for Zyre
Hi, There is a project - pyre (https://github.com/zeromq/pyre) which is a python implementation of the ZRE protocol. Will zyre directly support python bindings or should one use pyre instead? Thanks! Utsav ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Python Bindings for Zyre
Would the API be the same for both projects? That way I can start with pyre and move to python bindings for zyre when they mature. On Feb 21, 2015 7:15 PM, Pieter Hintjens p...@imatix.com wrote: Zyre has (since about a day) Python bindings via the zproject generator that Zyre uses. I'm not sure they are 100% complete yet. See latest commits to zproject. Pyre is an interesting alternative. On Sat, Feb 21, 2015 at 11:26 PM, Utsav Drolia utsavdro...@gmail.com wrote: Hi, There is a project - pyre (https://github.com/zeromq/pyre) which is a python implementation of the ZRE protocol. Will zyre directly support python bindings or should one use pyre instead? Thanks! Utsav ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] thread safe sockets and ZMQ_FD
It looks like the recently added code for supporting thread safe sockets does not support returning the signalling FD for a socket that is marked thread safe. How would these socket types be used in a polling reactor then? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
Specifically, something like epoll or kqueues? On Sat, Feb 21, 2015 at 9:27 AM, Thomas Rodgers rodg...@twrodgers.com wrote: It looks like the recently added code for supporting thread safe sockets does not support returning the signalling FD for a socket that is marked thread safe. How would these socket types be used in a polling reactor then? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
Currently the thread safe sockets doesn't support polling on multiple sockets as stated in the PR: https://github.com/zeromq/libzmq/pull/1349 I'm planning to solve that by epoll like API where you create a poll and then add sockets to it. The poll type will create a file descriptor that will be added to each of the associated thread safe sockets. The mailbox_safe type already ( https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have a method called add signaler which is actually a file descriptor. On Sat, Feb 21, 2015 at 5:27 PM, Thomas Rodgers rodg...@twrodgers.com wrote: It looks like the recently added code for supporting thread safe sockets does not support returning the signalling FD for a socket that is marked thread safe. How would these socket types be used in a polling reactor then? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
You will be able to create a signaler which is a file descriptor, associate it with the socket and add it to the epoll or kqueue On Sat, Feb 21, 2015 at 5:32 PM, Thomas Rodgers rodg...@twrodgers.com wrote: Specifically, something like epoll or kqueues? On Sat, Feb 21, 2015 at 9:27 AM, Thomas Rodgers rodg...@twrodgers.com wrote: It looks like the recently added code for supporting thread safe sockets does not support returning the signalling FD for a socket that is marked thread safe. How would these socket types be used in a polling reactor then? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
I will hold off until you've fleshed out those details, but it sounds like the API for getting the signaling FD will be different from what everything else uses currently, which means there will be a class of socket types that I will need to special case to support in my bindings. Is there anyway to just make that API get_sockopt() for ZMQ_FD but make the FD returned the 'new' signaling one? On Saturday, February 21, 2015, Doron Somech somdo...@gmail.com wrote: Currently the thread safe sockets doesn't support polling on multiple sockets as stated in the PR: https://github.com/zeromq/libzmq/pull/1349 I'm planning to solve that by epoll like API where you create a poll and then add sockets to it. The poll type will create a file descriptor that will be added to each of the associated thread safe sockets. The mailbox_safe type already ( https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have a method called add signaler which is actually a file descriptor. On Sat, Feb 21, 2015 at 5:27 PM, Thomas Rodgers rodg...@twrodgers.com javascript:_e(%7B%7D,'cvml','rodg...@twrodgers.com'); wrote: It looks like the recently added code for supporting thread safe sockets does not support returning the signalling FD for a socket that is marked thread safe. How would these socket types be used in a polling reactor then? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org javascript:_e(%7B%7D,'cvml','zeromq-dev@lists.zeromq.org'); http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
s/comdition/condition/ On Sat, Feb 21, 2015 at 10:45 AM, Thomas Rodgers rodg...@twrodgers.com wrote: Which all brings us to my next question - Can we just move on to -std=c++11 for future libzmq versions? The big 3 compilers (well mostly, Microsoft still presents a few issues) support C++11 at this point. Many of the issues below would just 'go away' with the use of std::mutex, std::unique_lock, and std::comdition_variable. On Saturday, February 21, 2015, Bjorn Reese bre...@mail1.stofanet.dk wrote: On 02/21/2015 04:44 PM, Doron Somech wrote: (https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have I had a quick look at this class... The workaround in the destructor is not thread-safe. Another thread may enter and wait between the sync-unlock() and the end of the destructor scope. You need to add a state variable to prevent this from happening. Furthermore, if another thread is waiting on the condition variable, then the mutex is unlocked while it is waiting. You need to notify (broadcast) the condition variable to wake up the other thread and get it out of the class. You will most likely need to to add a reference count to keep track of how many pending threads are waiting in order to know when it is safe to exit the destructor. The use of sync-lock() and sync-unlock() is not exception safe. I suggest that you use a scoped lock instead of the explicit calls. Read this for inspiration: http://en.cppreference.com/w/cpp/thread/unique_lock The code does not handle spurious wakeups from the condition variable. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
Destroying a socket is still not thread safe, so the user still need to synchronize that On Feb 21, 2015 6:36 PM, Bjorn Reese bre...@mail1.stofanet.dk wrote: On 02/21/2015 04:44 PM, Doron Somech wrote: (https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have I had a quick look at this class... The workaround in the destructor is not thread-safe. Another thread may enter and wait between the sync-unlock() and the end of the destructor scope. You need to add a state variable to prevent this from happening. Furthermore, if another thread is waiting on the condition variable, then the mutex is unlocked while it is waiting. You need to notify (broadcast) the condition variable to wake up the other thread and get it out of the class. You will most likely need to to add a reference count to keep track of how many pending threads are waiting in order to know when it is safe to exit the destructor. The use of sync-lock() and sync-unlock() is not exception safe. I suggest that you use a scoped lock instead of the explicit calls. Read this for inspiration: http://en.cppreference.com/w/cpp/thread/unique_lock The code does not handle spurious wakeups from the condition variable. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
On Sat, Feb 21, 2015 at 5:45 PM, Thomas Rodgers rodg...@twrodgers.com wrote: Can we just move on to -std=c++11 for future libzmq versions? The big 3 compilers (well mostly, Microsoft still presents a few issues) support C++11 at this point. I'd guess Yes, though it's worth asking the list. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
The one need to dispose it. My plan was to support zmq_poll by create signaler on each call. On Feb 21, 2015 6:21 PM, Thomas Rodgers rodg...@twrodgers.com wrote: I will hold off until you've fleshed out those details, but it sounds like the API for getting the signaling FD will be different from what everything else uses currently, which means there will be a class of socket types that I will need to special case to support in my bindings. Is there anyway to just make that API get_sockopt() for ZMQ_FD but make the FD returned the 'new' signaling one? On Saturday, February 21, 2015, Doron Somech somdo...@gmail.com wrote: Currently the thread safe sockets doesn't support polling on multiple sockets as stated in the PR: https://github.com/zeromq/libzmq/pull/1349 I'm planning to solve that by epoll like API where you create a poll and then add sockets to it. The poll type will create a file descriptor that will be added to each of the associated thread safe sockets. The mailbox_safe type already ( https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have a method called add signaler which is actually a file descriptor. On Sat, Feb 21, 2015 at 5:27 PM, Thomas Rodgers rodg...@twrodgers.com wrote: It looks like the recently added code for supporting thread safe sockets does not support returning the signalling FD for a socket that is marked thread safe. How would these socket types be used in a polling reactor then? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
On 02/21/2015 04:44 PM, Doron Somech wrote: (https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have I had a quick look at this class... The workaround in the destructor is not thread-safe. Another thread may enter and wait between the sync-unlock() and the end of the destructor scope. You need to add a state variable to prevent this from happening. Furthermore, if another thread is waiting on the condition variable, then the mutex is unlocked while it is waiting. You need to notify (broadcast) the condition variable to wake up the other thread and get it out of the class. You will most likely need to to add a reference count to keep track of how many pending threads are waiting in order to know when it is safe to exit the destructor. The use of sync-lock() and sync-unlock() is not exception safe. I suggest that you use a scoped lock instead of the explicit calls. Read this for inspiration: http://en.cppreference.com/w/cpp/thread/unique_lock The code does not handle spurious wakeups from the condition variable. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
Which all brings us to my next question - Can we just move on to -std=c++11 for future libzmq versions? The big 3 compilers (well mostly, Microsoft still presents a few issues) support C++11 at this point. Many of the issues below would just 'go away' with the use of std::mutex, std::unique_lock, and std::comdition_variable. On Saturday, February 21, 2015, Bjorn Reese bre...@mail1.stofanet.dk wrote: On 02/21/2015 04:44 PM, Doron Somech wrote: (https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have I had a quick look at this class... The workaround in the destructor is not thread-safe. Another thread may enter and wait between the sync-unlock() and the end of the destructor scope. You need to add a state variable to prevent this from happening. Furthermore, if another thread is waiting on the condition variable, then the mutex is unlocked while it is waiting. You need to notify (broadcast) the condition variable to wake up the other thread and get it out of the class. You will most likely need to to add a reference count to keep track of how many pending threads are waiting in order to know when it is safe to exit the destructor. The use of sync-lock() and sync-unlock() is not exception safe. I suggest that you use a scoped lock instead of the explicit calls. Read this for inspiration: http://en.cppreference.com/w/cpp/thread/unique_lock The code does not handle spurious wakeups from the condition variable. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org javascript:; http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
in addition to not being exception safe, all of the explicit entry and early exit from the mutex is a fully loaded footgun for some future developer working on this code. RAII is a core idiom of C++ and scoped locks/guards are the only sensible and safe way to write this sort of code so that will be both exception safe and have a chance of not biting someone down the road. Also, std::condition_variable has explicit support for handling spurious wakeups. see - http://en.cppreference.com/w/cpp/thread/condition_variable/wait The second variant of wait() takes a predicate which can be used to ignore spurious wakeups. On Sat, Feb 21, 2015 at 10:51 AM, Thomas Rodgers rodg...@twrodgers.com wrote: s/comdition/condition/ On Sat, Feb 21, 2015 at 10:45 AM, Thomas Rodgers rodg...@twrodgers.com wrote: Which all brings us to my next question - Can we just move on to -std=c++11 for future libzmq versions? The big 3 compilers (well mostly, Microsoft still presents a few issues) support C++11 at this point. Many of the issues below would just 'go away' with the use of std::mutex, std::unique_lock, and std::comdition_variable. On Saturday, February 21, 2015, Bjorn Reese bre...@mail1.stofanet.dk wrote: On 02/21/2015 04:44 PM, Doron Somech wrote: (https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have I had a quick look at this class... The workaround in the destructor is not thread-safe. Another thread may enter and wait between the sync-unlock() and the end of the destructor scope. You need to add a state variable to prevent this from happening. Furthermore, if another thread is waiting on the condition variable, then the mutex is unlocked while it is waiting. You need to notify (broadcast) the condition variable to wake up the other thread and get it out of the class. You will most likely need to to add a reference count to keep track of how many pending threads are waiting in order to know when it is safe to exit the destructor. The use of sync-lock() and sync-unlock() is not exception safe. I suggest that you use a scoped lock instead of the explicit calls. Read this for inspiration: http://en.cppreference.com/w/cpp/thread/unique_lock The code does not handle spurious wakeups from the condition variable. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
Hi all, Be careful with requiring C++11 constructs. Some environments may need to use older compilers. For example, building PyZMQ from source for Python 2.7 (and some early Python 3 releases) requires building with VS2008, which does not support C++11. Cheers, André On Sat, Feb 21, 2015 at 1:17 PM, Doron Somech somdo...@gmail.com wrote: Destroying a socket is still not thread safe, so the user still need to synchronize that On Feb 21, 2015 6:36 PM, Bjorn Reese bre...@mail1.stofanet.dk wrote: On 02/21/2015 04:44 PM, Doron Somech wrote: (https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have I had a quick look at this class... The workaround in the destructor is not thread-safe. Another thread may enter and wait between the sync-unlock() and the end of the destructor scope. You need to add a state variable to prevent this from happening. Furthermore, if another thread is waiting on the condition variable, then the mutex is unlocked while it is waiting. You need to notify (broadcast) the condition variable to wake up the other thread and get it out of the class. You will most likely need to to add a reference count to keep track of how many pending threads are waiting in order to know when it is safe to exit the destructor. The use of sync-lock() and sync-unlock() is not exception safe. I suggest that you use a scoped lock instead of the explicit calls. Read this for inspiration: http://en.cppreference.com/w/cpp/thread/unique_lock The code does not handle spurious wakeups from the condition variable. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
Why would these be not compile-able with newer versions of VC (I believe VS2008 Goes EOL in July of this year)? Note, changing the libzmq to build with -std=c++11 would not change the C89 compatibility of it's public API. On Saturday, February 21, 2015, André Caron andre.l.ca...@gmail.com wrote: Hi all, Be careful with requiring C++11 constructs. Some environments may need to use older compilers. For example, building PyZMQ from source for Python 2.7 (and some early Python 3 releases) requires building with VS2008, which does not support C++11. Cheers, André On Sat, Feb 21, 2015 at 1:17 PM, Doron Somech somdo...@gmail.com javascript:_e(%7B%7D,'cvml','somdo...@gmail.com'); wrote: Destroying a socket is still not thread safe, so the user still need to synchronize that On Feb 21, 2015 6:36 PM, Bjorn Reese bre...@mail1.stofanet.dk javascript:_e(%7B%7D,'cvml','bre...@mail1.stofanet.dk'); wrote: On 02/21/2015 04:44 PM, Doron Somech wrote: (https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have I had a quick look at this class... The workaround in the destructor is not thread-safe. Another thread may enter and wait between the sync-unlock() and the end of the destructor scope. You need to add a state variable to prevent this from happening. Furthermore, if another thread is waiting on the condition variable, then the mutex is unlocked while it is waiting. You need to notify (broadcast) the condition variable to wake up the other thread and get it out of the class. You will most likely need to to add a reference count to keep track of how many pending threads are waiting in order to know when it is safe to exit the destructor. The use of sync-lock() and sync-unlock() is not exception safe. I suggest that you use a scoped lock instead of the explicit calls. Read this for inspiration: http://en.cppreference.com/w/cpp/thread/unique_lock The code does not handle spurious wakeups from the condition variable. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org javascript:_e(%7B%7D,'cvml','zeromq-dev@lists.zeromq.org'); http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org javascript:_e(%7B%7D,'cvml','zeromq-dev@lists.zeromq.org'); http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] thread safe sockets and ZMQ_FD
You are right. I will change it to use scoped lock. On Sat, Feb 21, 2015 at 6:36 PM, Bjorn Reese bre...@mail1.stofanet.dk wrote: On 02/21/2015 04:44 PM, Doron Somech wrote: (https://github.com/zeromq/libzmq/blob/master/src/mailbox_safe.hpp) have I had a quick look at this class... The workaround in the destructor is not thread-safe. Another thread may enter and wait between the sync-unlock() and the end of the destructor scope. You need to add a state variable to prevent this from happening. Furthermore, if another thread is waiting on the condition variable, then the mutex is unlocked while it is waiting. You need to notify (broadcast) the condition variable to wake up the other thread and get it out of the class. You will most likely need to to add a reference count to keep track of how many pending threads are waiting in order to know when it is safe to exit the destructor. The use of sync-lock() and sync-unlock() is not exception safe. I suggest that you use a scoped lock instead of the explicit calls. Read this for inspiration: http://en.cppreference.com/w/cpp/thread/unique_lock The code does not handle spurious wakeups from the condition variable. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
