> > Adding a mutex, even one that is never contended, to the socket will > essentially triple this (one atomic CAS to acquire the mutex, one atomic > CAS to put the message on the pipe, one atomic CAS to release the mutex).
This is a bit of a "blue sky" view of the cost of acquiring a mutex. For the adventurous of spirit, chase down the call path of pthread_mutex sometime in GLIBC. It is substantially more involved than a single pair of 'lock; cmpxchg' instructions, but it tries really hard to make that the rough cost of the happy path. On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers <rodg...@twrodgers.com> wrote: > Having thought about this for a couple of more days, I want to at least > take a stab at arguing against "threadsafe" sockets - > > libzmq's thread safety guarantees, to me anyway, are very clear, > unsurprising and non-controversial - I cannot share a socket with another > thread without a full fence. > > The kinds of systems I generally build have very strict requirements on > overall latency, to the point that most of my networking IO is done through > kernel-bypass libraries and NICs that support this, for raw TCP and UDP > multicast. The latency sensitive code that does IO is in it's own thread, > with exclusive access to the NICs which are accessed via kernel bypass. > Coordination with other threads in the same process is done via inproc pair > sockets. Pair sockets + very small messages (small enough that libzmq does > not need to perform allocation) provide a very nice interface to a lock > free queue with low overhead using a single atomic CAS operation. Atomic > operations are cheap, but they are not free (~30 clocks on x86). Adding a > mutex, even one that is never contended, to the socket will essentially > triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the > message on the pipe, one atomic CAS to release the mutex). I would like to > have the option to avoid this. > > If a wrapper wants thread safe sockets to enable certain use-cases that > may be more idiomatic for the language in question, it can provide the full > fence. AZMQ <https://github.com/zeromq/azmq> does exactly this by > default, but you have the option to opt out of it. It does this because > Boost Asio by default allows it's sockets to be used from multiple threads > for async IO and I need to guard more than just exclusive access to the > ZeroMQ socket a the fence in this case. Putting a mutex inside of the > libzmq socket, essentially doubles the overhead for no gain in useful > functionality and runs completely counter to one of C and C++'s overarching > principles: "don't pay for what you don't use". > > If a class of apps really demands short lived exclusive access to a > socket, provide a pool abstraction. The pool is thread safe, obtain a > socket, perform I/O in a single thread, return the socket to the pool. > > On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier < > pelletier.mic...@gmail.com> wrote: > >> >> >> On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens <p...@imatix.com> wrote: >> >>> The discussion about thread safety was quite short iirc, though that >>> contributor did discuss other things... at length. I merged his >>> "thread safe socket" change rapidly, then we reverted it after a few >>> days, and he disappeared. It was rather brute force and I suspect did >>> not work at all, it simply wrapped all accesses to the socket >>> structure in mutexes. No discussion at the time of multipart data and >>> atomic send/recv. >>> >> >> My memory of the conversation at the time is pretty dim, I agree the >> changes were ugly and untested and the contributor was difficult to reason >> with and seemed to want to make the changes based on no real need at all. >> >> >>> >>> As for socket safety, I've no strong opinion. I see that many people >>> expect that to work and hit errors when it doesn't. I see that nanomsg >>> has threadsafe sockets and no multipart. I see that sharing sockets >>> across threads would make some actor models simpler, which is nice. >>> >> >> This is the classic problem with thread safe anything. Threads are hard, >> and there is a balance between the complexity of making a thread safe >> construct and the skill required of a programmer to use "unsafe" construct >> in a safe manner. I still think if the concrete problem is very short >> lived threads causing slow joiner problems, then the simple solution is >> pools (troupes of actors?). >> >> -Michel >> >> >>> >>> >>> >>> >>> >>> >>> On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier >>> <pelletier.mic...@gmail.com> wrote: >>> > I think Brian has some good points here, there are numerous unrelated >>> issues >>> > being discussed in this thread. >>> > >>> > A few points that I have: >>> > >>> > Multi part messages have also bothered me. However as a Python >>> programmer i >>> > see Min's points about the expense of buffer creation. To my knowledge >>> > zproto does not (yet) have Python generation support either, or maybe >>> > something like generated cffi or ctypes wrappers around the zproto >>> generated >>> > C code. That being said there are a variety of serialization >>> libraries for >>> > Python. With some ctypes and mmap magic they can also be done "zero >>> copy" >>> > but it's not pretty: >>> > >>> > https://gist.github.com/michelp/7522179 >>> > >>> > Multi part envelops are also how multi-hop routing is done. I don't >>> see how >>> > the new ideas handle that. I don't think we can just say "multi hop >>> routing >>> > is bad" and get rid of it. >>> > >>> > "Thread safe" sockets do not sound appealing to me. We did that, had >>> a long >>> > and contentious discussion with the person championing them, merged >>> it, then >>> > reverted it and that person is now no longer in the community. Pieter >>> was >>> > the most vocal opponent to them then and now he wants them back. Of >>> course, >>> > anyone can change their mind, but the only current argument I hear now >>> for >>> > them though is improving the performance of short lived threads, but >>> that >>> > can be solved, more correctly in my opinion, with thread or connection >>> > pools. If you creating and tearing down threads that rapidly then you >>> have >>> > two problems. >>> > >>> > -Michel >>> > >>> > On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox <bk...@digitalocean.com> >>> wrote: >>> >> >>> >> After catching up on this thread, I feel like at least three problems >>> are >>> >> being conflated into one problem. I'll state what I see being >>> discussed >>> >> from my perspective: >>> >> >>> >> 1. "Using multi part messages as a way to route to clients from a >>> router >>> >> socket is overly complicated and not how new users expect things to >>> work" >>> >> >>> >> 2. "Using multi part messages for message serialization is costly, and >>> >> potentially confusing to others." >>> >> >>> >> 3. "ZeroMQ sockets are not thread safe." >>> >> >>> >> While on an implementation level these three problems may be related, >>> on a >>> >> conceptual level I don't see them as related. I may agree with some >>> of >>> >> these problem statements and not others. >>> >> >>> >> For me, my first priority is to always have the ability to get back a >>> nice >>> >> agnostic blob of bytes from ZeroMQ. This makes it easy to make >>> ZeroMQ >>> >> socket use compatible with standard io interfaces in Go. Structure >>> for what >>> >> is contained in those bytes is a concern of a different layer. >>> Sometimes I >>> >> use zproto for this (which I like), and other times I don't. >>> >> >>> >> As a demonstration that the problems are different problems, I solved >>> #1 >>> >> for myself in goczmq without addressing anything else. >>> >> >>> >> I would assert some of the confusion in this discussion is that we're >>> >> talking about multiple problem statements at the same time. >>> >> >>> >> Cheers - and it was great meeting people this week! >>> >> >>> >> Brian >>> >> >>> >> >>> >> >>> >> >>> >> On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens <p...@imatix.com> >>> wrote: >>> >>> >>> >>> Ironically, in my testing of high message rate), allowing multipart >>> >>> creates significant costs. Multipart is just one way of getting >>> >>> zero-copy, and even then only works on writing, not reading. >>> >>> >>> >>> For high performance brokers like Malamute I'd *really* like to be >>> >>> moving blobs around instead of lists of blobs. >>> >>> >>> >>> >>> >>> On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin < >>> gr...@pointillistic.com> >>> >>> wrote: >>> >>> > M> Perhaps it is because I spend my days in a higher level language >>> >>> > M> like Python, but zproto is not an attractive option. >>> >>> > >>> >>> > Same here. I will read in detail about it shortly, but it may not >>> make >>> >>> > it into my toolbox as a multipart replacement. Multipart looked >>> very >>> >>> > cool when I found 0MQ, but I've ended up not using it much. I'm not >>> >>> > doing high performance stuff though. Simplicity and ease of use are >>> >>> > tops on my list. >>> >>> > >>> >>> > -- Gregg >>> >>> > >>> >>> > _______________________________________________ >>> >>> > zeromq-dev mailing list >>> >>> > zeromq-dev@lists.zeromq.org >>> >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >>> _______________________________________________ >>> >>> zeromq-dev mailing list >>> >>> zeromq-dev@lists.zeromq.org >>> >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> zeromq-dev mailing list >>> >> zeromq-dev@lists.zeromq.org >>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >>> > >>> > >>> > _______________________________________________ >>> > zeromq-dev mailing list >>> > zeromq-dev@lists.zeromq.org >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> > >>> _______________________________________________ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >> >> _______________________________________________ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
