Hi *!
I just read through this thread and would like to share some thoughts about several things in discussion: I picked ZMQ to have a straight forward lib that is easy to handle and makes it easy to both stick to high level things (application concerned) as well as going into details where needed (adressing, routing etc.). Of course, it's discussable to have routing data somehow "mixed" with application data (adress being just another frame >>next to<< the net load and thus having the need to "find" the real data between other things). But overall it is a slender approach that works well. I apprechiate the idea of improving the separation between technical data and semantic data by (as I understand it) replacing technical frames (routing etc.) by metadata structures, but what I'm really afraid of is the fealing, that you are going to blow up every logic that currently deals with ZMQ3 and 4, replacing things by a new API completely incompatible in it's logical structures with old APIs. I had (and still have) heavy work with migrating from the antique ZMQ2 to actual ZMQ4 lib, just because the API allready switched in some integral parts. E.g. it's a constant question in background, if the used function will take ownership and free up data after sending or if there will be a (new) leak in my code (using C++). I wrote a (rather large) wrapper arround every network logic just to be able to easily deal with another API switch. E.g. in ZMQ2 I could fetch real data and adressing in separate reliable functions in zmq itself. In ZMQ4 I have to tell adress and data appart by myself. Easy and short code will break as soon as the routing changes somehow. Flexible code will be complicated and timeconsuming and I intensionally took ZMQ because it does some valuable things for me (e.g. getting routing frames separated from application data). What I did not really like, was the abstraction of ZMQ wrappers in the form of void* everywhere. This increases the issues concerning type safety (compiler wont complain when accidencially having an missing * or an ampersand too much). For being able to deal with application data (high level) as well as with technical data (routing to several workers etc.) I had to use different wrapper APIs (native libzmq as well as CZMQ, but CZMQ has "hidden" internal structures and I did not find ways to take classes from CZMQ and work on them using libzmq (e.g. because of that hiding void* _ in zmsg_t). Indeed, I was thinking about investing even more time in my own wrapper (having funcs like address(int), data(), size(), routing_path_length() and others) to make it even more foolproove, but after having read this thread, I'm afraid, that I'm just about to do a thing, that you are also just planing, expecting, that you will do it in another way and my wrapper breaks again. Long talk - short sense: IMO it's the most valuable thing for every framework, library and whatever to have a STABLE API, that users can rely upon to work for several versions. Call it queer, but I expect a library of such a high quality (as zmq definitely is) to be API-stable, requiring just to link the new lib, do a test and be productive again. Why do I write all this? As I wrote, I'm still busy in switching from a (working) ZMQ2 to ZMQ4, which somehow does not work stable in our system. Propably, this is NO bug in zmq, but the changed API from 2 to 4 required such a big deal of changes, that I now cannot find my bugs. The network stuff was developed and tested long ago having a rather slender skeleton. Now, there are many thousand lines (5 digits!) of code, I allready did a complete leak checking etc. inside this complex system and still it does loose messages (upon MY bugs surely). Writing a minimal test case does not show up the message loss. In short again: switching the API required too much chances in a complex system, so when the API changes again even in its logic, I'll trhink about going to another MQ system and rewrite the whole wrapper again, but hoping it's the final. ... but I dont want to ! ZMQ is great, but please do not break application code interface again! Last thing: I did not yet get the ides of making socket thread safe. Your whole concept was build atop the statement: "we don't need thread safety, because we separate via sockets: each thread it's own socket, so no thread battles". And I think you were right with that. Why now change this for such a high price (breaking the API logic)? ^5 sven Am 2015-02-09 10:02, schrieb Pieter Hintjens: > James, thanks a lot for this report. It's the kind of data we don't > get often enough. > > For me it's become clear over the last years that very few people can > properly write messaging layers. Actually I've known for decades that > only a small slice of programmers are fit to write platform code. > ZeroMQ promised to democratize this, yet it can't solve human > limitations. OTOH we've taught thousands of people to write brokers > (quite amazing, really). On the other hand, we've probably enabled > billions of lines of horrid code. > > My answer is to provide more pre-packaged pieces like Zyre and > Malamute, and to slowly steer people away from writing their own > brokers and stacks. And if they do, to be less creative about it, and > better equipped with tools like zproto that do the work for them. (And > yes, people then abuse zproto, and we need more answers for that...) > > -Pieter > > On Mon, Feb 9, 2015 at 1:57 AM, James Gatannah <james.gatan...@gmail.com> wrote: > >> This ties into the heart of some related discussions we've been having at work recently. We have a bunch of new hires who are trying to come to grips with a messaging protocol that's evolved over the past year or so as we've gotten more familiar with 0mq and adapted it to our changing needs. It's really just meandering commentary about the way I perceive the proposal as it relates to the way we're currently using the library (in C++, C#, python, and clojure via jzmq). It may be a complete waste of time, or it may be a validation of the proposed changes. Our current incarnation looks something like: Frame 1: address header (originally for pub/sub) Frame 2: bundle of serialized data (with a bunch of HTTP-style metadata, like Verb, URI, headers, the "actual" message body etc). Normal router/dealer identity frame sequences. New team members would very much like to split that apart into 7 or 8 more frames, just so the "actual" message body is totally isolated. We could probably accomplish something similar using zproto, though I'd much rather just cram everything into one single frame. We can't do either, mainly due to time constraints around changing the C++ layer. Pieter Hintjens <p...@imatix.com> wrote: >> >>> I think we can either get the routing id as a message property, and/or add a "reply to message" semantic that does this implicitly. The second is easier for simple servers, the first is better for realistic servers. Since it only applies to server sockets, I think we would benefit from a zserver_reply () method (or similar). >> For our app, we have "clients" that have to establish the initial connection (they use dealer sockets. The central messaging hub has the router). After that, those "clients" really switch to being "servers." We're attaching the identity frame(s) to each incoming message so we can use them to send the response. We're also tracking them on the "server" side when a "client" first connects so the "server" (which will usually act more like a client) can send off its requests. We decided on Friday to just start to thinking of everything as a "peer" and just think in terms of "message events" instead of thinking in terms of "client/server" and "request/response". We haven't run into scenarios where peers drop connections, but we're also barely at the PoC stage. Arnaud Kapp <kapp.a...@gmail.com> wrote: >> >>> I think thread-safe socket would be cool. As in great to have in some particular situation, but most of the case it shouldn't be needed. >> This would be a huge win for us. We have a ridiculously complicated pair of event loops in our central message hub(s) to make the multi-threaded/asynchronous pieces work correctly. I wrote it, and I'm absolutely positive that I'll regret it in a couple of months. That's on the clojure side, where multi-threading is about as simple as it can get. We can get away with single-threading in the C++ parts. The .NET parts are going to be really ugly. The python pieces...well, I keep being forced to say "I told you so" about them. > > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev [1] Links: ------ [1] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
