Re: [zeromq-dev] Welcome to the "zeromq-dev" mailing list

[Tim Crowder]

Hi Jeremy-

You might take a look at existing systems for how they handle it.
Apache Kafka, for instance, always maintains a backlog of messages,
via fast-append to logfiles. Clients keep track of the "offset" of the
last message they processed, and actively pull new messages from
the publisher.

Even if you know when the connection dropped, it's hard to know
which message was completely processed (vs delivered) last.
So, without client pulls, you need explicit client acknowledgement of
the last processed message.
This means that you have to keep saving/buffering messages until all
clients acknowledge them, then you can discard messages up to that point.

Cheers!
.timrc


From: zeromq-dev-boun...@lists.zeromq.org [zeromq-dev-boun...@lists.zeromq.org] 
on behalf of Jeremy Richemont [jrichem...@gmail.com]
Sent: Friday, May 30, 2014 7:21 AM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Welcome to the "zeromq-dev" mailing list

Thanks, Charles. I did, in fact, find that pattern. The problem is it does not 
match what I am trying to do. That pattern for when you have state + deltas. 
What I have is a continuous message stream which, once started to client x must 
be preserved even if client x dies for a bit (not forever of course, I put an 
SLA of 1 million messages/client) and then reconnects, every message it missed 
is replayed, in order, then the live stream resumes.

It needs to handle n clients, any of which may drop and reconnect so each one 
will need an independent message cache. PUB/SUB will not do for this because I 
may need to send messages 10 - 100 to client x on reconnect but 50 - 200 to 
client y.

Asking for state is a good idea - ask for missing updates in my case - but the 
question remains; how does the server know the client is no longer available 
and it must therefore start backing up messages from a PUB socket? The client 
can't tell it over OOB because it died already.

If I could just query PUB and get a list of clients plus a notification when 
one drops that'd solve the problem I think. But how to do that?

Jeremy

On 30 May 2014 15:00, Charles Remes 
mailto:li...@chuckremes.com>> wrote:
Take a look at the Clone pattern in the zguide.

http://zguide.zeromq.org/page:all#Reliable-Pub-Sub-Clone-Pattern

This might be what you need.

cr

On May 29, 2014, at 11:20 AM, Jeremy Richemont 
mailto:jrichem...@gmail.com>> wrote:

>
> Hi. I am struggling to work out how to use zmq to implement the architecture 
> I need. I have a classic publish/subscribe situation except that once client 
> x has subscribed to a topic I need the topic data to be sent to it to be 
> cached if the client dies and resent on reconnect. The data order is 
> important and I can't miss messages should the client be offline for a while.
>
> The PUB/SUB pattern doesn't seem to know about individual clients and will 
> just stop sending to client x if it dies. Plus I can't find out this has 
> happened and cache the messages, or know when it reconnects.
>
> To try to get around this I used the REQ/REP pattern so the clients can 
> announce themselves and have some persistence but this is not ideal for a 
> couple of reasons:
>
> 1) The clients must constantly ask "got any data for me?" which offends my 
> sensibilities
>
> 2) What happens if there's no data to send to client x but there is to client 
> y? Without zmq I'd have had a thread per client and simply block the one with 
> no data but I can't block client x without also blocking client y in a single 
> thread.
>
> Am I trying to shove a round peg in a square hole, here? Is there some way I 
> can get feedback from PUB saying 'failed to send to client x'? so I can cache 
> the messages instead? Or is there some other pattern I should be using?
>
> Otherwise it's back to low level tcp for me...
>
> Many thanks;
>
> Jeremy
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Welcome to the "zeromq-dev" mailing list

[Tim Crowder]

Hi Jeremy-

You've set an SLA: 1M messages max.
If any client falls behind, you have to start saving messages to spinning rust.
So, you need to provide disk storage with sufficient bandwidth up-front,
just-in-case.

If you go the acknowledge route, then you have to make the server smart.
You still have to supply the persistent storage part, along with some logic
to track the clients. It has to know how many clients there are, and where
they are at, and make a number of arbitrary decisions.

If a client has a drive failure, or shuts down uncleanly, you'll still have to 
have
a way to recover. This may involve backing up in the queue for that client,
further than the last acknowledge. So, now the server has to store more than
you expected, *and* you need a way for clients to request from a specific point.

Given that, you can make your code much simpler from the outset, and just
store up to your SLA limit on the server (which tests that you actually have
capacity), and do explicit pulls from your clients.
The server is relatively simple/dumb:
  On message, store to disk, throw away blocks >1M messages past.
  On request, seek to the appropriate spot, read, and shove it in a socket.
The client is relatively simple:
  Request messages at last offset.
  On reply, process messages, advance offset.

You can optimize performance by keeping the last block in memory,
since most clients shouldn't be too far behind, most of the time.
Why add complexity?

Also, you don't have to have 10 connections for 10 streams.
Your request format could be {stream_name,offset, count} to fetch
a set of messages from the appropriate stream. Of course, you'll
need to identify which stream the replies are from, and the next offset.

Cheers!
.timrc


From: zeromq-dev-boun...@lists.zeromq.org [zeromq-dev-boun...@lists.zeromq.org] 
on behalf of Jeremy Richemont [jrichem...@gmail.com]
Sent: Monday, June 02, 2014 5:59 AM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Welcome to the "zeromq-dev" mailing list

Hi Tim. I agree, a message can only be deemed to have been 'sent' if an 
acknowledgement for it is received. As soon as no ack is received the server 
must start backing up messages - but only for that client. It's perfectly 
possible to have one client up-to-date, one that's catching up from 200 
messages ago and another catching up from 1000 messages ago. The server will 
need to know about each client - either because it's been configured to expect 
a certain set of clients or through some discovery mechanism whereby the client 
connects and tells the server about itself.

I kind of like the latter version as it's more flexible but it is more complex 
to implement.

The situation is also complicated by the fact that the server handles ten 
independent message streams. In a way a client may subscribe to all ten but in 
practice I think I'd keep it one to one. A system requiring all ten streams 
would create ten clients to subscribe to each.

Cheers;

Jeremy


On 30 May 2014 20:03, Tim Crowder 
mailto:crowd...@yahoo-inc.com>> wrote:
Hi Jeremy-

You might take a look at existing systems for how they handle it.
Apache Kafka, for instance, always maintains a backlog of messages,
via fast-append to logfiles. Clients keep track of the "offset" of the
last message they processed, and actively pull new messages from
the publisher.

Even if you know when the connection dropped, it's hard to know
which message was completely processed (vs delivered) last.
So, without client pulls, you need explicit client acknowledgement of
the last processed message.
This means that you have to keep saving/buffering messages until all
clients acknowledge them, then you can discard messages up to that point.

Cheers!
.timrc


From: 
zeromq-dev-boun...@lists.zeromq.org<mailto:zeromq-dev-boun...@lists.zeromq.org> 
[zeromq-dev-boun...@lists.zeromq.org<mailto:zeromq-dev-boun...@lists.zeromq.org>]
 on behalf of Jeremy Richemont 
[jrichem...@gmail.com<mailto:jrichem...@gmail.com>]
Sent: Friday, May 30, 2014 7:21 AM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Welcome to the "zeromq-dev" mailing list

Thanks, Charles. I did, in fact, find that pattern. The problem is it does not 
match what I am trying to do. That pattern for when you have state + deltas. 
What I have is a continuous message stream which, once started to client x must 
be preserved even if client x dies for a bit (not forever of course, I put an 
SLA of 1 million messages/client) and then reconnects, every message it missed 
is replayed, in order, then the live stream resumes.

It needs to handle n clients, any of which may drop and reconnect so each one 
will need an independent message cache. PUB/SUB will not do for this because I 
may need to send messages 10 - 100 to client x on reconnect but 50 

Re: [zeromq-dev] zeromq in ros (robot operating system)

[Tim Crowder]

Here are a few common choices:
  https://github.com/google/flatbuffers/
  http://code.google.com/p/protobuf/
  https://thrift.apache.org/
  http://avro.apache.org/
  http://jsoncpp.sourceforge.net/

Protocol buffers are a good general purpose solution,
with good performance, and tons of language bindings.
Flatbuffers is newer, but smaller/faster.
JSON is bigger/slower, but very easy to visually debug.

Cheers!
-timrc

From: zeromq-dev-boun...@lists.zeromq.org [zeromq-dev-boun...@lists.zeromq.org] 
on behalf of Nayab Rasul [rasulnra...@gmail.com]
Sent: Tuesday, June 24, 2014 1:56 PM
To: zeromq-dev@lists.zeromq.org
Subject: [zeromq-dev] zeromq in ros (robot operating system)

Hi all,

I am trying to use zeromq in ros (robot operating system) on ubuntu to receive 
messages from another system with windows. I would like to know what is the for 
bestway to convert from c++ classes to binary ( zeromq messages) and vice versa.


--
Regards,
Nayab Rasul
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] libzmq v4.0.5

[Tim Crowder]

Complaints are not the only (or best) measure of quality.
How many questions does the multitude of semi-redundant functions generate on 
this list,
especially from new users trying to get started? (More so, since some functions 
take void*
and so can't validate by argument type).

Perhaps it's time for a major version bump with some consolidation... 

Cheers!

.timrc

From: zeromq-dev-boun...@lists.zeromq.org [zeromq-dev-boun...@lists.zeromq.org] 
on behalf of Pieter Hintjens [p...@imatix.com]
Sent: Thursday, September 25, 2014 5:20 AM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] libzmq v4.0.5

On Thu, Sep 25, 2014 at 1:39 PM, Goswin von Brederlow  wrote:

> That is fine for major changes where a completly new function makes
> sense. But not for minor behaviour changes that only conern corner
> cases. It doesn't make sense to end up with 20 zmq_sock() like
> functions that all return a slightly differently behaving socket.

We've already done this, and no-one complains:

- zmq_send
- zmq_sendmsg
- zmq_msg_send
- zmsg_send
- zstr_send
- zframe_send
- zsock_send
- zactor_send

All do kind of the same thing. The only time people complained
(rightly) was when 3.0 changed zmq_send.

In practice you don't get 20 variations. You get 2 or 3, one for each
stable generation. Remember we can change draft APIs at will. We make
a stable release maybe once a year. We save old APIs for a few
generations, then delete them.

> HTTP does do exactly that. When you connect you say:
>
> GET /index.html HTTP/1.1

Yes, except it's used for almost nothing significant, and there was
never a successful 1.2 or 2.0. That kind of proves my point. (I've
written many complete HTTP server stacks.)

> Which is fine if you follow your own contract. Zeromq has broken it 3
> times now so that isn't the best track record.

Indeed. However this isn't my project, it's *our* project and we all
enforce the rules, or we don't. My job has been to define workable
rules. I'll enforce them on aspects of the project I use and thus care
about.

Anyone using those pieces that got broken had the tools to stop that
happening, and didn't say anything. We can investigate why, and fix
that, if someone cares.

> And there are a number
> of things I would like to change eventually. Like ZMQ_STREAM sockets
> shouldn't require the MORE flag for every frame, alternating between
> destination and data frames. Instead they should work like ROUTER
> sockets and take the first frame of a multi-part message as the
> destination address and send the rest onwards.

Indeed. You can already do this in several ways. You don't need to
expose ABI versions for this. Indeed we have a good, existing
mechanism for feature updates to socket types. Check the many ROUTER
socket options.

> That I think is the advantage of the FUSE solution. By defining a
> higher FUSE_USE_VERSION you can not only get new features enabled but
> also deprecated features removed.

OK, I do accept the problem statement of multiple versions of an API.

There are many solutions. Remember our focus is distributed software
and a distributed community. This is important. It means we work on
many pieces at the same time, and they evolve independently over a
long time, and at a distance.

What this means is that we do not have a single API contract stream.
Yes, you can say "this is version X" however you cannot use the version
number as a mechanism for changing semantics.

Here's the proof: you have piece A and now someone wants to change that
to A'. New version number. Now, user can choose between A and A' using
the version number. However now we want to improve A and make A''. This
is not an improvement of A'. How does this work? New version again?

We've effectively forced an internal fork of the code. Now multiply
this by the number of aspects we play with.

This is why I called it "insane".

The sane and simple solution, which we use and which is proven, is to
use different namespaces that can evolve independently, and never break
existing code.

This is how we have lots of socket options, how we have many CZMQ classes,
and so on, and how this all works with many old versions and in a real
distributed community, without version madness or internal confusion.

-Pieter
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] libzmq v4.0.5

[Tim Crowder]

Hi Pieter-
 
The kernel is in a fairly unique situation: you can only have one installed at 
a time.
(Yes, I know there was some effort to make ZMQ a kernel module.)

For libraries, you can have many different major versions installed at a time,
and executables can explicitly link against the one they need.

I maintain a library that's ~15yrs old (and on something north of 150,000 
machines), 
and we had full backwards compatibility for 10 years. But at some point, you 
have to 
cut the cord, or you spend all your time maintaining multiple version of your 
API.
When you start thinking that breaking changes are off the table, then 
sub-optimal
interfaces become entrenched, and real improvements get sidelined for hacky
patches.

The path you're suggesting: maintaining those versions within the same codebase
(all with a spec of the latest docs), means that you are forced to update them 
all for
every change. Splitting them out into versioned libraries means that you can 
pick
which patches are important (security, stability, etc), and selectively port 
those back
(or forward) to other versions.

You will get complaints no matter what you do. Even if you deprecate functions, 
and
give *years* of advance notice (dealt with this recently). But with separation 
of major
versions, users of the lagging versions can stick with older libraries. So they 
won't 
be an albatross for changes you'd like to make in newer versions (not to 
mention 
adding needless complexity to the API, and needless redundancy to the codebase).

Cheers!

.timrc


From: zeromq-dev-boun...@lists.zeromq.org [zeromq-dev-boun...@lists.zeromq.org] 
on behalf of Pieter Hintjens [p...@imatix.com]
Sent: Thursday, September 25, 2014 12:05 PM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] libzmq v4.0.5

The surface API remains simple enough, as we deprecate older calls and
remove them from examples and so on. The man pages are quite explicit
about marking deprecated API calls.

We can be more brutal about removing deprecated code, though I've not
seen anyone raise an issue with the current situation. Removing
deprecated pieces will certainly cause complaints.

This is really basic. A version bump does NOT excuse breakage. I don't
know how to keep explaining this in different ways. Let me quote Linus
Torvalds: "DON'T BREAK EFFING USER SPACE" or something like that.

Even a ZeroMQ v5.0 would have to support 3.2 and 2.0 API calls (which
can be taken out of the man pages) unless there is a real, significant
benefit in removing them.

-Pieter



On Thu, Sep 25, 2014 at 8:26 PM, Tim Crowder  wrote:
> Complaints are not the only (or best) measure of quality.
> How many questions does the multitude of semi-redundant functions generate on 
> this list,
> especially from new users trying to get started? (More so, since some 
> functions take void*
> and so can't validate by argument type).
>
> Perhaps it's time for a major version bump with some consolidation...
>
> Cheers!
>
> .timrc
> 
> From: zeromq-dev-boun...@lists.zeromq.org 
> [zeromq-dev-boun...@lists.zeromq.org] on behalf of Pieter Hintjens 
> [p...@imatix.com]
> Sent: Thursday, September 25, 2014 5:20 AM
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] libzmq v4.0.5
>
> On Thu, Sep 25, 2014 at 1:39 PM, Goswin von Brederlow  
> wrote:
>
>> That is fine for major changes where a completly new function makes
>> sense. But not for minor behaviour changes that only conern corner
>> cases. It doesn't make sense to end up with 20 zmq_sock() like
>> functions that all return a slightly differently behaving socket.
>
> We've already done this, and no-one complains:
>
> - zmq_send
> - zmq_sendmsg
> - zmq_msg_send
> - zmsg_send
> - zstr_send
> - zframe_send
> - zsock_send
> - zactor_send
>
> All do kind of the same thing. The only time people complained
> (rightly) was when 3.0 changed zmq_send.
>
> In practice you don't get 20 variations. You get 2 or 3, one for each
> stable generation. Remember we can change draft APIs at will. We make
> a stable release maybe once a year. We save old APIs for a few
> generations, then delete them.
>
>> HTTP does do exactly that. When you connect you say:
>>
>> GET /index.html HTTP/1.1
>
> Yes, except it's used for almost nothing significant, and there was
> never a successful 1.2 or 2.0. That kind of proves my point. (I've
> written many complete HTTP server stacks.)
>
>> Which is fine if you follow your own contract. Zeromq has broken it 3
>> times now so that isn't the best track record.
>
> Indeed. However this isn't my project, it's *our* project and we all
> enforce the ru

[zeromq-dev] Router Identify Events for Socket-Monitor

[Tim Crowder]

Hi All-

I was looking for a way to manage actions/resources around clients connecting 
and
disconnecting from a router socket. Was happy to see the socket-monitor 
interface,
but it only gets partway there...
Since the socket-monitor messages deal with file-descriptors as an ID, there 
still
needs to be a way to tie an fd to the identities that router devices use.

I ran across this post:
  http://article.gmane.org/gmane.network.zeromq.devel/20496/
but it seems that item #2 was never implemented.

So I took a crack at it, and have functioning code. But because the router 
device
doesn't have access to the fd, I had to make it accessible from i_engine and 
pipe.
The patch is very short (~80 lines of actual changes), but I'm still wondering 
if
there's a cleaner way to do it?

Also, any reason that this feature wasn't added when socket-montitor was added?
Anything special I should do to make sure that a patch gets accepted?

Thanks!

-timrc

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev