Re: [zeromq-dev] forwarding a message with metadata

[Thomas Rodgers]

The recently landed pull request #1161
 would now seem to preclude
forwarding on metadata to a PAIR socket.


On Mon, Aug 11, 2014 at 11:23 AM, Thomas Rodgers 
wrote:

> 1) I want to use CURVE and zmq_msg_gets()
>>
>> When I recv() a message on the ROUTER socket it has metadata attached
>> to it. Can I (and how) forward this message across the PAIR socket savely
>> keeping the metadata pointer intact so the app can still call
>> zmq_msg_gets() on it?
>
>
> caveat; I haven't tried it, but I did not see anything obvious in the
> implementation that would preclude metadata attached to a message from
> being forwarded over a PAIR socket by simply sending the received message.
>  The underlying pipe just copies the msg struct by-value to a queue then
> calls init(), resetting the supplied message to an empty state, without
> messing with the refcount on the attached metadata.  The receiving side
> will take ownership of the associated metadata reference when it dequeues
> the message.
>
>
>
> On Mon, Aug 11, 2014 at 10:14 AM, Goswin von Brederlow 
> wrote:
>
>> Hi,
>>
>> I hope you don't mind but I want to brainstorm a bit.
>>
>> I want a generic solution to handle heartbeats that I can reuse in
>> different applications. The most transparent way for this seems to be
>> to create a seperate thread for it. So the design looks something like
>> this:
>>
>> [client] PAIR <--inproc--> [heartbeat thread] PAIR-DEALER <--tcp-->
>>
>> and
>>
>> <--tcp--> ROUTER-PAIR [heartbeat thread] <--inproc--> PAIR [server]
>>
>>
>> The heartbeat thread just forwards messages between the app and the
>> outside world and back monitoring the traffic. And when there is no
>> traffic it inserts heartbeats at regular intervals. It also filters
>> out incoming heartbeats from the outside world.
>>
>> So far this is pretty simple. But now for the interesting parts:
>>
>> 1) I want to use CURVE and zmq_msg_gets()
>>
>> When I recv() a message on the ROUTER socket it has metadata attached
>> to it. Can I (and how) forward this message across the PAIR socket savely
>> keeping the metadata pointer intact so the app can still call
>> zmq_msg_gets() on it?
>>
>> 2) I want to report disapeared peers to the app
>>
>> Basically I have normal messages and control/monitoring messages. I've
>> been thinking that on a ROUTER socket each message starts with an
>> identity frame. So I could send a 0 frame to indicate the message is a
>> control message.
>>
>> But what do I do with the client? I can't use the same with a DEALER
>> socket. I would have to prefix every message with 0 (control) or 1
>> (data) and that would need change the application. I couldn't just
>> plug in or remove the heartbeat without rewriting the message parsing.
>>
>> A short time ago we talked about using some bit in the message itself
>> to say wether it is a control message or normal and having in-band
>> control messages for connect/dicsonnect and so on. Maybe it would be a
>> good time to design and implement something like that now and use it
>> here?
>>
>> 2b) Combining 2 and 1. Can I set my own metadata for control messages?
>>
>> I think the lack of a User-Id on a message could be used to identify
>> control messages. Right?
>>
>> 3) I want to connect/disconnect/bind/unbind or set HWM and so on
>>
>> The app and the DEALER/ROUTER sockets are in different threads. This
>> makes modifying them directly impossible (never share a socket between
>> threads, right?).
>>
>> So this would also need some form of control message but going the
>> other way this time.
>>
>> Should this be just a custom control message that is specific to my
>> use case or would it make sense to define a set of control messages
>> for all the socket operations in zmq so other use cases use the same
>> syntax? I'm thinking this could be reusable for proxies.
>>
>> MfG
>> Goswin
>> ___
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] forwarding a message with metadata

[Thomas Rodgers]

>
> 1) I want to use CURVE and zmq_msg_gets()
>
> When I recv() a message on the ROUTER socket it has metadata attached
> to it. Can I (and how) forward this message across the PAIR socket savely
> keeping the metadata pointer intact so the app can still call
> zmq_msg_gets() on it?


caveat; I haven't tried it, but I did not see anything obvious in the
implementation that would preclude metadata attached to a message from
being forwarded over a PAIR socket by simply sending the received message.
 The underlying pipe just copies the msg struct by-value to a queue then
calls init(), resetting the supplied message to an empty state, without
messing with the refcount on the attached metadata.  The receiving side
will take ownership of the associated metadata reference when it dequeues
the message.



On Mon, Aug 11, 2014 at 10:14 AM, Goswin von Brederlow 
wrote:

> Hi,
>
> I hope you don't mind but I want to brainstorm a bit.
>
> I want a generic solution to handle heartbeats that I can reuse in
> different applications. The most transparent way for this seems to be
> to create a seperate thread for it. So the design looks something like
> this:
>
> [client] PAIR <--inproc--> [heartbeat thread] PAIR-DEALER <--tcp-->
>
> and
>
> <--tcp--> ROUTER-PAIR [heartbeat thread] <--inproc--> PAIR [server]
>
>
> The heartbeat thread just forwards messages between the app and the
> outside world and back monitoring the traffic. And when there is no
> traffic it inserts heartbeats at regular intervals. It also filters
> out incoming heartbeats from the outside world.
>
> So far this is pretty simple. But now for the interesting parts:
>
> 1) I want to use CURVE and zmq_msg_gets()
>
> When I recv() a message on the ROUTER socket it has metadata attached
> to it. Can I (and how) forward this message across the PAIR socket savely
> keeping the metadata pointer intact so the app can still call
> zmq_msg_gets() on it?
>
> 2) I want to report disapeared peers to the app
>
> Basically I have normal messages and control/monitoring messages. I've
> been thinking that on a ROUTER socket each message starts with an
> identity frame. So I could send a 0 frame to indicate the message is a
> control message.
>
> But what do I do with the client? I can't use the same with a DEALER
> socket. I would have to prefix every message with 0 (control) or 1
> (data) and that would need change the application. I couldn't just
> plug in or remove the heartbeat without rewriting the message parsing.
>
> A short time ago we talked about using some bit in the message itself
> to say wether it is a control message or normal and having in-band
> control messages for connect/dicsonnect and so on. Maybe it would be a
> good time to design and implement something like that now and use it
> here?
>
> 2b) Combining 2 and 1. Can I set my own metadata for control messages?
>
> I think the lack of a User-Id on a message could be used to identify
> control messages. Right?
>
> 3) I want to connect/disconnect/bind/unbind or set HWM and so on
>
> The app and the DEALER/ROUTER sockets are in different threads. This
> makes modifying them directly impossible (never share a socket between
> threads, right?).
>
> So this would also need some form of control message but going the
> other way this time.
>
> Should this be just a custom control message that is specific to my
> use case or would it make sense to define a set of control messages
> for all the socket operations in zmq so other use cases use the same
> syntax? I'm thinking this could be reusable for proxies.
>
> MfG
> Goswin
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

[zeromq-dev] forwarding a message with metadata

[Goswin von Brederlow]

Hi,

I hope you don't mind but I want to brainstorm a bit.

I want a generic solution to handle heartbeats that I can reuse in
different applications. The most transparent way for this seems to be
to create a seperate thread for it. So the design looks something like
this:

[client] PAIR <--inproc--> [heartbeat thread] PAIR-DEALER <--tcp-->

and

<--tcp--> ROUTER-PAIR [heartbeat thread] <--inproc--> PAIR [server]


The heartbeat thread just forwards messages between the app and the
outside world and back monitoring the traffic. And when there is no
traffic it inserts heartbeats at regular intervals. It also filters
out incoming heartbeats from the outside world.

So far this is pretty simple. But now for the interesting parts:

1) I want to use CURVE and zmq_msg_gets()

When I recv() a message on the ROUTER socket it has metadata attached
to it. Can I (and how) forward this message across the PAIR socket savely
keeping the metadata pointer intact so the app can still call
zmq_msg_gets() on it?

2) I want to report disapeared peers to the app

Basically I have normal messages and control/monitoring messages. I've
been thinking that on a ROUTER socket each message starts with an
identity frame. So I could send a 0 frame to indicate the message is a
control message.

But what do I do with the client? I can't use the same with a DEALER
socket. I would have to prefix every message with 0 (control) or 1
(data) and that would need change the application. I couldn't just
plug in or remove the heartbeat without rewriting the message parsing.

A short time ago we talked about using some bit in the message itself
to say wether it is a control message or normal and having in-band
control messages for connect/dicsonnect and so on. Maybe it would be a
good time to design and implement something like that now and use it
here?

2b) Combining 2 and 1. Can I set my own metadata for control messages?

I think the lack of a User-Id on a message could be used to identify
control messages. Right?

3) I want to connect/disconnect/bind/unbind or set HWM and so on

The app and the DEALER/ROUTER sockets are in different threads. This
makes modifying them directly impossible (never share a socket between
threads, right?).

So this would also need some form of control message but going the
other way this time.

Should this be just a custom control message that is specific to my
use case or would it make sense to define a set of control messages
for all the socket operations in zmq so other use cases use the same
syntax? I'm thinking this could be reusable for proxies.

MfG
Goswin
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev