Re: [zeromq-dev] Monitoring and management question
Martin,
Why not simply post a warning in the manual that using this information for
business logic is severely disrecommended. You cannot really 'lock down'
functionality, and abuse is other a relative point of view.
It would be more useful imo to open up the data in a usable for and clearly
explain WHY using it for higher level decisions is inaccurate and fragile,
than either not exposing the data, or obfuscating it in some way.
Assume our users are mostly not idiots...
Pieter
On 23 Dec 2010 09:32, "Martin Sustrik" wrote:
> On 12/23/2010 12:47 AM, Paul Colomiets wrote:
>
>> Or even maybe socket type, which could look like:
>>
>> sock = ctx.socket(zmq.MONITOR)
>> sock.bind('inproc://connections')
>> realsock = ctx.socket(zmq.REQ)
>> realsock.connect('inproc://connections')
>> realsock.send("active")
>> print realsock.recv()
>
> Yes. It's done that way. The only difference is that you use zmq.SUB
> instead of zmq.MONITOR and "sys://log" instead of "inproc://connections".
>
>> This way monitoring capabilities can be directly exposed to the
>> monitoring software on the network.
>>
>> This way you will make improper usage of this information more hard.
>
> It's not particularly hard. I mean, as hard as to prevent unappropriate
> usage.
>
> What would do would be reporting is such a format that it can't be
> parsed by a machine, say "There are three hundred and twenty three
> connections and seven thousand five hundred and seven messages queued
> altogether." with some randomisation in the word order and spelling as
> to confuse parsers.
>
> Not really a neat solution :)
>
> Martin
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Monitoring and management question
On 12/23/2010 12:47 AM, Paul Colomiets wrote:
> Or even maybe socket type, which could look like:
>
> sock = ctx.socket(zmq.MONITOR)
> sock.bind('inproc://connections')
> realsock = ctx.socket(zmq.REQ)
> realsock.connect('inproc://connections')
> realsock.send("active")
> print realsock.recv()
Yes. It's done that way. The only difference is that you use zmq.SUB
instead of zmq.MONITOR and "sys://log" instead of "inproc://connections".
> This way monitoring capabilities can be directly exposed to the
> monitoring software on the network.
>
> This way you will make improper usage of this information more hard.
It's not particularly hard. I mean, as hard as to prevent unappropriate
usage.
What would do would be reporting is such a format that it can't be
parsed by a machine, say "There are three hundred and twenty three
connections and seven thousand five hundred and seven messages queued
altogether." with some randomisation in the word order and spelling as
to confuse parsers.
Not really a neat solution :)
Martin
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Monitoring and management question
Hi Juan, > I will insist in the observability matter. If you find inappropriate or > unproductive my insistence, just tell me. No problem. I'm cc'ing the list btw. > > However, what I fear is that exposing this kind of info would > immediately result in people using it to > > drive business logic rather then using it for system monitoring. > That in turn breaks the "not-connected" > > design of 0MQ, severely hurts scalability etc. > > My feeling is that not having monitoring is less harmful than > introducing features that directly defeat > > the goals of the project. > > I don't know what exactly mean "severely hurts scalability". If adding > this features affect people that's not using them, I agree that it is a > bad thing. I think that "if you don't use it, you'll not have to pay for > it" should be a design principle in this kind of software (system > software?). The problem is this: When you add the feature, people are going to use "monitor connections" feature to check presence of individual peers. They are also going to use "queue size" to check whether particular peer is busy or not. The problem is that while this kind of usage doesn't really work (having connection doesn't necessarily mean the peer is present; small queue doesn't necessarily mean the peer is not busy etc.) it works well enough to be used by clueless users just to be bitterly regretted later on when the production deployment starts to misbehave. As for the scalability, the main mechanism for scaling 0MQ deployments is adding devices in the middle. However, when you depend on peer monitoring in your business logic, adding a device into the middle breaks that business logic. For example, having a connection to the intermediary node doesn't necessarily mean there are any peer present further away in the topology. > One use case of using "monitoring" information in business logic is the > treatment of congestion in telephony networks. > Suppose a system connected to a telephony network, the network derives > some special calls, for example calling card call, to the system. The > system process them, asking for the calling card number, verifying the > balance, calculating the max call duration, etc. > The system is composed of one computed connected to the network, acting > as a gateway and distributing the work that arrives to a pool of worker > computers through 0MQ. > The workers must inform if they are in "congestion state". There are > several congestion states: near congestion, congestion, congestion > discarding work (dropping calls). With this information, the gateway > calculate the congestion state of the overall system and inform it the > network. The network, with this information, can redirect part of the > workload to another system, or start a plan B, i.e. giving a polite > message that means more or less "the the network is in congestion, try > latter". This will certainly happen on Christmas Day. Happy Christmas btw! :) Well, you see, your use case is not about monitoring. It actually binds business logic (redirecting the traffic) to the data you get. So, your system is prone to the problems described above. 0MQ solves the problem of load balancing and peers signaling congestion in a different way. A worker simply stops reading messages when it is congested, which causes messages to queue up to HWM. When HWM is reached, the peer is removed from load-balancing, thus no more messages are sent to it. It gets more messages only when it gets out of congested state and starts reading messages again. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Monitoring and management question
21.12.2010, 20:32, "Martin Sustrik" :
> However, what I fear is that exposing this kind of info would
> immediately result in people using it to drive business logic rather
> then using it for system monitoring. That in turn breaks the
> "not-connected" design of 0MQ, severely hurts scalability etc.
Maybe a special socket protocol for this stuff would help? Something like:
monitoring://connections
Which will list all current and/or pending connections.
Or even maybe socket type, which could look like:
sock = ctx.socket(zmq.MONITOR)
sock.bind('inproc://connections')
realsock = ctx.socket(zmq.REQ)
realsock.connect('inproc://connections')
realsock.send("active")
print realsock.recv()
This way monitoring capabilities can be directly exposed to the
monitoring software on the network.
This way you will make improper usage of this information more hard.
Any thoughts?
--
Paul
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Monitoring and management question
Juan, > I understand it is a messaging middleware, but wouldn't it be a good > thing if you can monitor the state of the middleware? : > - state of connection > - visibility of peers > - occupation of internal queues > - any other already known data There's an internal monitoring system built into 0MQ/2.1, so yes, this can be done. However, what I fear is that exposing this kind of info would immediately result in people using it to drive business logic rather then using it for system monitoring. That in turn breaks the "not-connected" design of 0MQ, severely hurts scalability etc. My feeling is that not having monitoring is less harmful than introducing features that directly defeat the goals of the project. Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Monitoring and management question
2010/12/21 Steven McCoy > On 21 December 2010 21:12, Juan Carlos Franzoy wrote: > >> Is there any plan to add some observability to ZMQ library? Something like >> a getsockopt that list the visibility state of the endpoints the socket is >> connected to? >> >> > The general reply is that 0mq is a messaging middleware not a streaming > middleware, therefore there is no link state. You could technically have > an equivalent to Linux netlink sockets providing link layer notifications as > a workaround. The more useful statistic would be to have a decaying > indicator of last response from each peer. > > http://en.wikipedia.org/wiki/Netlink > > I understand it is a messaging middleware, but wouldn't it be a good thing if you can monitor the state of the middleware? : - state of connection - visibility of peers - occupation of internal queues - any other already known data ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Monitoring and management question
On 21 December 2010 21:12, Juan Carlos Franzoy wrote: > Is there any plan to add some observability to ZMQ library? Something like > a getsockopt that list the visibility state of the endpoints the socket is > connected to? > > The general reply is that 0mq is a messaging middleware not a streaming middleware, therefore there is no link state. You could technically have an equivalent to Linux netlink sockets providing link layer notifications as a workaround. The more useful statistic would be to have a decaying indicator of last response from each peer. http://en.wikipedia.org/wiki/Netlink -- Steve-o ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Monitoring and management question
Hello. First of all, thank for reading and congratulations for ZMQ. We are considering using ZMQ to implement a communication library for all our applications, which are almost always distributed. We make telephonic applications. All our clients require us to inform, usually through SNMP, the state of all the links that the application establishes. ZMQ has non connected protocols, which is good. But we also need information about the visibility of the remote peers. The question is: Is there any plan to add some observability to ZMQ library? Something like a getsockopt that list the visibility state of the endpoints the socket is connected to? Thank in advance. Juan Carlos Franzoy Licenciado en Sistemas ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
