Re: [zeromq-dev] nomenclature

[Andrew Hume]

o. i think this is a misunderstanding.  it seems we’re using some terms 
very differently.

when i say “how the data will flow thru the system”, i mean “datum X needs to 
go to entity E via intermediate point Z”.
Z might be a queue and i don’t care about any subdetails (at this level) such 
as TCP/IP details or somesuch.
Z might be a router (or a malamute) and i don’t care (at this level) about any 
subdetails of Z.
so that is your “what data should be where”.

because i am talking about a communication web, i think that specifying what 
the endpoint does with
the data is out of scope. but in your context, it is plausible to say what 
responses (if any) there might be,
and how they are handled.

to my mind, it is a stretch to call that a protocol, but it is not wrong.

your analysis of teh highway system fits with the above; my claim that you need 
to know your destination
is clearly true. i never cared about “all possible configurations” (i never 
said it, but i see that you took
“flow” that way).

alex, thanks for the long reply; it helped me figure how i misspoke.
(assuming my analysis is correct!)

andrew

> On Aug 26, 2016, at 12:08 AM, alex.  wrote:
> 
> Hi Andrew,
> 
> I have to agree with Pieter here. What you want to be thinking about is: What 
> data should be where? And what shall happen with it when it is there?
> 
> I feel this is what you also want to do, but I think Pieter is right in 
> pointing out that thinking about all possible data flow is a mistake, and 
> protocols are a better vantage point. The problem, imho, is briefly: "data 
> flow" encompasses all possible events in your system, protocols define only 
> what events are legal in your system.
> 
> To use your analogy: The national highway system in Germany isn't greatly 
> defined (with it's countless driveways, reststops, roadworks, etc.), and I 
> don't suppose any one person in Germany knows *all* about it, let alone know 
> how all vehicles behave at any one point in time on the highway system. 
> However, the law on traffic in Germany (STVO) is still just 53 paragraphs 
> long and only one of them (§18) is concerned specifically about highways. 
> Still, our highway system works, everday, with all the countless vehicles 
> driving there every day. I'd say the STVO scaled extremely well!
> 
> And here is the interesting part: In almost all cases accidents aren't faults 
> of the driving rules, but of the drivers not upholding the rules. As a 
> computer scientist, you could say that bad drivers are the bugs of the road 
> system. And for the few cases where your protocol was faulty, you only have 
> to change one or two sentences in your law/protocol (not all the drivers).
> 
> So I would also recommend to you to not think about all the moving parts in 
> all their possible configurations (i.e. all the cars on the road), but to 
> really boil it down to: What data should be where? And how should receivers 
> of that data respond to it? That is, write a protocol.
> 
> Best,
> alex.
> 
> P.S. This is also why I came to love finite state machines: as they translate 
> your protocol-law perfectly into code.
> 
> Am 25.08.2016 um 21:36 schrieb Andrew Hume:
>> pieter,
>> 
>>  i can’t figure out what you’re saying here. based on my several years 
>> experience
>> with you, i have to conclude that i have somehow misspoken.
>> 
>>  i have a distributed application. that is, there are a number of 
>> discrete components
>> that (for this discussion) are distinct processes running on some number of 
>> servers.
>> some are accessed thru RPC calls, others generate work that then processed by
>> some number of worker components.
>> 
>>  in order for this to work, i clearly need to figure out how data will 
>> flow
>> through this system. as i have understood the design methodology around 
>> zeromq,
>> this means figuring out the fixed points in the data flow and organising the
>> components around those fixed points. some of these allow for “scaling”
>> (as in increasing the size of a worker pool).
>> 
>>  so i don’t understand
>> 1) how can this sort of design be a “mistake”? how can you do anything 
>> without
>> (trying to) understand where the data needs to go?
>> 
>> 2) many protocols (which i guess you mean things defined by RFCs) don’t 
>> scale;
>> they simply detail teh bi-lateral contract between (roughly) user and 
>> supplier of a service
>> (like NTP). i know some address this directly (like the Gossip protocol), 
>> but surely you don’t mean just those?
>> 
>>  andrew
>> 
>> 
>>> On Aug 24, 2016, at 11:06 AM, Pieter Hintjens  wrote:
>>> 
>>> FWIW I have come to believe that trying to design or even understand
>>> the overall flows of data is a mistake, at least when you want to
>>> scale.
>>> 
>>> What does scale is to speak of protocols and implementations. I know
>>> this isn't a happy answer yet it's a proven one (RFCs, Internet).
>>> 
>>> 

Re: [zeromq-dev] nomenclature

[alex.]

Hi Andrew,

I have to agree with Pieter here. What you want to be thinking about is: 
What data should be where? And what shall happen with it when it is there?


I feel this is what you also want to do, but I think Pieter is right in 
pointing out that thinking about all possible data flow is a mistake, 
and protocols are a better vantage point. The problem, imho, is briefly: 
"data flow" encompasses all possible events in your system, protocols 
define only what events are legal in your system.


To use your analogy: The national highway system in Germany isn't 
greatly defined (with it's countless driveways, reststops, roadworks, 
etc.), and I don't suppose any one person in Germany knows *all* about 
it, let alone know how all vehicles behave at any one point in time on 
the highway system. However, the law on traffic in Germany (STVO) is 
still just 53 paragraphs long and only one of them (§18) is concerned 
specifically about highways. Still, our highway system works, everday, 
with all the countless vehicles driving there every day. I'd say the 
STVO scaled extremely well!


And here is the interesting part: In almost all cases accidents aren't 
faults of the driving rules, but of the drivers not upholding the rules. 
As a computer scientist, you could say that bad drivers are the bugs of 
the road system. And for the few cases where your protocol was faulty, 
you only have to change one or two sentences in your law/protocol (not 
all the drivers).


So I would also recommend to you to not think about all the moving parts 
in all their possible configurations (i.e. all the cars on the road), 
but to really boil it down to: What data should be where? And how should 
receivers of that data respond to it? That is, write a protocol.


Best,
alex.

P.S. This is also why I came to love finite state machines: as they 
translate your protocol-law perfectly into code.


Am 25.08.2016 um 21:36 schrieb Andrew Hume:

pieter,

i can’t figure out what you’re saying here. based on my several years 
experience
with you, i have to conclude that i have somehow misspoken.

i have a distributed application. that is, there are a number of 
discrete components
that (for this discussion) are distinct processes running on some number of 
servers.
some are accessed thru RPC calls, others generate work that then processed by
some number of worker components.

in order for this to work, i clearly need to figure out how data will 
flow
through this system. as i have understood the design methodology around zeromq,
this means figuring out the fixed points in the data flow and organising the
components around those fixed points. some of these allow for “scaling”
(as in increasing the size of a worker pool).

so i don’t understand
1) how can this sort of design be a “mistake”? how can you do anything without
(trying to) understand where the data needs to go?

2) many protocols (which i guess you mean things defined by RFCs) don’t scale;
they simply detail teh bi-lateral contract between (roughly) user and supplier 
of a service
(like NTP). i know some address this directly (like the Gossip protocol), but 
surely you don’t mean just those?

andrew



On Aug 24, 2016, at 11:06 AM, Pieter Hintjens  wrote:

FWIW I have come to believe that trying to design or even understand
the overall flows of data is a mistake, at least when you want to
scale.

What does scale is to speak of protocols and implementations. I know
this isn't a happy answer yet it's a proven one (RFCs, Internet).

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] nomenclature

[Michel Pelletier]

I prefer the term Flow Graph

https://en.wikipedia.org/wiki/Flow_graph_(mathematics)

It might not mathematically accurate but my feeling is most distributed
systems like you're describing can be modeled with a flow graph.  In the
example you provided your scaling points would be coefficients and
bottlenecks can be found by flow analysis.  Like the wikipedia page says:

"The nomenclature is far from standardized, and...no standardization can be
expected in the foreseeable future."

Sounds about right. :)

-Michel

On Thu, Aug 25, 2016 at 12:36 PM, Andrew Hume  wrote:

> pieter,
>
> i can’t figure out what you’re saying here. based on my several
> years experience
> with you, i have to conclude that i have somehow misspoken.
>
> i have a distributed application. that is, there are a number of
> discrete components
> that (for this discussion) are distinct processes running on some number
> of servers.
> some are accessed thru RPC calls, others generate work that then processed
> by
> some number of worker components.
>
> in order for this to work, i clearly need to figure out how data
> will flow
> through this system. as i have understood the design methodology around
> zeromq,
> this means figuring out the fixed points in the data flow and organising
> the
> components around those fixed points. some of these allow for “scaling”
> (as in increasing the size of a worker pool).
>
> so i don’t understand
> 1) how can this sort of design be a “mistake”? how can you do anything
> without
> (trying to) understand where the data needs to go?
>
> 2) many protocols (which i guess you mean things defined by RFCs) don’t
> scale;
> they simply detail teh bi-lateral contract between (roughly) user and
> supplier of a service
> (like NTP). i know some address this directly (like the Gossip protocol),
> but surely you don’t mean just those?
>
> andrew
>
>
> > On Aug 24, 2016, at 11:06 AM, Pieter Hintjens  wrote:
> >
> > FWIW I have come to believe that trying to design or even understand
> > the overall flows of data is a mistake, at least when you want to
> > scale.
> >
> > What does scale is to speak of protocols and implementations. I know
> > this isn't a happy answer yet it's a proven one (RFCs, Internet).
> >
> > ___
> > zeromq-dev mailing list
> > zeromq-dev@lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] nomenclature

[Andrew Hume]

pieter,

i can’t figure out what you’re saying here. based on my several years 
experience
with you, i have to conclude that i have somehow misspoken.

i have a distributed application. that is, there are a number of 
discrete components
that (for this discussion) are distinct processes running on some number of 
servers.
some are accessed thru RPC calls, others generate work that then processed by
some number of worker components.

in order for this to work, i clearly need to figure out how data will 
flow
through this system. as i have understood the design methodology around zeromq,
this means figuring out the fixed points in the data flow and organising the
components around those fixed points. some of these allow for “scaling”
(as in increasing the size of a worker pool).

so i don’t understand
1) how can this sort of design be a “mistake”? how can you do anything without
(trying to) understand where the data needs to go?

2) many protocols (which i guess you mean things defined by RFCs) don’t scale;
they simply detail teh bi-lateral contract between (roughly) user and supplier 
of a service
(like NTP). i know some address this directly (like the Gossip protocol), but 
surely you don’t mean just those?

andrew


> On Aug 24, 2016, at 11:06 AM, Pieter Hintjens  wrote:
> 
> FWIW I have come to believe that trying to design or even understand
> the overall flows of data is a mistake, at least when you want to
> scale.
> 
> What does scale is to speak of protocols and implementations. I know
> this isn't a happy answer yet it's a proven one (RFCs, Internet).
> 
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] nomenclature

[Pieter Hintjens]

FWIW I have come to believe that trying to design or even understand
the overall flows of data is a mistake, at least when you want to
scale.

What does scale is to speak of protocols and implementations. I know
this isn't a happy answer yet it's a proven one (RFCs, Internet).


On Wed, Aug 24, 2016 at 6:28 PM, Arnaud Loonstra  wrote:
> We like to relate to musical terms. We sometimes talk about orchestra or
> composition.
>
> Rg,
>
> Arnaud
>
> On 2016-08-24 18:17, Andrew Hume wrote:
>>
>> thanks for the input.
>>
>> protocols are the wrong sort of thing, i think. that would be like
>> saying another
>> way to talk about the national highway system is to talk about the
>> driving rules.
>> (regardless of how important those rules, and protocols, in general,
>> are.)
>>
>> after a quick read of teh FMC, the best term might be “compositional
>> structure”,
>> but thats uglier than communications web (i think).
>>
>>> On Aug 24, 2016, at 9:04 AM, Andriy Drozdyuk 
>>> wrote:
>>>
>>> Protocols?
>>>
>>> The FMC folks use "channels" to describe this kind of stuff:
>>> http://www.fmc-modeling.org [4]
>>>
>>> Sounds like you might benefit from a "system map":
>>> http://www.fmc-modeling.org/fmcbook [5]
>>>
>>> On Wed, 24 Aug 2016 at 11:31 Andrew Hume 
>>> wrote:
>>>
 in discussions within my group, we often refer to the complete
 data flows between the various
 parts of a distributed application as a “communication web”.
 this includes a bunch of different
 stuff, including (but not limited to) normal zeromq links,
 pub-sub connections, broker-mediated
 links, job queues.

 technically, this is more like a directed graph of data flow.

 the term “communication web” is clumsy at best. are there
 better terms we could use? how do people on
 this list describe the data flow parts of their architectures?

 andrew

 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org [1]
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev [2]
>>>
>>> ___
>>> zeromq-dev mailing list
>>> zeromq-dev@lists.zeromq.org [7]
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>>
>> Links:
>> --
>> [1] mailto:zeromq-dev@lists.zeromq.org
>> [2] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> [3] mailto:dro...@gmail.com
>> [4] http://www.fmc-modeling.org/
>> [5] http://www.fmc-modeling.org/fmcbook
>> [6] mailto:and...@humeweb.com
>> [7] mailto:zeromq-dev@lists.zeromq.org
>
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] nomenclature

[Arnaud Loonstra]

We like to relate to musical terms. We sometimes talk about orchestra 
or composition.


Rg,

Arnaud

On 2016-08-24 18:17, Andrew Hume wrote:

thanks for the input.

protocols are the wrong sort of thing, i think. that would be like
saying another
way to talk about the national highway system is to talk about the
driving rules.
(regardless of how important those rules, and protocols, in general,
are.)

after a quick read of teh FMC, the best term might be “compositional
structure”,
but thats uglier than communications web (i think).


On Aug 24, 2016, at 9:04 AM, Andriy Drozdyuk 
wrote:

Protocols?

The FMC folks use "channels" to describe this kind of stuff:
http://www.fmc-modeling.org [4]

Sounds like you might benefit from a "system map":
http://www.fmc-modeling.org/fmcbook [5]

On Wed, 24 Aug 2016 at 11:31 Andrew Hume 
wrote:


in discussions within my group, we often refer to the complete
data flows between the various
parts of a distributed application as a “communication web”.
this includes a bunch of different
stuff, including (but not limited to) normal zeromq links,
pub-sub connections, broker-mediated
links, job queues.

technically, this is more like a directed graph of data flow.

the term “communication web” is clumsy at best. are there
better terms we could use? how do people on
this list describe the data flow parts of their architectures?

andrew

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org [1]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev [2]

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org [7]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev




Links:
--
[1] mailto:zeromq-dev@lists.zeromq.org
[2] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[3] mailto:dro...@gmail.com
[4] http://www.fmc-modeling.org/
[5] http://www.fmc-modeling.org/fmcbook
[6] mailto:and...@humeweb.com
[7] mailto:zeromq-dev@lists.zeromq.org


___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] nomenclature

[Andrew Hume]

thanks for the input.

protocols are the wrong sort of thing, i think. that would be like saying 
another
way to talk about the national highway system is to talk about the driving 
rules.
(regardless of how important those rules, and protocols, in general, are.)

after a quick read of teh FMC, the best term might be “compositional structure”,
but thats uglier than communications web (i think).


> On Aug 24, 2016, at 9:04 AM, Andriy Drozdyuk  wrote:
> 
> Protocols?
> 
>  
> The FMC folks use "channels" to describe this kind of stuff:
> http://www.fmc-modeling.org 
> Sounds like you might benefit from a "system map": 
> http://www.fmc-modeling.org/fmcbook 
> On Wed, 24 Aug 2016 at 11:31 Andrew Hume  > wrote:
> in discussions within my group, we often refer to the complete data flows 
> between the various
> parts of a distributed application as a “communication web”. this includes a 
> bunch of different
> stuff, including (but not limited to) normal zeromq links, pub-sub 
> connections, broker-mediated
> links, job queues.
> 
> technically, this is more like a directed graph of data flow.
> 
> the term “communication web” is clumsy at best. are there better terms we 
> could use? how do people on
> this list describe the data flow parts of their architectures?
> 
> andrew
> 
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org 
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev 
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] nomenclature

[Andriy Drozdyuk]

Protocols?


The FMC folks use "channels" to describe this kind of stuff:
http://www.fmc-modeling.org
Sounds like you might benefit from a "system map":
http://www.fmc-modeling.org/fmcbook

On Wed, 24 Aug 2016 at 11:31 Andrew Hume  wrote:

> in discussions within my group, we often refer to the complete data flows
> between the various
> parts of a distributed application as a “communication web”. this includes
> a bunch of different
> stuff, including (but not limited to) normal zeromq links, pub-sub
> connections, broker-mediated
> links, job queues.
>
> technically, this is more like a directed graph of data flow.
>
> the term “communication web” is clumsy at best. are there better terms we
> could use? how do people on
> this list describe the data flow parts of their architectures?
>
> andrew
>
> ___
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev