Re: Ethernet suggestions

[kelly stephenson]

Thanks all, I completely agree with you and you have confirmed my initial
assessment of their system.  Now the fun part, letting my client know they
have wasted 2 years and several million dollars on a custom
hardware/software solution that is terrible.

On Sat, Oct 5, 2019, 5:44 PM Michael Butash  wrote:

> I remember seeing something like a 4 port switch pci card a few decades
> ago, but they're simply not useful anymore when you can buy a 4 port gig
> switch for 5 bucks from china now.
>
> Any other 2-4 port card treats them as standalone nics each, and really
> aren't meant to be bridged together on a workstation or server.  As Mac and
> I stated, it's more for redundancy to one or more switches.
>
> Bridging from physical to virtual interfaces for virtual machines is the
> primary use case for any bridging at all in a server, and should best be
> left at that outside of network appliances.
>
> I don't see why there is any aversion to running a switch to do this vs. a
> ring topology.
>
> -mb
>
>
> On Sat, Oct 5, 2019 at 1:44 PM Stephen Partington 
> wrote:
>
>> I'll have to look. But I think there are ethernet controllers with small
>> switching fabric in them.
>>
>> That might be a level of scaling that would maybe work.
>>
>> On Sat, Oct 5, 2019, 9:15 AM Donald Mac McCarthy 
>> wrote:
>>
>>> Kelly,
>>>
>>> Maybe I am missing something as to why this is a requirement. Is a ring
>>> configuration using RSTP a requirement? If that is the case, I haven't an
>>> answer that I think would help. I know RSTP allows for fast convergence of
>>> failure, I just haven't come across a case where the benefit mattered vs
>>> the complexity of scale. We tried to test RSTP when I was a cluster
>>> administrator at a university, (802.1W must be better than 802.1D right?)
>>> because a professor insisted that the performance of distributed operations
>>> would be better. This was a 4 rack cluster of ~70 nodes. The performance
>>> tanked. After a lot of trial and error, we settled on the architecture that
>>> I am attaching a drawing of using STP.
>>>
>>> If redundancy and ease of operation is what you want - I would use
>>> redundant switches and us Linux to create a bonded interface that is in an
>>> active-passive state. You will have to use a non LACP bond (teaming) as
>>> LACP does not work across switches. Your switch's backplane and uplinks
>>> will be the only bottlenecks that would occur in the network. Most
>>> enterprise switch manufactures build a backplane that can handle the
>>> traffic that is possible to send through the all the ports combined at
>>> theoretical max.
>>>
>>> 2 switches that have 2 or 4 port LACP bonds or if you use switches that
>>> have proprietary stacking cables, use the stacking cable. Also have an LACP
>>> to upstream switching as well.
>>>
>>> Hopefully the drawing attahed will help.
>>>
>>> I have run clusters of over 2500 nodes with a nearly identical
>>> configuration. We used 4x 10Gb per node, 2 LACP bonds per node into 48 port
>>> switches. Those switches had a 6x 40Gb uplinks that were split in LACP to 2
>>> top of rack switches. Top of rack switches had 100Gb uplinks to core. At
>>> the core were multiple internal networks as well as multiple wan
>>> connections.
>>>
>>> My point in talking about the size and speed is not to brag (well, kinda
>>> - don't we all like cool toys), but to point out that this architecture
>>> will work with 1Gb switches and machines of 6 nodes all the way to
>>> thousands of nodes with bigger uplinks. You can scale the switching as your
>>> hardware changes and scales. The architecture remains the same.
>>>
>>> If you are only using 100 nodes, you have less complication. As for plug
>>> and play like behavior, as long as you don't mac lock the switchports - the
>>> switches wont care what you plug into them as long as the NICs are properly
>>> configured.
>>>
>>> Hope this helps. If I have missed something - I hope someone else finds
>>> this useful.
>>>
>>> Mac
>>>
>>> kelly stephenson wrote on 10/4/19 3:34 PM:
>>>
>>> Looking for some networking advice from the group.
>>>
>>> The system I have has several devices connected in a ring configuration
>>> using one Ethernet port IN and one Ethernet port out.  The system uses RSTP
>>> for loop free operation.  The idea is simplicity for installation, you just
>>> unplug and plugin a new device in the ring plus you gain redundancy, if one
>>> Ethernet cable breaks you still have another one.  This works but my client
>>> has never had more then a half dozen devices on the network yet.
>>> When I say devices just imagine very large machines.  The number of
>>> devices could be as many as 100 in the ring or network.  Everything I've
>>> researched on RSTP says over 8 devices and its not effective/efficient so
>>> I'm researching other Ethernet failover/failsafe/redundant solutions.
>>> So, the local network configuration needs to scale up to 100 devices,
>>> have redundancy, and low latency for M2M con

Re: Ethernet suggestions

[Michael Butash]

I remember seeing something like a 4 port switch pci card a few decades
ago, but they're simply not useful anymore when you can buy a 4 port gig
switch for 5 bucks from china now.

Any other 2-4 port card treats them as standalone nics each, and really
aren't meant to be bridged together on a workstation or server.  As Mac and
I stated, it's more for redundancy to one or more switches.

Bridging from physical to virtual interfaces for virtual machines is the
primary use case for any bridging at all in a server, and should best be
left at that outside of network appliances.

I don't see why there is any aversion to running a switch to do this vs. a
ring topology.

-mb


On Sat, Oct 5, 2019 at 1:44 PM Stephen Partington 
wrote:

> I'll have to look. But I think there are ethernet controllers with small
> switching fabric in them.
>
> That might be a level of scaling that would maybe work.
>
> On Sat, Oct 5, 2019, 9:15 AM Donald Mac McCarthy 
> wrote:
>
>> Kelly,
>>
>> Maybe I am missing something as to why this is a requirement. Is a ring
>> configuration using RSTP a requirement? If that is the case, I haven't an
>> answer that I think would help. I know RSTP allows for fast convergence of
>> failure, I just haven't come across a case where the benefit mattered vs
>> the complexity of scale. We tried to test RSTP when I was a cluster
>> administrator at a university, (802.1W must be better than 802.1D right?)
>> because a professor insisted that the performance of distributed operations
>> would be better. This was a 4 rack cluster of ~70 nodes. The performance
>> tanked. After a lot of trial and error, we settled on the architecture that
>> I am attaching a drawing of using STP.
>>
>> If redundancy and ease of operation is what you want - I would use
>> redundant switches and us Linux to create a bonded interface that is in an
>> active-passive state. You will have to use a non LACP bond (teaming) as
>> LACP does not work across switches. Your switch's backplane and uplinks
>> will be the only bottlenecks that would occur in the network. Most
>> enterprise switch manufactures build a backplane that can handle the
>> traffic that is possible to send through the all the ports combined at
>> theoretical max.
>>
>> 2 switches that have 2 or 4 port LACP bonds or if you use switches that
>> have proprietary stacking cables, use the stacking cable. Also have an LACP
>> to upstream switching as well.
>>
>> Hopefully the drawing attahed will help.
>>
>> I have run clusters of over 2500 nodes with a nearly identical
>> configuration. We used 4x 10Gb per node, 2 LACP bonds per node into 48 port
>> switches. Those switches had a 6x 40Gb uplinks that were split in LACP to 2
>> top of rack switches. Top of rack switches had 100Gb uplinks to core. At
>> the core were multiple internal networks as well as multiple wan
>> connections.
>>
>> My point in talking about the size and speed is not to brag (well, kinda
>> - don't we all like cool toys), but to point out that this architecture
>> will work with 1Gb switches and machines of 6 nodes all the way to
>> thousands of nodes with bigger uplinks. You can scale the switching as your
>> hardware changes and scales. The architecture remains the same.
>>
>> If you are only using 100 nodes, you have less complication. As for plug
>> and play like behavior, as long as you don't mac lock the switchports - the
>> switches wont care what you plug into them as long as the NICs are properly
>> configured.
>>
>> Hope this helps. If I have missed something - I hope someone else finds
>> this useful.
>>
>> Mac
>>
>> kelly stephenson wrote on 10/4/19 3:34 PM:
>>
>> Looking for some networking advice from the group.
>>
>> The system I have has several devices connected in a ring configuration
>> using one Ethernet port IN and one Ethernet port out.  The system uses RSTP
>> for loop free operation.  The idea is simplicity for installation, you just
>> unplug and plugin a new device in the ring plus you gain redundancy, if one
>> Ethernet cable breaks you still have another one.  This works but my client
>> has never had more then a half dozen devices on the network yet.
>> When I say devices just imagine very large machines.  The number of
>> devices could be as many as 100 in the ring or network.  Everything I've
>> researched on RSTP says over 8 devices and its not effective/efficient so
>> I'm researching other Ethernet failover/failsafe/redundant solutions.
>> So, the local network configuration needs to scale up to 100 devices,
>> have redundancy, and low latency for M2M control.  Any thoughts?
>>
>> Thanks
>> Kelly
>>
>>
>> ---
>> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
>> To subscribe, unsubscribe, or to change your mail 
>> settings:https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>
>>
>> ---
>> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
>> T

Re: Ethernet suggestions

[Stephen Partington]

I'll have to look. But I think there are ethernet controllers with small
switching fabric in them.

That might be a level of scaling that would maybe work.

On Sat, Oct 5, 2019, 9:15 AM Donald Mac McCarthy  wrote:

> Kelly,
>
> Maybe I am missing something as to why this is a requirement. Is a ring
> configuration using RSTP a requirement? If that is the case, I haven't an
> answer that I think would help. I know RSTP allows for fast convergence of
> failure, I just haven't come across a case where the benefit mattered vs
> the complexity of scale. We tried to test RSTP when I was a cluster
> administrator at a university, (802.1W must be better than 802.1D right?)
> because a professor insisted that the performance of distributed operations
> would be better. This was a 4 rack cluster of ~70 nodes. The performance
> tanked. After a lot of trial and error, we settled on the architecture that
> I am attaching a drawing of using STP.
>
> If redundancy and ease of operation is what you want - I would use
> redundant switches and us Linux to create a bonded interface that is in an
> active-passive state. You will have to use a non LACP bond (teaming) as
> LACP does not work across switches. Your switch's backplane and uplinks
> will be the only bottlenecks that would occur in the network. Most
> enterprise switch manufactures build a backplane that can handle the
> traffic that is possible to send through the all the ports combined at
> theoretical max.
>
> 2 switches that have 2 or 4 port LACP bonds or if you use switches that
> have proprietary stacking cables, use the stacking cable. Also have an LACP
> to upstream switching as well.
>
> Hopefully the drawing attahed will help.
>
> I have run clusters of over 2500 nodes with a nearly identical
> configuration. We used 4x 10Gb per node, 2 LACP bonds per node into 48 port
> switches. Those switches had a 6x 40Gb uplinks that were split in LACP to 2
> top of rack switches. Top of rack switches had 100Gb uplinks to core. At
> the core were multiple internal networks as well as multiple wan
> connections.
>
> My point in talking about the size and speed is not to brag (well, kinda -
> don't we all like cool toys), but to point out that this architecture will
> work with 1Gb switches and machines of 6 nodes all the way to thousands of
> nodes with bigger uplinks. You can scale the switching as your hardware
> changes and scales. The architecture remains the same.
>
> If you are only using 100 nodes, you have less complication. As for plug
> and play like behavior, as long as you don't mac lock the switchports - the
> switches wont care what you plug into them as long as the NICs are properly
> configured.
>
> Hope this helps. If I have missed something - I hope someone else finds
> this useful.
>
> Mac
>
> kelly stephenson wrote on 10/4/19 3:34 PM:
>
> Looking for some networking advice from the group.
>
> The system I have has several devices connected in a ring configuration
> using one Ethernet port IN and one Ethernet port out.  The system uses RSTP
> for loop free operation.  The idea is simplicity for installation, you just
> unplug and plugin a new device in the ring plus you gain redundancy, if one
> Ethernet cable breaks you still have another one.  This works but my client
> has never had more then a half dozen devices on the network yet.
> When I say devices just imagine very large machines.  The number of
> devices could be as many as 100 in the ring or network.  Everything I've
> researched on RSTP says over 8 devices and its not effective/efficient so
> I'm researching other Ethernet failover/failsafe/redundant solutions.
> So, the local network configuration needs to scale up to 100 devices, have
> redundancy, and low latency for M2M control.  Any thoughts?
>
> Thanks
> Kelly
>
>
> ---
> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail 
> settings:https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>
>
> ---
> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
---
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss

Re: Ethernet suggestions

[kelly stephenson]

Awesome responses, I'm still digesting the information.  The original ring
idea was not mine but they decided to do this because of simplicity and
redundancy, both are requirements.  The system is an embedded device and
they even made custom Ethernet "cards".  When I initially looked at the
topology of what they have it seems broken to me.
Seems like they could just connect both Ethernet connections into a switch
and let rstp loop resolve that single connection and move from a ring to a
star topology.  Having multiple Ethernet connections is a requirement for
redundancy.

On Sat, Oct 5, 2019, 12:32 AM Michael Butash  wrote:

> Just in a joking "friends don't let friends do x" sort of way, and even
> that was more directed at the original author.
>
> You're right about RSTP, but the daisy chaining hosts thing is just bad
> mojo in general.  Some random host between to others conversion gets wonky,
> you have to figure out which is causing a problem, and why.  Enterprise
> networking doesn't even like to daisy chain switches anymore, let alone
> hosts.
>
> Not to mention it's entirely unsupportable by anyone but the original
> author setting up hosts as bridges, as no one else would do something like
> this in the business world, let alone even at home.
>
> Just go with a switch and keep your life easy.
>
> -mb
>
>
> On Fri, Oct 4, 2019 at 11:57 PM Stephen Partington 
> wrote:
>
>> It appears that some buttons were pushed. My initial reading did suggest
>> rstp was very good to maintain switch to switch redundancy, which is what i
>> initially thought. Re reading your initial email I am still very curious
>> about why you were looking to use rstp as a nic to nic design.
>>
>> On Fri, Oct 4, 2019, 11:48 PM Michael Butash  wrote:
>>
>>> I really don't get any anyone in their right mind would do this other
>>> than an experiment to say they can/did.  Host ethernet chaining is not what
>>> (Rapid) Spanning Tree Protocol was designed for, and with modern (or old)
>>> switching, there is no reason to.  As a network engineer for 20 years, it
>>> offends certain sensibilities as something you should never do.
>>>
>>> There is a reason people have been using ethernet hubs/switches for 30
>>> years now - speed and simplicity.  If you walked into any sort of
>>> enterprise or business with any network knowledge and proposed that,
>>> someone might just fire you.
>>>
>>> Switches are designed to forward quickly and effectively, some as low as
>>> 350 down to 8 nano seconds these days with special nics,  Even a server cpu
>>> bridging at a kernel level will *never* do so as quickly as that,
>>> particularly cumulative latency in a chain.  Servers that do have more than
>>> one nic certainly aren't intended to be daisy-chained, rather they home
>>> each nic to multiple vlan segments, or they aggregate nics as
>>> active/passive or active/active link aggregation to multiple switches
>>> (redundancy).  Hosts as a rule should NEVER talk spanning-tree, only switch
>>> to switch.
>>>
>>> Just... don't ever chain hosts like that, particularly not if said
>>> client is paying you for a network solution.  Get a switch or multiple with
>>> as many ports as you need.  Ebay is always good for slightly older kit, and
>>> just get a spare to keep around just in case.
>>>
>>> If you're *that* interested in networking to build that sort of science
>>> experiment, pick up a CCNA switching book to learn why you're barking up
>>> the wrong tree.
>>>
>>> -mb
>>>
>>>
>>> On Fri, Oct 4, 2019 at 6:34 PM Stephen Partington 
>>> wrote:
>>>
 I am still wrapping my head around why this was the root design.

 I am not sure what gains you have vs having a pair of switches for
 redundancy. time to research RSTP.

 On Fri, Oct 4, 2019 at 3:34 PM kelly stephenson <
 stephenson2...@gmail.com> wrote:

> Looking for some networking advice from the group.
>
> The system I have has several devices connected in a ring
> configuration using one Ethernet port IN and one Ethernet port out.  The
> system uses RSTP for loop free operation.  The idea is simplicity for
> installation, you just unplug and plugin a new device in the ring plus you
> gain redundancy, if one Ethernet cable breaks you still have another one.
> This works but my client has never had more then a half dozen devices on
> the network yet.
> When I say devices just imagine very large machines.  The number of
> devices could be as many as 100 in the ring or network.  Everything I've
> researched on RSTP says over 8 devices and its not effective/efficient so
> I'm researching other Ethernet failover/failsafe/redundant solutions.
> So, the local network configuration needs to scale up to 100 devices,
> have redundancy, and low latency for M2M control.  Any thoughts?
>
> Thanks
> Kelly
> ---
> PLUG-discuss mailing list - PLUG-discuss@lists.

Re: Ethernet suggestions

[Michael Butash]

Just in a joking "friends don't let friends do x" sort of way, and even
that was more directed at the original author.

You're right about RSTP, but the daisy chaining hosts thing is just bad
mojo in general.  Some random host between to others conversion gets wonky,
you have to figure out which is causing a problem, and why.  Enterprise
networking doesn't even like to daisy chain switches anymore, let alone
hosts.

Not to mention it's entirely unsupportable by anyone but the original
author setting up hosts as bridges, as no one else would do something like
this in the business world, let alone even at home.

Just go with a switch and keep your life easy.

-mb


On Fri, Oct 4, 2019 at 11:57 PM Stephen Partington 
wrote:

> It appears that some buttons were pushed. My initial reading did suggest
> rstp was very good to maintain switch to switch redundancy, which is what i
> initially thought. Re reading your initial email I am still very curious
> about why you were looking to use rstp as a nic to nic design.
>
> On Fri, Oct 4, 2019, 11:48 PM Michael Butash  wrote:
>
>> I really don't get any anyone in their right mind would do this other
>> than an experiment to say they can/did.  Host ethernet chaining is not what
>> (Rapid) Spanning Tree Protocol was designed for, and with modern (or old)
>> switching, there is no reason to.  As a network engineer for 20 years, it
>> offends certain sensibilities as something you should never do.
>>
>> There is a reason people have been using ethernet hubs/switches for 30
>> years now - speed and simplicity.  If you walked into any sort of
>> enterprise or business with any network knowledge and proposed that,
>> someone might just fire you.
>>
>> Switches are designed to forward quickly and effectively, some as low as
>> 350 down to 8 nano seconds these days with special nics,  Even a server cpu
>> bridging at a kernel level will *never* do so as quickly as that,
>> particularly cumulative latency in a chain.  Servers that do have more than
>> one nic certainly aren't intended to be daisy-chained, rather they home
>> each nic to multiple vlan segments, or they aggregate nics as
>> active/passive or active/active link aggregation to multiple switches
>> (redundancy).  Hosts as a rule should NEVER talk spanning-tree, only switch
>> to switch.
>>
>> Just... don't ever chain hosts like that, particularly not if said client
>> is paying you for a network solution.  Get a switch or multiple with as
>> many ports as you need.  Ebay is always good for slightly older kit, and
>> just get a spare to keep around just in case.
>>
>> If you're *that* interested in networking to build that sort of science
>> experiment, pick up a CCNA switching book to learn why you're barking up
>> the wrong tree.
>>
>> -mb
>>
>>
>> On Fri, Oct 4, 2019 at 6:34 PM Stephen Partington 
>> wrote:
>>
>>> I am still wrapping my head around why this was the root design.
>>>
>>> I am not sure what gains you have vs having a pair of switches for
>>> redundancy. time to research RSTP.
>>>
>>> On Fri, Oct 4, 2019 at 3:34 PM kelly stephenson <
>>> stephenson2...@gmail.com> wrote:
>>>
 Looking for some networking advice from the group.

 The system I have has several devices connected in a ring configuration
 using one Ethernet port IN and one Ethernet port out.  The system uses RSTP
 for loop free operation.  The idea is simplicity for installation, you just
 unplug and plugin a new device in the ring plus you gain redundancy, if one
 Ethernet cable breaks you still have another one.  This works but my client
 has never had more then a half dozen devices on the network yet.
 When I say devices just imagine very large machines.  The number of
 devices could be as many as 100 in the ring or network.  Everything I've
 researched on RSTP says over 8 devices and its not effective/efficient so
 I'm researching other Ethernet failover/failsafe/redundant solutions.
 So, the local network configuration needs to scale up to 100 devices,
 have redundancy, and low latency for M2M control.  Any thoughts?

 Thanks
 Kelly
 ---
 PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
 To subscribe, unsubscribe, or to change your mail settings:
 https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>>
>>>
>>>
>>> --
>>> A mouse trap, placed on top of your alarm clock, will prevent you from
>>> rolling over and going back to sleep after you hit the snooze button.
>>>
>>> Stephen
>>>
>>> ---
>>> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
>>> To subscribe, unsubscribe, or to change your mail settings:
>>> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>
>> ---
>> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
>> To subscribe, unsubscribe, or to change your m

Re: Ethernet suggestions

[Michael Butash]

Just my two cents on these comments, you should never get collisions on a
modern network if both switch port and host nic are running in full
duplex.  Any collisions are a sign of a mismatch of negotiation or setting
of the nic to end up in half-duplex on one side or the other, or some other
physical crossing of wires occurring, usually indicated by other physical
errors.

This is what CSMA/CD is for in Ethernet.  I've seen this on 10mb ethernet
up to 100gb, and collisions simply don't occur without a config or cabling
issue to go into half-duplex.  Anything above 100BaseT doesn't even allow
for operation in half-duplex still.

That doesn't mean you won't get other drops, but drops are 99% of the time
caused by buffer starvation of some kind, usually traffic sending faster
than the switch (or host cpu) can forward.  These usually show as an
interface or queue drop, but certainly not related to a collision occurring
in full duplex.

-mb


On Fri, Oct 4, 2019 at 7:08 PM David Schwartz 
wrote:

> I’d advise against it without finding someone who’s already done it and is
> willing to share the details about how they made it work, and what their
> throughput stats are like. (Their stats could be horrible, but they don’t
> know and don’t care simply because their needs are being met.)
>
> Loops may be a potential problem, for sure.
>
> But a bigger potential problem is that Ethernet implements a “CSMA/CD with
> Binary Backoff” protocol. If you look into its origins with AlohaNet,
> you’ll see that it was designed to support maximum bandwidth under “random
> noise” conditions. That means the more random the packets are sent to the
> “ether” in relation to others on the line, the more likely they are to get
> delivered quickly.
>
> CSMA/CD is also considered an “unreliable” protocol at the Transport layer
> because in cases where there are packet collisions, there’s no guarantee
> that either of the packets will ever reach its destination in a timely
> fashion, if ever. Packets are not even guaranteed to arrive in the same
> order they were sent.
>
> The more that CSMA/CD traffic looks like it’s clocked, the more likely
> there will be collisions, and the more irregular the deliveries become
> because of that Binary Backoff strategy.
>
> To solve this problem, IBM invented the TokenRing architecture. In a TR
> network, there’s a bounded time for a packet to make it around the entire
> network. If a packet didn’t arrive in that time, you could be assured that
> it was lost. So if a sender didn’t get back an ACK within 2x the time since
> it was sent, it knew it had to send it again. In an Ethernet environment,
> you simply don’t know because there are no guarantees about timeliness at
> all.
>
> When you’ve got equipment sending out data at regular intervals, that
> looks like a “clock” to the network (like a heartbeat or drumbeat). As long
> as they’re all set to different intervals, it’s ok. But if any are
> coincident, there are bound to be more frequent collisions, causing delays
> or packet losses. If there are enough nodes all on the same schedule, your
> potential throughput could drop into the gutter. That’s exactly the
> scenario you’ll have if you daisy-chain a bunch of these things together.
>
> With a fast enough circuit (ie., gigabit Ethernet) and small enough
> packets, your risks will be minimized. But slower circuits and/or larger
> packets will lead to more collisions, which will tend to slow things down.
>
> Your best bet is to hook them up in a ’star’ configuration the way
> Ethernet was designed. Set up 8- or 16-port routers and drop a node on each
> port. The routers can be stacked hierarchically, and everything will work
> just fine. That’s in part because routers are “active" hubs (as opposed to
> regular “hubs” which are passive); ie, they have a CPU in them, and they
> help optimize the traffic flow to a certain extent.
>
> Again, I wouldn’t invest the time into this unless / until you can find
> someone who’s already built a similar-sized network and can share the
> pitfalls they encountered (if any).
>
> -David Schwartz
>
> On Oct 4, 2019, at 3:34 PM, kelly stephenson 
> wrote:
>
> Looking for some networking advice from the group.
>
> The system I have has several devices connected in a ring configuration
> using one Ethernet port IN and one Ethernet port out. The system uses RSTP
> for loop free operation. The idea is simplicity for installation, you just
> unplug and plugin a new device in the ring plus you gain redundancy, if one
> Ethernet cable breaks you still have another one. This works but my client
> has never had more then a half dozen devices on the network yet. When I say
> devices just imagine very large machines. The number of devices could be as
> many as 100 in the ring or network. Everything I've researched on RSTP says
> over 8 devices and its not effective/efficient so I'm researching other
> Ethernet failover/failsafe/redundant solutions. So, the local network
> conf

Re: Ethernet suggestions

[Stephen Partington]

It appears that some buttons were pushed. My initial reading did suggest
rstp was very good to maintain switch to switch redundancy, which is what i
initially thought. Re reading your initial email I am still very curious
about why you were looking to use rstp as a nic to nic design.

On Fri, Oct 4, 2019, 11:48 PM Michael Butash  wrote:

> I really don't get any anyone in their right mind would do this other than
> an experiment to say they can/did.  Host ethernet chaining is not what
> (Rapid) Spanning Tree Protocol was designed for, and with modern (or old)
> switching, there is no reason to.  As a network engineer for 20 years, it
> offends certain sensibilities as something you should never do.
>
> There is a reason people have been using ethernet hubs/switches for 30
> years now - speed and simplicity.  If you walked into any sort of
> enterprise or business with any network knowledge and proposed that,
> someone might just fire you.
>
> Switches are designed to forward quickly and effectively, some as low as
> 350 down to 8 nano seconds these days with special nics,  Even a server cpu
> bridging at a kernel level will *never* do so as quickly as that,
> particularly cumulative latency in a chain.  Servers that do have more than
> one nic certainly aren't intended to be daisy-chained, rather they home
> each nic to multiple vlan segments, or they aggregate nics as
> active/passive or active/active link aggregation to multiple switches
> (redundancy).  Hosts as a rule should NEVER talk spanning-tree, only switch
> to switch.
>
> Just... don't ever chain hosts like that, particularly not if said client
> is paying you for a network solution.  Get a switch or multiple with as
> many ports as you need.  Ebay is always good for slightly older kit, and
> just get a spare to keep around just in case.
>
> If you're *that* interested in networking to build that sort of science
> experiment, pick up a CCNA switching book to learn why you're barking up
> the wrong tree.
>
> -mb
>
>
> On Fri, Oct 4, 2019 at 6:34 PM Stephen Partington 
> wrote:
>
>> I am still wrapping my head around why this was the root design.
>>
>> I am not sure what gains you have vs having a pair of switches for
>> redundancy. time to research RSTP.
>>
>> On Fri, Oct 4, 2019 at 3:34 PM kelly stephenson 
>> wrote:
>>
>>> Looking for some networking advice from the group.
>>>
>>> The system I have has several devices connected in a ring configuration
>>> using one Ethernet port IN and one Ethernet port out.  The system uses RSTP
>>> for loop free operation.  The idea is simplicity for installation, you just
>>> unplug and plugin a new device in the ring plus you gain redundancy, if one
>>> Ethernet cable breaks you still have another one.  This works but my client
>>> has never had more then a half dozen devices on the network yet.
>>> When I say devices just imagine very large machines.  The number of
>>> devices could be as many as 100 in the ring or network.  Everything I've
>>> researched on RSTP says over 8 devices and its not effective/efficient so
>>> I'm researching other Ethernet failover/failsafe/redundant solutions.
>>> So, the local network configuration needs to scale up to 100 devices,
>>> have redundancy, and low latency for M2M control.  Any thoughts?
>>>
>>> Thanks
>>> Kelly
>>> ---
>>> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
>>> To subscribe, unsubscribe, or to change your mail settings:
>>> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>
>>
>>
>> --
>> A mouse trap, placed on top of your alarm clock, will prevent you from
>> rolling over and going back to sleep after you hit the snooze button.
>>
>> Stephen
>>
>> ---
>> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
>> To subscribe, unsubscribe, or to change your mail settings:
>> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>
> ---
> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
---
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss

Re: Ethernet suggestions

[Michael Butash]

I really don't get any anyone in their right mind would do this other than
an experiment to say they can/did.  Host ethernet chaining is not what
(Rapid) Spanning Tree Protocol was designed for, and with modern (or old)
switching, there is no reason to.  As a network engineer for 20 years, it
offends certain sensibilities as something you should never do.

There is a reason people have been using ethernet hubs/switches for 30
years now - speed and simplicity.  If you walked into any sort of
enterprise or business with any network knowledge and proposed that,
someone might just fire you.

Switches are designed to forward quickly and effectively, some as low as
350 down to 8 nano seconds these days with special nics,  Even a server cpu
bridging at a kernel level will *never* do so as quickly as that,
particularly cumulative latency in a chain.  Servers that do have more than
one nic certainly aren't intended to be daisy-chained, rather they home
each nic to multiple vlan segments, or they aggregate nics as
active/passive or active/active link aggregation to multiple switches
(redundancy).  Hosts as a rule should NEVER talk spanning-tree, only switch
to switch.

Just... don't ever chain hosts like that, particularly not if said client
is paying you for a network solution.  Get a switch or multiple with as
many ports as you need.  Ebay is always good for slightly older kit, and
just get a spare to keep around just in case.

If you're *that* interested in networking to build that sort of science
experiment, pick up a CCNA switching book to learn why you're barking up
the wrong tree.

-mb


On Fri, Oct 4, 2019 at 6:34 PM Stephen Partington 
wrote:

> I am still wrapping my head around why this was the root design.
>
> I am not sure what gains you have vs having a pair of switches for
> redundancy. time to research RSTP.
>
> On Fri, Oct 4, 2019 at 3:34 PM kelly stephenson 
> wrote:
>
>> Looking for some networking advice from the group.
>>
>> The system I have has several devices connected in a ring configuration
>> using one Ethernet port IN and one Ethernet port out.  The system uses RSTP
>> for loop free operation.  The idea is simplicity for installation, you just
>> unplug and plugin a new device in the ring plus you gain redundancy, if one
>> Ethernet cable breaks you still have another one.  This works but my client
>> has never had more then a half dozen devices on the network yet.
>> When I say devices just imagine very large machines.  The number of
>> devices could be as many as 100 in the ring or network.  Everything I've
>> researched on RSTP says over 8 devices and its not effective/efficient so
>> I'm researching other Ethernet failover/failsafe/redundant solutions.
>> So, the local network configuration needs to scale up to 100 devices,
>> have redundancy, and low latency for M2M control.  Any thoughts?
>>
>> Thanks
>> Kelly
>> ---
>> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
>> To subscribe, unsubscribe, or to change your mail settings:
>> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
>
>
>
> --
> A mouse trap, placed on top of your alarm clock, will prevent you from
> rolling over and going back to sleep after you hit the snooze button.
>
> Stephen
>
> ---
> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
---
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss

Re: Ethernet suggestions

[Snyder, Alexander J]

That was a great explanation David, thanks!

Wasn't my question, but I certainly learned a lot.

Thanks,
Alexander

Sent from my Galaxy S10+

On Fri, Oct 4, 2019, 19:08 David Schwartz 
wrote:

> I’d advise against it without finding someone who’s already done it and is
> willing to share the details about how they made it work, and what their
> throughput stats are like. (Their stats could be horrible, but they don’t
> know and don’t care simply because their needs are being met.)
>
> Loops may be a potential problem, for sure.
>
> But a bigger potential problem is that Ethernet implements a “CSMA/CD with
> Binary Backoff” protocol. If you look into its origins with AlohaNet,
> you’ll see that it was designed to support maximum bandwidth under “random
> noise” conditions. That means the more random the packets are sent to the
> “ether” in relation to others on the line, the more likely they are to get
> delivered quickly.
>
> CSMA/CD is also considered an “unreliable” protocol at the Transport layer
> because in cases where there are packet collisions, there’s no guarantee
> that either of the packets will ever reach its destination in a timely
> fashion, if ever. Packets are not even guaranteed to arrive in the same
> order they were sent.
>
> The more that CSMA/CD traffic looks like it’s clocked, the more likely
> there will be collisions, and the more irregular the deliveries become
> because of that Binary Backoff strategy.
>
> To solve this problem, IBM invented the TokenRing architecture. In a TR
> network, there’s a bounded time for a packet to make it around the entire
> network. If a packet didn’t arrive in that time, you could be assured that
> it was lost. So if a sender didn’t get back an ACK within 2x the time since
> it was sent, it knew it had to send it again. In an Ethernet environment,
> you simply don’t know because there are no guarantees about timeliness at
> all.
>
> When you’ve got equipment sending out data at regular intervals, that
> looks like a “clock” to the network (like a heartbeat or drumbeat). As long
> as they’re all set to different intervals, it’s ok. But if any are
> coincident, there are bound to be more frequent collisions, causing delays
> or packet losses. If there are enough nodes all on the same schedule, your
> potential throughput could drop into the gutter. That’s exactly the
> scenario you’ll have if you daisy-chain a bunch of these things together.
>
> With a fast enough circuit (ie., gigabit Ethernet) and small enough
> packets, your risks will be minimized. But slower circuits and/or larger
> packets will lead to more collisions, which will tend to slow things down.
>
> Your best bet is to hook them up in a ’star’ configuration the way
> Ethernet was designed. Set up 8- or 16-port routers and drop a node on each
> port. The routers can be stacked hierarchically, and everything will work
> just fine. That’s in part because routers are “active" hubs (as opposed to
> regular “hubs” which are passive); ie, they have a CPU in them, and they
> help optimize the traffic flow to a certain extent.
>
> Again, I wouldn’t invest the time into this unless / until you can find
> someone who’s already built a similar-sized network and can share the
> pitfalls they encountered (if any).
>
> -David Schwartz
>
> On Oct 4, 2019, at 3:34 PM, kelly stephenson 
> wrote:
>
> Looking for some networking advice from the group.
>
> The system I have has several devices connected in a ring configuration
> using one Ethernet port IN and one Ethernet port out. The system uses RSTP
> for loop free operation. The idea is simplicity for installation, you just
> unplug and plugin a new device in the ring plus you gain redundancy, if one
> Ethernet cable breaks you still have another one. This works but my client
> has never had more then a half dozen devices on the network yet. When I say
> devices just imagine very large machines. The number of devices could be as
> many as 100 in the ring or network. Everything I've researched on RSTP says
> over 8 devices and its not effective/efficient so I'm researching other
> Ethernet failover/failsafe/redundant solutions. So, the local network
> configuration needs to scale up to 100 devices, have redundancy, and low
> latency for M2M control. Any thoughts?
>
> Thanks Kelly ---
> PLUG-discuss mailing list – PLUG-discuss@lists.phxlinux.org To subscribe,
> unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
> 
>
> --

Re: Ethernet suggestions

[David Schwartz]

I’d advise against it without finding someone who’s already done it and is 
willing to share the details about how they made it work, and what their 
throughput stats are like. (Their stats could be horrible, but they don’t know 
and don’t care simply because their needs are being met.)

Loops may be a potential problem, for sure.

But a bigger potential problem is that Ethernet implements a "CSMA/CD with 
Binary Backoff” protocol. If you look into its origins with AlohaNet, you’ll 
see that it was designed to support maximum bandwidth under "random noise" 
conditions. That means the more random the packets are sent to the “ether" in 
relation to others on the line, the more likely they are to get delivered 
quickly.

CSMA/CD is also considered an “unreliable” protocol at the Transport layer 
because in cases where there are packet collisions, there’s no guarantee that 
either of the packets will ever reach its destination in a timely fashion, if 
ever. Packets are not even guaranteed to arrive in the same order they were 
sent.

The more that CSMA/CD traffic looks like it’s clocked, the more likely there 
will be collisions, and the more irregular the deliveries become because of 
that Binary Backoff strategy.

To solve this problem, IBM invented the TokenRing architecture. In a TR 
network, there’s a bounded time for a packet to make it around the entire 
network. If a packet didn’t arrive in that time, you could be assured that it 
was lost. So if a sender didn’t get back an ACK within 2x the time since it was 
sent, it knew it had to send it again. In an Ethernet environment, you simply 
don’t know because there are no guarantees about timeliness at all.

When you’ve got equipment sending out data at regular intervals, that looks 
like a “clock” to the network (like a heartbeat or drumbeat). As long as 
they’re all set to different intervals, it’s ok. But if any are coincident, 
there are bound to be more frequent collisions, causing delays or packet 
losses. If there are enough nodes all on the same schedule, your potential 
throughput could drop into the gutter. That’s exactly the scenario you’ll have 
if  you daisy-chain a bunch of these things together.

With a fast enough circuit (ie., gigabit Ethernet) and small enough packets, 
your risks will be minimized. But slower circuits and/or larger packets will 
lead to more collisions, which will tend to slow things down.

Your best bet is to hook them up in a ’star’ configuration the way Ethernet was 
designed. Set up 8- or 16-port routers and drop a node on each port. The 
routers can be stacked hierarchically, and everything will work just fine. 
That’s in part because routers are “active" hubs (as opposed to regular “hubs” 
which are passive); ie, they have a CPU in them, and they help optimize the 
traffic flow to a certain extent.

Again, I wouldn’t invest the time into this unless / until you can find someone 
who’s already built a similar-sized network and can share the pitfalls they 
encountered (if any).

-David Schwartz



> On Oct 4, 2019, at 3:34 PM, kelly stephenson  wrote:
> 
> Looking for some networking advice from the group.
> 
> The system I have has several devices connected in a ring configuration using 
> one Ethernet port IN and one Ethernet port out.  The system uses RSTP for 
> loop free operation.  The idea is simplicity for installation, you just 
> unplug and plugin a new device in the ring plus you gain redundancy, if one 
> Ethernet cable breaks you still have another one.  This works but my client 
> has never had more then a half dozen devices on the network yet.
> When I say devices just imagine very large machines.  The number of devices 
> could be as many as 100 in the ring or network.  Everything I've researched 
> on RSTP says over 8 devices and its not effective/efficient so I'm 
> researching other Ethernet failover/failsafe/redundant solutions.
> So, the local network configuration needs to scale up to 100 devices, have 
> redundancy, and low latency for M2M control.  Any thoughts?  
> 
> Thanks
> Kelly
> ---
> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss

---
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss

Re: Ethernet suggestions

[Stephen Partington]

I am still wrapping my head around why this was the root design.

I am not sure what gains you have vs having a pair of switches for
redundancy. time to research RSTP.

On Fri, Oct 4, 2019 at 3:34 PM kelly stephenson 
wrote:

> Looking for some networking advice from the group.
>
> The system I have has several devices connected in a ring configuration
> using one Ethernet port IN and one Ethernet port out.  The system uses RSTP
> for loop free operation.  The idea is simplicity for installation, you just
> unplug and plugin a new device in the ring plus you gain redundancy, if one
> Ethernet cable breaks you still have another one.  This works but my client
> has never had more then a half dozen devices on the network yet.
> When I say devices just imagine very large machines.  The number of
> devices could be as many as 100 in the ring or network.  Everything I've
> researched on RSTP says over 8 devices and its not effective/efficient so
> I'm researching other Ethernet failover/failsafe/redundant solutions.
> So, the local network configuration needs to scale up to 100 devices, have
> redundancy, and low latency for M2M control.  Any thoughts?
>
> Thanks
> Kelly
> ---
> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss



-- 
A mouse trap, placed on top of your alarm clock, will prevent you from
rolling over and going back to sleep after you hit the snooze button.

Stephen
---
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss

Ethernet suggestions

[kelly stephenson]

Looking for some networking advice from the group.

The system I have has several devices connected in a ring configuration
using one Ethernet port IN and one Ethernet port out.  The system uses RSTP
for loop free operation.  The idea is simplicity for installation, you just
unplug and plugin a new device in the ring plus you gain redundancy, if one
Ethernet cable breaks you still have another one.  This works but my client
has never had more then a half dozen devices on the network yet.
When I say devices just imagine very large machines.  The number of devices
could be as many as 100 in the ring or network.  Everything I've
researched on RSTP says over 8 devices and its not effective/efficient so
I'm researching other Ethernet failover/failsafe/redundant solutions.
So, the local network configuration needs to scale up to 100 devices, have
redundancy, and low latency for M2M control.  Any thoughts?

Thanks
Kelly
---
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss