[AFMUG] oversubscription again
so, we hit a wall this week with lopsided providers. we hit an approximate 20:1 and choked. with some policy routing we took it to about a 14:1 ratio and got things buffered. this is upstream, i still am comfortable with 12-15:1 on the ap/cpe side. At what ratio do you decide to buy more provider bandwidth? rule of thumb, because we are all different. With our size, we could probably still afford a 1:1 on the upstream, but we would be wasting a shit ton of cash. Powercode sucks ass for reporting, so its a manual process to see what your ratio is, even though they could pop out report very easily, and I assume query guys already are because Powercode makes it too hard. I am concerned with my personal accountability on this however. Im normally pretty anal about monitoring for points of failure, and I completely dropped the ball on this, relegating some customer complaints to the "quit whining you little bitch" bin, assuming it wa a customer end issue when I should have realized we had sold too much and put it out the small pipe. Any advice (aside from maintaining a 1:1) for metrics to monitor that indicate you oversubscribe? -- If you only see yourself as part of the team but you don't see your team as part of yourself you have already failed as part of the team.
Re: [AFMUG] oversubscription again
Even if you just graph simple 5 minutes averages with Cacti you should consider upgrading if you see utilization approach 80%. ~Seth
Re: [AFMUG] oversubscription again
In auditing, I have found ownership to have oversold themselves a considerable percentage of bandwidth, both above our aggregate, as well as substantially above our sustainable, spread out amongst multiple locations. >From a sysadmin, perspective, or whatever moniker that applies to a person who does as they are told like good little nazis, I chose, as a network preservation techniquue, to alter ownerships bandwidth plans, which on their own allocated a 12:1 network oversubscription. pretty confident this will result in a termination. anybody hiring? On Sat, Oct 1, 2016 at 11:43 PM, Seth Mattinen wrote: > > Even if you just graph simple 5 minutes averages with Cacti you should > consider upgrading if you see utilization approach 80%. > > ~Seth > -- If you only see yourself as part of the team but you don't see your team as part of yourself you have already failed as part of the team.
Re: [AFMUG] oversubscription again
If you are fired for taking appropriate action based on verifiable facts then maybe it wasn't meant to be. I heard once that TW Telecom was using 3:1. That's at Tier2, so the customer is expecting 100meg to be 100meg at any given time. IMO, an oversubscription rate is necessary for capacity planning, but demand only trends upwards over time so whatever you plan for now might not match expectations at a later date. -- Original Message -- From: "That One Guy /sarcasm" To: "af@afmug.com" Sent: 10/2/2016 1:41:09 AM Subject: Re: [AFMUG] oversubscription again In auditing, I have found ownership to have oversold themselves a considerable percentage of bandwidth, both above our aggregate, as well as substantially above our sustainable, spread out amongst multiple locations. From a sysadmin, perspective, or whatever moniker that applies to a person who does as they are told like good little nazis, I chose, as a network preservation techniquue, to alter ownerships bandwidth plans, which on their own allocated a 12:1 network oversubscription. pretty confident this will result in a termination. anybody hiring? On Sat, Oct 1, 2016 at 11:43 PM, Seth Mattinen wrote: Even if you just graph simple 5 minutes averages with Cacti you should consider upgrading if you see utilization approach 80%. ~Seth -- If you only see yourself as part of the team but you don't see your team as part of yourself you have already failed as part of the team.
Re: [AFMUG] oversubscription again
I haven't really paid much attention to monitoring those things. Define some threshold pipe utilization and keep it below that. Web hosting companies keep their pipes between 1/3 and 2/3 utilization to be non-customer-impacting. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "That One Guy /sarcasm" To: af@afmug.com Sent: Saturday, October 1, 2016 11:38:04 PM Subject: [AFMUG] oversubscription again so, we hit a wall this week with lopsided providers. we hit an approximate 20:1 and choked. with some policy routing we took it to about a 14:1 ratio and got things buffered. this is upstream, i still am comfortable with 12-15:1 on the ap/cpe side. At what ratio do you decide to buy more provider bandwidth? rule of thumb, because we are all different. With our size, we could probably still afford a 1:1 on the upstream, but we would be wasting a shit ton of cash. Powercode sucks ass for reporting, so its a manual process to see what your ratio is, even though they could pop out report very easily, and I assume query guys already are because Powercode makes it too hard. I am concerned with my personal accountability on this however. Im normally pretty anal about monitoring for points of failure, and I completely dropped the ball on this, relegating some customer complaints to the "quit whining you little bitch" bin, assuming it wa a customer end issue when I should have realized we had sold too much and put it out the small pipe. Any advice (aside from maintaining a 1:1) for metrics to monitor that indicate you oversubscribe? -- If you only see yourself as part of the team but you don't see your team as part of yourself you have already failed as part of the team.
Re: [AFMUG] oversubscription again
For upstream circuits, we dont upgrade based on a ratio of mbps sold. We upgrade based on utilizaton. we never want to hit over 80%. On Oct 2, 2016 12:38 AM, "That One Guy /sarcasm" wrote: > so, we hit a wall this week with lopsided providers. we hit an approximate > 20:1 and choked. with some policy routing we took it to about a 14:1 ratio > and got things buffered. this is upstream, i still am comfortable with > 12-15:1 on the ap/cpe side. > > At what ratio do you decide to buy more provider bandwidth? rule of thumb, > because we are all different. With our size, we could probably still afford > a 1:1 on the upstream, but we would be wasting a shit ton of cash. > > Powercode sucks ass for reporting, so its a manual process to see what > your ratio is, even though they could pop out report very easily, and I > assume query guys already are because Powercode makes it too hard. > > > I am concerned with my personal accountability on this however. Im > normally pretty anal about monitoring for points of failure, and I > completely dropped the ball on this, relegating some customer complaints to > the "quit whining you little bitch" bin, assuming it wa a customer end > issue when I should have realized we had sold too much and put it out the > small pipe. Any advice (aside from maintaining a 1:1) for metrics to > monitor that indicate you oversubscribe? > > > > -- > If you only see yourself as part of the team but you don't see your team > as part of yourself you have already failed as part of the team. >
Re: [AFMUG] oversubscription again
Over sub ratio can vary greatly from one ISP to the next. I would take note of what it is, but I wouldn't base anything on it. On Oct 2, 2016 8:23 AM, "Chris Fabien" wrote: > For upstream circuits, we dont upgrade based on a ratio of mbps sold. We > upgrade based on utilizaton. we never want to hit over 80%. > > On Oct 2, 2016 12:38 AM, "That One Guy /sarcasm" < > thatoneguyst...@gmail.com> wrote: > >> so, we hit a wall this week with lopsided providers. we hit an >> approximate 20:1 and choked. with some policy routing we took it to about a >> 14:1 ratio and got things buffered. this is upstream, i still am >> comfortable with 12-15:1 on the ap/cpe side. >> >> At what ratio do you decide to buy more provider bandwidth? rule of >> thumb, because we are all different. With our size, we could probably still >> afford a 1:1 on the upstream, but we would be wasting a shit ton of cash. >> >> Powercode sucks ass for reporting, so its a manual process to see what >> your ratio is, even though they could pop out report very easily, and I >> assume query guys already are because Powercode makes it too hard. >> >> >> I am concerned with my personal accountability on this however. Im >> normally pretty anal about monitoring for points of failure, and I >> completely dropped the ball on this, relegating some customer complaints to >> the "quit whining you little bitch" bin, assuming it wa a customer end >> issue when I should have realized we had sold too much and put it out the >> small pipe. Any advice (aside from maintaining a 1:1) for metrics to >> monitor that indicate you oversubscribe? >> >> >> >> -- >> If you only see yourself as part of the team but you don't see your team >> as part of yourself you have already failed as part of the team. >> >
Re: [AFMUG] oversubscription again
um, graph the entire upstream and see when it gets congested? :) - Original Message - From: That One Guy /sarcasm To: af@afmug.com Sent: Saturday, October 01, 2016 11:38 PM Subject: [AFMUG] oversubscription again so, we hit a wall this week with lopsided providers. we hit an approximate 20:1 and choked. with some policy routing we took it to about a 14:1 ratio and got things buffered. this is upstream, i still am comfortable with 12-15:1 on the ap/cpe side. At what ratio do you decide to buy more provider bandwidth? rule of thumb, because we are all different. With our size, we could probably still afford a 1:1 on the upstream, but we would be wasting a shit ton of cash. Powercode sucks ass for reporting, so its a manual process to see what your ratio is, even though they could pop out report very easily, and I assume query guys already are because Powercode makes it too hard. I am concerned with my personal accountability on this however. Im normally pretty anal about monitoring for points of failure, and I completely dropped the ball on this, relegating some customer complaints to the "quit whining you little bitch" bin, assuming it wa a customer end issue when I should have realized we had sold too much and put it out the small pipe. Any advice (aside from maintaining a 1:1) for metrics to monitor that indicate you oversubscribe? -- If you only see yourself as part of the team but you don't see your team as part of yourself you have already failed as part of the team.
Re: [AFMUG] oversubscription again
When you run out of headroom Any flat-topping is reason for concern. From: That One Guy /sarcasm Sent: Saturday, October 1, 2016 10:38 PM To: af@afmug.com Subject: [AFMUG] oversubscription again At what ratio do you decide to buy more provider bandwidth?
Re: [AFMUG] oversubscription again
Yep, need some headroom for usage variation, also things like mitigating DDoS attacks. You don’t want to be flatlining every time Apple puts out an iOS or OSX update, or every Microsoft Patch Tuesday. Or on a 4 day weekend in winter when all the kids are on Netflix/Youtube/Facetime. Or when some new game is released. From: Af [mailto:af-boun...@afmug.com] On Behalf Of ch...@wbmfg.com Sent: Sunday, October 2, 2016 11:48 AM To: af@afmug.com Subject: Re: [AFMUG] oversubscription again When you run out of headroom Any flat-topping is reason for concern. From: That One Guy /sarcasm Sent: Saturday, October 1, 2016 10:38 PM To: af@afmug.com <mailto:af@afmug.com> Subject: [AFMUG] oversubscription again At what ratio do you decide to buy more provider bandwidth?
Re: [AFMUG] oversubscription again
We buy burstable 10GE Internet transit and commit on the 95th percentile. We have 2 Geographically separated Network Cores with a 10GE private pipe in between for edge failover scenarios. We have had fibre cuts which with this arrangement we have the capacity we need… no worries…Just a bigger bill in which the provider that failed will pick up the excess in the event of a longer than 4 hour outage as per our agreement. Cheers, Andreas Wiatowski, CEO Silo Wireless Inc. 519-449-5656 x-600 From: That One Guy /sarcasm Reply-To: "af@afmug.com<mailto:af@afmug.com>" Date: Sunday, October 2, 2016 at 12:38 AM To: "af@afmug.com<mailto:af@afmug.com>" Subject: [AFMUG] oversubscription again so, we hit a wall this week with lopsided providers. we hit an approximate 20:1 and choked. with some policy routing we took it to about a 14:1 ratio and got things buffered. this is upstream, i still am comfortable with 12-15:1 on the ap/cpe side. At what ratio do you decide to buy more provider bandwidth? rule of thumb, because we are all different. With our size, we could probably still afford a 1:1 on the upstream, but we would be wasting a shit ton of cash. Powercode sucks ass for reporting, so its a manual process to see what your ratio is, even though they could pop out report very easily, and I assume query guys already are because Powercode makes it too hard. I am concerned with my personal accountability on this however. Im normally pretty anal about monitoring for points of failure, and I completely dropped the ball on this, relegating some customer complaints to the "quit whining you little bitch" bin, assuming it wa a customer end issue when I should have realized we had sold too much and put it out the small pipe. Any advice (aside from maintaining a 1:1) for metrics to monitor that indicate you oversubscribe? -- If you only see yourself as part of the team but you don't see your team as part of yourself you have already failed as part of the team.
