Re: Evaluating Tier 1 Internet providers
On Wed, Aug 28, 2013 at 09:54:28AM -0700, Michael Smith wrote: It's really can reach versus how well can they reach. I can't any provider that would have less than a full view of the DFZ but, if your primary traffic is to Provider X, and one of your Tier 1's peers locally and the other peers in France, then you would look more closely at the closer one. Unless, of course, that local peer was saturated 99% of the time. Then France might be attractive. One thing to keep in mind is that for major Tier 1s, it's not at all uncommon to see some very large percentages of traffic (like say well north of 50%) stay completely on-net, going from customer to customer. In this type of model, capacity to other third party peers (typically the other Tier 1's) becomes secondary to other considerations like backbone capacity, which is why those huge Tier 1 networks often have much less peering capacity than you might otherwise expect. Tier 2's on the other hand, typically spend the vast majority of their time/money/effort figuring out how they can deliver traffic to other networks via peering and transit relationships. This usually means they have much smaller amounts of backbone capacity, but relative to their total sizes they often have a lot more capacity to the other major peering/transit networks. The economics of each model are vastly different too. Tier 2's are typically always looking to take advantage of tricks like hot potato routing and 95th percentile billing to get free inbound to minimize their backhaul costs. All too often people tend to get caught in the mentral trap of thinking peering == free, but in reality the Tier 1's are just shifting the majority of their operational costs into backbone instead, and peering becomes the way to handle the leftovers. Each model has its advantages and weaknesses, but most people who haven't lived in both worlds tend to vastly underestimate the realities of the other side's cost models. There is a lot to be said for the value of a Tier 2 network. Sometimes throwing a token amount of money at a problem solves it much more effectively than waiting for two squabbling Tier 1's to fight over the principal of not paying anything or risking being perceived as weak. And a Tier 2 with multiple transit paths and extensive peering options may be able to easily reroute traffic around a particular problem spot in a way that a Tier 1 just doesn't have the ability to do. Then again, sometimes there is value in just buying transit from someone who operates a massive entwork, with the economy of scale necessary to implement terabits of backbone capacity for cheap, and a huge customer base. As for the which one should I buy question, a smart person would realize the different strengths and weaknesses of each model, and probably end up buying from (at least) one of each to take advantage of this. Of course in reality 99% of people fail to understand any of this, and turn off their brains after thinking things like 1 2 so it must be better. :) -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Evaluating Tier 1 Internet providers
+10 Good explanation. This is a lot of why I have someone like Cogent/L3/etc and some random transit provider in most of my pops I spec, plus a backhaul to another node. On Thu, Aug 29, 2013 at 9:37 PM, Richard A Steenbergen r...@e-gerbil.netwrote: On Wed, Aug 28, 2013 at 09:54:28AM -0700, Michael Smith wrote: It's really can reach versus how well can they reach. I can't any provider that would have less than a full view of the DFZ but, if your primary traffic is to Provider X, and one of your Tier 1's peers locally and the other peers in France, then you would look more closely at the closer one. Unless, of course, that local peer was saturated 99% of the time. Then France might be attractive. One thing to keep in mind is that for major Tier 1s, it's not at all uncommon to see some very large percentages of traffic (like say well north of 50%) stay completely on-net, going from customer to customer. In this type of model, capacity to other third party peers (typically the other Tier 1's) becomes secondary to other considerations like backbone capacity, which is why those huge Tier 1 networks often have much less peering capacity than you might otherwise expect. Tier 2's on the other hand, typically spend the vast majority of their time/money/effort figuring out how they can deliver traffic to other networks via peering and transit relationships. This usually means they have much smaller amounts of backbone capacity, but relative to their total sizes they often have a lot more capacity to the other major peering/transit networks. The economics of each model are vastly different too. Tier 2's are typically always looking to take advantage of tricks like hot potato routing and 95th percentile billing to get free inbound to minimize their backhaul costs. All too often people tend to get caught in the mentral trap of thinking peering == free, but in reality the Tier 1's are just shifting the majority of their operational costs into backbone instead, and peering becomes the way to handle the leftovers. Each model has its advantages and weaknesses, but most people who haven't lived in both worlds tend to vastly underestimate the realities of the other side's cost models. There is a lot to be said for the value of a Tier 2 network. Sometimes throwing a token amount of money at a problem solves it much more effectively than waiting for two squabbling Tier 1's to fight over the principal of not paying anything or risking being perceived as weak. And a Tier 2 with multiple transit paths and extensive peering options may be able to easily reroute traffic around a particular problem spot in a way that a Tier 1 just doesn't have the ability to do. Then again, sometimes there is value in just buying transit from someone who operates a massive entwork, with the economy of scale necessary to implement terabits of backbone capacity for cheap, and a huge customer base. As for the which one should I buy question, a smart person would realize the different strengths and weaknesses of each model, and probably end up buying from (at least) one of each to take advantage of this. Of course in reality 99% of people fail to understand any of this, and turn off their brains after thinking things like 1 2 so it must be better. :) -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Evaluating Tier 1 Internet providers
On 08/29/2013 07:43 PM, Blake Dunlap wrote: +10 Good explanation. This is a lot of why I have someone like Cogent/L3/etc and some random transit provider in most of my pops I spec, plus a backhaul to another node. ... One thing to keep in mind is that for major Tier 1s, it's not at all uncommon to see some very large percentages of traffic (like say well north of 50%) stay completely on-net, going from customer to customer. In this type of model, capacity to other third party peers (typically the other Tier 1's) becomes secondary to other considerations like backbone capacity, which is why those huge Tier 1 networks often have much less peering capacity than you might otherwise expect. a major problem here is that some providers try too hard to be tier 1... - my pager has gone off many times because $lowcost_tier1 decided to route a packet from them in san jose destined for them in Sacramento through texas. Problem is, often that is still fewer hops, (even if it's many more ms) than going through my tier2 provider, so having the backup did not help me. Nor would taking customer-only routes from $lowcost_tier1... the shortest path, in terms of hops, was through them, through texas. There was nothing to be done short of switching to my tier2. I have no idea how to solve this sort of problem automatically. Ideally, if someone has a congested or down link, I'd prefer that they not announce routes to that part of the internet, as I do have a backup, but that isn't how it works.
Re: Evaluating Tier 1 Internet providers
On Thu, Aug 29, 2013 at 08:25:41PM -0700, Luke S. Crawford wrote: I have no idea how to solve this sort of problem automatically. Ideally, if someone has a congested or down link, I'd prefer that they not announce routes to that part of the internet, as I do have a backup, but that isn't how it works. BGP best path routing decisions are made by completely irrelevent criteria like AS-PATH lengths and lower router-id's, and are completely blind to things that actualy matter like latency, capacity, packet loss, etc. Fundamentally it's impossible to fix automatically with the current routing protocols, and at best the protocol extensions like BGP AIGP (which could help at least convey some of the data, like the oh crap I just got rerouted to a different exit with much higher latency situation you mentioned) are still a long way from being practically usable. At best you can aim your default/tie breaks towards networks you have more faith in, but that doesn't mean much in practice. :) -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Evaluating Tier 1 Internet providers
On Tue, Aug 27, 2013 at 12:14 PM, Joe Abley jab...@hopcount.ca wrote: I would add: - response you can expect when you call one day and say our 10GE is maxed out with inbound traffic from apparently everywhere, it has been going on for an hour, please help That was good for a laugh. If it's a DoS, you know what the answer already is. We no longer offer filtering for any of our customers. You must upgrade to the DDoS prevention service. We've actually made a list of other companies that share our providers' downstream links in each facility and reached out to them. We get them to call up and complain to said tier1 provider that something is affecting our traffic. That usually gets filters installedotherwise no dice. If it's a legit capacity issue, you'll get a response whenever your sales guy comes back into the office. -richard
Re: Evaluating Tier 1 Internet providers
* Richard Hesse On Tue, Aug 27, 2013 at 12:14 PM, Joe Abley jab...@hopcount.ca wrote: - response you can expect when you call one day and say our 10GE is maxed out with inbound traffic from apparently everywhere, it has been going on for an hour, please help That was good for a laugh. If it's a DoS, you know what the answer already is. We no longer offer filtering for any of our customers. You must upgrade to the DDoS prevention service. We've actually made a list of other companies that share our providers' downstream links in each facility and reached out to them. We get them to call up and complain to said tier1 provider that something is affecting our traffic. That usually gets filters installedotherwise no dice. Several providers have a self-service blackholing functionality which may alleviate DDoS attacks. Typically you announce the attacked /32 or a /128 to your upstreams, tagged with some special blackhole community, and/or to a special multihop BGP session dedicated for blackholing purposes. Doing so will cause your upstreams to automatically drop the attack traffic within their network, *before* it gets to saturate your uplinks. Clearly, this is a blunt and last-resort type of tool which will cement the efficiency of the attack from a global perspective, but that may be an acceptable trade-off depending on the circumstances; you may prevent collateral damage from impacting your other customers, and by cutting out global attack traffic might enable the attacked customer to serve his primary markets just fine through local peering sessions, regional transits, and so forth. I'm not buying transit from a network that don't give me such blackholing functionality, FWIW. Tore
Re: Evaluating Tier 1 Internet providers
how is that really much different than reachability? If I look at my present Netflow results, it's actually a pretty amusing mix - lots of Netflix traffic (bear in mind we're a business ISP, not residential), Google (probably YouTube in there, I haven't dissected it thoroughly), Amazon, Yahoo, Microsoft/MSN, and that's all covered in the peering fabric connection. Outside of that, some private VPN-type traffic, I don't see a lot of government networks, just normal Internet browsing and email. Since I'm not at the Data Center much, I don't interact with the other customers there. (It's 150 miles away) Due to non-disclosure, the Data Center gang aren't much going to share their customer contact info with me. But it's a nice thought, for sure. -e- From: Michael Smith mksm...@mac.com To: Eric Louie elo...@yahoo.com Cc: nanog@nanog.org Sent: Tuesday, August 27, 2013 6:48 PM Subject: Re: Evaluating Tier 1 Internet providers You should also consider who exactly your customers (or you alone) want to reach. Are you mostly looking to connect to eyeball networks? Enterprise networks? Government networks? If you have some target networks you should do some due diligence to find out how well connected your various options are to the networks that mean the most to you. If possible, I would also recommend talking to other people that are in your data centers, if that's possible. You might find out about hidden vendor-specific gremlins in that location. Regards, Mike On Aug 27, 2013, at 12:02 PM, Eric Louie elo...@yahoo.com wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? routing stability BGP community offerings congestion issues BGP Peering relationships path diversity IPv6 table size Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) much appreciated, Eric Louie
Re: Evaluating Tier 1 Internet providers
On Aug 27, 2013, at 5:11 PM, Eric Louie elo...@yahoo.com wrote: Tier 1 = Internet backbone providers (United States - ATT, UUNET, Sprint, AboveNet/Zayo, Cogent, Qwest/CenturyLink, L3/GBLX). However, I might be better served with a Tier 2 for reachability as pointed out in another You may want to revise your list, and look at the 3rd parties that measure and rank this data. http://as-rank.caida.org/ http://www.renesys.com/2013/01/a-bakers-dozen-2012-edition/ You are missing a few networks that are important. Much of what someone considers a major network IMHO depends on how you scope them. Maybe you don't care about things not on your continent. Maybe you don't mind having a different ASN in Asia/Europe. Maybe you don't need to connect in Australia with the same routing policy. The real answer is it depends, and your criteria may not be the same as someone else. - Jared
Re: Evaluating Tier 1 Internet providers
On 8/27/2013 5:04 PM, Ben Hatton wrote: - time taken to turn around BGP import filter changes So much This... You don't realize how important this is until your nationwide provider takes 8 WEEKS to add one network to your (already set up and working for 20 other networks) peering. Then decides to charge you a fee for the change. Ben Hatton Network Systems Engineer Internet Rule 492b - Name and shame that provider.
Re: Evaluating Tier 1 Internet providers
On Aug 28, 2013, at 1:18 AM, Eric A Louie elo...@yahoo.com wrote: how is that really much different than reachability? If I look at my present Netflow results, it's actually a pretty amusing mix - lots of Netflix traffic (bear in mind we're a business ISP, not residential), Google (probably YouTube in there, I haven't dissected it thoroughly), Amazon, Yahoo, Microsoft/MSN, and that's all covered in the peering fabric connection. Outside of that, some private VPN-type traffic, I don't see a lot of government networks, just normal Internet browsing and email. It's really can reach versus how well can they reach. I can't any provider that would have less than a full view of the DFZ but, if your primary traffic is to Provider X, and one of your Tier 1's peers locally and the other peers in France, then you would look more closely at the closer one. Unless, of course, that local peer was saturated 99% of the time. Then France might be attractive. In short, it's good to do a lot of due diligence in finding out exactly how your providers of choice are connected to your destinations of choice. Mike Since I'm not at the Data Center much, I don't interact with the other customers there. (It's 150 miles away) Due to non-disclosure, the Data Center gang aren't much going to share their customer contact info with me. But it's a nice thought, for sure. -e- From: Michael Smith mksm...@mac.com To: Eric Louie elo...@yahoo.com Cc: nanog@nanog.org Sent: Tuesday, August 27, 2013 6:48 PM Subject: Re: Evaluating Tier 1 Internet providers You should also consider who exactly your customers (or you alone) want to reach. Are you mostly looking to connect to eyeball networks? Enterprise networks? Government networks? If you have some target networks you should do some due diligence to find out how well connected your various options are to the networks that mean the most to you. If possible, I would also recommend talking to other people that are in your data centers, if that's possible. You might find out about hidden vendor-specific gremlins in that location. Regards, Mike On Aug 27, 2013, at 12:02 PM, Eric Louie elo...@yahoo.com wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? routing stability BGP community offerings congestion issues BGP Peering relationships path diversity IPv6 table size Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) much appreciated, Eric Louie
Evaluating Tier 1 Internet providers
Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? routing stability BGP community offerings congestion issues BGP Peering relationships path diversity IPv6 table size Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) much appreciated, Eric Louie
Re: Evaluating Tier 1 Internet providers
On 2013-08-27, at 15:02, Eric Louie elo...@yahoo.com wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? routing stability BGP community offerings congestion issues BGP Peering relationships path diversity IPv6 table size I would add: - presence of staff in key locations (if 60 Hudson is a critical location for you, find out whether there's someone regularly present in or near the building to clean fibre and help run loopback tests when you need them) - expected time to clue when calling the support number (bonus points for being xkcd-806 compliant) - time taken to turn around BGP import filter changes - response you can expect when you call one day and say our 10GE is maxed out with inbound traffic from apparently everywhere, it has been going on for an hour, please help - billing accuracy, and turnaround time for questions raised about invoices received A lot of this comes down to conversations in the NANOG bar with people who have war stories to share. To that extent, I think reputation is a good indicator, so long as your sample size is reasonable, and depending on the amount of beer involved. One other thing to think about -- Tier 1 providers are transit free, so your can be reached by an IP packet from is naturally limited to specific peering relationships with other Tier 1 providers. Tier 2 providers (those who buy transit from a suitably-diverse set of Tier 1s) can insulate you from route fade due to peering spats. Joe
Re: Evaluating Tier 1 Internet providers
On Tue, 27 Aug 2013, Eric Louie wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? Define Tier 1 provider. I ask this because it's something that many people don't know what it means, but assume that Tier 1 Tier !=1. routing stability Routeviews.org can shed some light here. BGP community offerings If $provider has a page on www.peeringdb.com, they might publish a list of their BGP communities there. Other places to look would be the provider's whois/IRR entries, and on their respective websites, or the sales/marketing folks might be able to get this information for you. congestion issues There are various internet traffic report / weather report sites that can give you indirect insight into things like. By indirect, I mean that you might be able to infer things like congestion at a specific point based on what you see on those sites. BGP Peering relationships You can look at pages like www.peeringdb.com, and you will typically see if $provider is at an exchange, however the peering relationships that many providers have other providers (locations, speeds, etc) are confidential. path diversity You can ask $provider's sales and marketing folks, but there is no guarantee that you will get an answer (actual routes are considered confidential and proprietary information, despite the fact that a lot of providers' fiber ends up converging in a small handful of routes in some areas - i.e. many of them follow the same set of railroad tracks or cross a river at the same bridge, possibly even in the same conduit) or a correct answer (wave X might be re-groomed onto path Y without a whole lot of customer notification). IPv6 table size Sites like routeviews.org can give you some visibility here. Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) Absolutely reputation should be a factor. I would argue that Internet access is largely commoditized anymore (and has been for several years), so the real differentiators are cost and level of service. jms
Re: Evaluating Tier 1 Internet providers
On Aug 27, 2013, at 12:02 PM, Eric Louie elo...@yahoo.com wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. It's easy. Tier 1 is yourself. Tier 2 is your customers and your competitors. Tier 3 is your customers' customers, your competitors' customers, and your customers' competitors. But yes, I'm sure there are as many criteria as there are NANOG subscribers. But Joe Abley's are the correct ones. ducking -Bill signature.asc Description: Message signed with OpenPGP using GPGMail
RE: Evaluating Tier 1 Internet providers
Clued-in support is a good criteria. (I've been using a broker for some of my connections and there was virtually no value-add there, especially in the prefix-list modifications, and a liability in other MACs) That's a good point with the Tier 2 providers. So that begs the question, why wouldn't I just get my upstream from a Tier 2? (Because my management is under the perception that we're better off with Tier 1 providers, but that doesn't mean their perception is accurate) much appreciated, Eric Louie -Original Message- From: Joe Abley [mailto:jab...@hopcount.ca] Sent: Tuesday, August 27, 2013 12:15 PM To: Eric Louie Cc: nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers On 2013-08-27, at 15:02, Eric Louie elo...@yahoo.com wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? routing stability BGP community offerings congestion issues BGP Peering relationships path diversity IPv6 table size I would add: - presence of staff in key locations (if 60 Hudson is a critical location for you, find out whether there's someone regularly present in or near the building to clean fibre and help run loopback tests when you need them) - expected time to clue when calling the support number (bonus points for being xkcd-806 compliant) - time taken to turn around BGP import filter changes - response you can expect when you call one day and say our 10GE is maxed out with inbound traffic from apparently everywhere, it has been going on for an hour, please help - billing accuracy, and turnaround time for questions raised about invoices received A lot of this comes down to conversations in the NANOG bar with people who have war stories to share. To that extent, I think reputation is a good indicator, so long as your sample size is reasonable, and depending on the amount of beer involved. One other thing to think about -- Tier 1 providers are transit free, so your can be reached by an IP packet from is naturally limited to specific peering relationships with other Tier 1 providers. Tier 2 providers (those who buy transit from a suitably-diverse set of Tier 1s) can insulate you from route fade due to peering spats. Joe
RE: Evaluating Tier 1 Internet providers
Good stuff Justin - Any other criteria that you would use? much appreciated, Eric Louie -Original Message- From: Justin M. Streiner [mailto:strei...@cluebyfour.org] Sent: Tuesday, August 27, 2013 9:17 AM To: nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers On Tue, 27 Aug 2013, Eric Louie wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? Define Tier 1 provider. I ask this because it's something that many people don't know what it means, but assume that Tier 1 Tier !=1. routing stability Routeviews.org can shed some light here. BGP community offerings If $provider has a page on www.peeringdb.com, they might publish a list of their BGP communities there. Other places to look would be the provider's whois/IRR entries, and on their respective websites, or the sales/marketing folks might be able to get this information for you. congestion issues There are various internet traffic report / weather report sites that can give you indirect insight into things like. By indirect, I mean that you might be able to infer things like congestion at a specific point based on what you see on those sites. BGP Peering relationships You can look at pages like www.peeringdb.com, and you will typically see if $provider is at an exchange, however the peering relationships that many providers have other providers (locations, speeds, etc) are confidential. path diversity You can ask $provider's sales and marketing folks, but there is no guarantee that you will get an answer (actual routes are considered confidential and proprietary information, despite the fact that a lot of providers' fiber ends up converging in a small handful of routes in some areas - i.e. many of them follow the same set of railroad tracks or cross a river at the same bridge, possibly even in the same conduit) or a correct answer (wave X might be re-groomed onto path Y without a whole lot of customer notification). IPv6 table size Sites like routeviews.org can give you some visibility here. Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) Absolutely reputation should be a factor. I would argue that Internet access is largely commoditized anymore (and has been for several years), so the real differentiators are cost and level of service. jms
RE: Evaluating Tier 1 Internet providers
On Tue, 27 Aug 2013, Eric Louie wrote: Good stuff Justin - Any other criteria that you would use? Joe covered a lot of good stuff in his response. A few providers call themselves Tier 1, though the accuracy of those assertions is often suspect. The truth can be somewhat more complicated... and exactly how much more complicated isn't always clear until Provider X gets de-peered by Provider Y and finds themselves having to negotiate a quick fix, often by cutting a check. I would also ask people here who they have had very good experiences with, regardless of what tier the provider fits into. jms -Original Message- From: Justin M. Streiner [mailto:strei...@cluebyfour.org] Sent: Tuesday, August 27, 2013 9:17 AM To: nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers On Tue, 27 Aug 2013, Eric Louie wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? Define Tier 1 provider. I ask this because it's something that many people don't know what it means, but assume that Tier 1 Tier !=1. routing stability Routeviews.org can shed some light here. BGP community offerings If $provider has a page on www.peeringdb.com, they might publish a list of their BGP communities there. Other places to look would be the provider's whois/IRR entries, and on their respective websites, or the sales/marketing folks might be able to get this information for you. congestion issues There are various internet traffic report / weather report sites that can give you indirect insight into things like. By indirect, I mean that you might be able to infer things like congestion at a specific point based on what you see on those sites. BGP Peering relationships You can look at pages like www.peeringdb.com, and you will typically see if $provider is at an exchange, however the peering relationships that many providers have other providers (locations, speeds, etc) are confidential. path diversity You can ask $provider's sales and marketing folks, but there is no guarantee that you will get an answer (actual routes are considered confidential and proprietary information, despite the fact that a lot of providers' fiber ends up converging in a small handful of routes in some areas - i.e. many of them follow the same set of railroad tracks or cross a river at the same bridge, possibly even in the same conduit) or a correct answer (wave X might be re-groomed onto path Y without a whole lot of customer notification). IPv6 table size Sites like routeviews.org can give you some visibility here. Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) Absolutely reputation should be a factor. I would argue that Internet access is largely commoditized anymore (and has been for several years), so the real differentiators are cost and level of service. jms
Re: Evaluating Tier 1 Internet providers
On Tue, 27 Aug 2013 13:45:34 -0700, Eric Louie said: That's a good point with the Tier 2 providers. So that begs the question, why wouldn't I just get my upstream from a Tier 2? (Because my management is under the perception that we're better off with Tier 1 providers, but that doesn't mean their perception is accurate) The good thing about your upstream being a Tier 2 is that it usually means that if somebody's baking a peering cake, you're not one of the AS's that's suffering. Hmmm... if you're going for a connection to a Tier 1, maybe peering cakes per decade is a valid criterion? pgp7jWzZPadXc.pgp Description: PGP signature
Re: Evaluating Tier 1 Internet providers
http://www.renesys.com/products/ provide some guidance, but probably not the kind of detailed tech you want. Judging from my own experience, we have mostly been hit by limited path diversity everything seems fine support in the past. -- Tassos Eric Louie wrote on 27/8/2013 22:02: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? routing stability BGP community offerings congestion issues BGP Peering relationships path diversity IPv6 table size Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) much appreciated, Eric Louie
RE: Evaluating Tier 1 Internet providers
Tier 1 = Internet backbone providers (United States - ATT, UUNET, Sprint, AboveNet/Zayo, Cogent, Qwest/CenturyLink, L3/GBLX). However, I might be better served with a Tier 2 for reachability as pointed out in another response. When you say level of service, what are you referring to? Customer service? Service level agreement (which is pretty much the same across all the Tier 1's)? much appreciated, Eric Louie -Original Message- From: Justin M. Streiner [mailto:strei...@cluebyfour.org] Sent: Tuesday, August 27, 2013 9:17 AM To: nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers On Tue, 27 Aug 2013, Eric Louie wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? Define Tier 1 provider. I ask this because it's something that many people don't know what it means, but assume that Tier 1 Tier !=1. routing stability Routeviews.org can shed some light here. BGP community offerings If $provider has a page on www.peeringdb.com, they might publish a list of their BGP communities there. Other places to look would be the provider's whois/IRR entries, and on their respective websites, or the sales/marketing folks might be able to get this information for you. congestion issues There are various internet traffic report / weather report sites that can give you indirect insight into things like. By indirect, I mean that you might be able to infer things like congestion at a specific point based on what you see on those sites. BGP Peering relationships You can look at pages like www.peeringdb.com, and you will typically see if $provider is at an exchange, however the peering relationships that many providers have other providers (locations, speeds, etc) are confidential. path diversity You can ask $provider's sales and marketing folks, but there is no guarantee that you will get an answer (actual routes are considered confidential and proprietary information, despite the fact that a lot of providers' fiber ends up converging in a small handful of routes in some areas - i.e. many of them follow the same set of railroad tracks or cross a river at the same bridge, possibly even in the same conduit) or a correct answer (wave X might be re-groomed onto path Y without a whole lot of customer notification). IPv6 table size Sites like routeviews.org can give you some visibility here. Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) Absolutely reputation should be a factor. I would argue that Internet access is largely commoditized anymore (and has been for several years), so the real differentiators are cost and level of service. jms
RE: Evaluating Tier 1 Internet providers
I'm thinking that same thing, although after researching, the de-peering King is probably not a contender as one of our primary upstream connection. (And I don't have secondary or tertiary connections) much appreciated, Eric Louie -Original Message- From: valdis.kletni...@vt.edu [mailto:valdis.kletni...@vt.edu] Sent: Tuesday, August 27, 2013 2:03 PM To: Eric Louie Cc: nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers On Tue, 27 Aug 2013 13:45:34 -0700, Eric Louie said: That's a good point with the Tier 2 providers. So that begs the question, why wouldn't I just get my upstream from a Tier 2? (Because my management is under the perception that we're better off with Tier 1 providers, but that doesn't mean their perception is accurate) The good thing about your upstream being a Tier 2 is that it usually means that if somebody's baking a peering cake, you're not one of the AS's that's suffering. Hmmm... if you're going for a connection to a Tier 1, maybe peering cakes per decade is a valid criterion?
Re: Evaluating Tier 1 Internet providers
If you don't have secondary connectivity, then I don't suggest going with a Teir 1. Using a peer-only as a transit link is not something I would recommend in general unless you know what you are doing in that regard, and have designed around the inevitable peering issues related to that decision. -Blake On Tue, Aug 27, 2013 at 4:14 PM, Eric Louie elo...@yahoo.com wrote: I'm thinking that same thing, although after researching, the de-peering King is probably not a contender as one of our primary upstream connection. (And I don't have secondary or tertiary connections) much appreciated, Eric Louie -Original Message- From: valdis.kletni...@vt.edu [mailto:valdis.kletni...@vt.edu] Sent: Tuesday, August 27, 2013 2:03 PM To: Eric Louie Cc: nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers On Tue, 27 Aug 2013 13:45:34 -0700, Eric Louie said: That's a good point with the Tier 2 providers. So that begs the question, why wouldn't I just get my upstream from a Tier 2? (Because my management is under the perception that we're better off with Tier 1 providers, but that doesn't mean their perception is accurate) The good thing about your upstream being a Tier 2 is that it usually means that if somebody's baking a peering cake, you're not one of the AS's that's suffering. Hmmm... if you're going for a connection to a Tier 1, maybe peering cakes per decade is a valid criterion?
RE: Evaluating Tier 1 Internet providers
On Tue, 27 Aug 2013, Eric Louie wrote: Tier 1 = Internet backbone providers (United States - ATT, UUNET, Sprint, AboveNet/Zayo, Cogent, Qwest/CenturyLink, L3/GBLX). However, I might be better served with a Tier 2 for reachability as pointed out in another response. Some of those providers are probably not in the DFZ. I know Cogent has been involved in some peering spats in the past. I don't know off-hand if Zayo/Above lives in the DFZ. When you say level of service, what are you referring to? Customer service? Service level agreement (which is pretty much the same across all the Tier 1's)? Mainly customer service. Things like how easy it is to get a clued individual on the phone when there's an issue, turnaround time for things like BGP filter update requests. Like you mentioned, most providers' SLA terms are likely to look pretty similar if you were to compare them side-by-side. I would also look at which providers are on-net in your location, or would be willing to build into your location for a reasonable cost. One thing you want to avoid is all of your providers using the same local loop provider to get into the building, or local dark fiber providers using the same right-of-way / conduit / manhole to get into your building. Many providers might subcontract the physical last-mile construction to a local dark fiber provider. Entrance diversity and last-mile diversity is something you can probably have more influence over than how provider X gets between Chicago and New York. Many providers will build into your location if they're in your city if you either pay the build costs, or are purchasing enough service that the construction costs can amortized over the term of the contract. If they amortize, make sure you keep that in mind when the contract is up for re-negotiation, so they're no longer trying to ding you for construction costs that you've already paid :) jms
RE: Evaluating Tier 1 Internet providers
I appreciate that warning. The bigger truth is, No secondary/tertiary on that router/in that location. I do have iBGP with alternate providers through my core. much appreciated, Eric Louie -Original Message- From: Blake Dunlap [mailto:iki...@gmail.com] Sent: Tuesday, August 27, 2013 2:23 PM To: nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers If you don't have secondary connectivity, then I don't suggest going with a Teir 1. Using a peer-only as a transit link is not something I would recommend in general unless you know what you are doing in that regard, and have designed around the inevitable peering issues related to that decision. -Blake On Tue, Aug 27, 2013 at 4:14 PM, Eric Louie elo...@yahoo.com wrote: I'm thinking that same thing, although after researching, the de-peering King is probably not a contender as one of our primary upstream connection. (And I don't have secondary or tertiary connections) much appreciated, Eric Louie -Original Message- From: valdis.kletni...@vt.edu [mailto:valdis.kletni...@vt.edu] Sent: Tuesday, August 27, 2013 2:03 PM To: Eric Louie Cc: nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers On Tue, 27 Aug 2013 13:45:34 -0700, Eric Louie said: That's a good point with the Tier 2 providers. So that begs the question, why wouldn't I just get my upstream from a Tier 2? (Because my management is under the perception that we're better off with Tier 1 providers, but that doesn't mean their perception is accurate) The good thing about your upstream being a Tier 2 is that it usually means that if somebody's baking a peering cake, you're not one of the AS's that's suffering. Hmmm... if you're going for a connection to a Tier 1, maybe peering cakes per decade is a valid criterion?
Re: Evaluating Tier 1 Internet providers
To add some more from recent experiences.. Most of these are in colocation datacenters. - speed to handle your emergency support call. (recent experience, some tier1 can take a couple hours) - if support requires a portal opened ticket, is the staff to reset a password also 24/7. - Latency in your region.(recent experience: I removed 4 circuits because the backbones weren't the same in different areas). - Is you location a pop, metro ring or dedicated fiber elsewhere. - To get more specific, where is their peering in relationship to you. Strong peering not near you could mean a lot of extra latency just to get off their network. thanks, Bryan Socha
RE: Evaluating Tier 1 Internet providers
-Original Message- From: Justin M. Streiner [mailto:strei...@cluebyfour.org] Sent: Tuesday, August 27, 2013 10:36 AM To: nanog@nanog.org Subject: RE: Evaluating Tier 1 Internet providers On Tue, 27 Aug 2013, Eric Louie wrote: I would also look at which providers are on-net in your location, or would be willing to build into your location for a reasonable cost. One thing you want to avoid is all of your providers using the same local loop provider to get into the building, or local dark fiber providers using the same right-of-way / conduit / manhole to get into your building. Many providers might subcontract the physical last-mile construction to a local dark fiber provider. Entrance diversity and last-mile diversity is something you can probably have more influence over than how provider X gets between Chicago and New York. The only thing I'm looking at are on-net solutions - luckily or unluckily we are at data center locations (carrier neutral) so my choices are limited to the on-nets that they already have (I'm not going through the pain of bringing in a new one) and most of them are offering free install Many providers will build into your location if they're in your city if you either pay the build costs, or are purchasing enough service that the construction costs can amortized over the term of the contract. If they amortize, make sure you keep that in mind when the contract is up for re-negotiation, so they're no longer trying to ding you for construction costs that you've already paid :) jms
RE: Evaluating Tier 1 Internet providers
From: Bryan Socha [mailto:br...@serverstack.com] Sent: Tuesday, August 27, 2013 2:45 PM To: Eric Louie; nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers To add some more from recent experiences.. Most of these are in colocation datacenters. [EL] I'm colocated too. - speed to handle your emergency support call. (recent experience, some tier1 can take a couple hours) [EL] time to respond / time to resolve are good ones (hard to get them to provide the true values, though) - if support requires a portal opened ticket, is the staff to reset a password also 24/7. - Latency in your region.(recent experience: I removed 4 circuits because the backbones weren't the same in different areas). - Is you location a pop, metro ring or dedicated fiber elsewhere. - To get more specific, where is their peering in relationship to you. Strong peering not near you could mean a lot of extra latency just to get off their network. [EL] How many hops to their edge? Will they admit that? can I get a traceroute? (however, this is in downtown LA so I'm guessing it's close to the edge thanks, Bryan Socha
Re: Evaluating Tier 1 Internet providers
If this was previously mentioned, my apologies. The time they can respond to a PNI upgrade. If you have an existing 10G and wish to add another. Can this be provisioned off the same device to form a LAG or can they only provide ECMP. May not be something you can evaluate at contract signing, but it can quickly become an issue when you need it. On Tue, Aug 27, 2013 at 12:02 PM, Eric Louie elo...@yahoo.com wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? routing stability BGP community offerings congestion issues BGP Peering relationships path diversity IPv6 table size Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) much appreciated, Eric Louie -- Bill Blackford Logged into reality and abusing my sudo privileges.
Re: Evaluating Tier 1 Internet providers
- speed to handle your emergency support call. (recent experience, some tier1 can take a couple hours) *[EL] * time to respond / time to resolve are good ones (hard to get them to provide the true values, though) Call and pretend your a customer with an emergency.You might be surprised how long it takes the first person to be on the call with you. - To get more specific, where is their peering in relationship to you. Strong peering not near you could mean a lot of extra latency just to get off their network. *[EL] *“How many hops to their edge”? Will they admit that? can I get a traceroute? (however, this is in downtown LA so I’m guessing it’s close to the edge * * This one can be harder to get any answers on depending on who you are. You can ask what locations they have most of their peering with. Also ask for a POP list they are located in. Usually they are marked with the type of service each building is (pop vs metro ring vs extension).unless it's a private peer, I woudlnt' expect any peering at locations that are not pops and you can see what is nearby your location that is a pop. Somethign else I just thought of that I do ask providers. Ask how they get into your building. If they are using some sort of metro ring between their routers make sure your not about to screw yourself with no diversity when that ring needs to be worked on. Thanks, Bryan
Re: Evaluating Tier 1 Internet providers
On Tue, Aug 27, 2013 at 3:02 PM, Eric Louie elo...@yahoo.com wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? Billing issues such as: attitude during a billing dispute traceability and accountability (Which service is this 35 cent blah fee attached to?) zombie service rate (Bills showing up for long-ago cancelled products) flexibility (I want you to send me two bills, each for half of that. You can't? Why not?) nickle and dime (There's a $100 monthly rental fee for that 50 foot cat-5 cable!? Really!?) Also, abuse desk knee-jerkiness. If someone reports a problem originating from my system, how much leeway do I have to fix it before you decide to fix it for me? If some knucklehead with a port-scanning worm earns me a no-notice cut off, you and I will have words. At the same time, I don't want to fund someone who would turn a blind eye. Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) Reputations are well earned and are certainly a factor. They're heavily qualitative, though. I don't know that's it's practical or useful to measure them. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Evaluating Tier 1 Internet providers
You should also consider who exactly your customers (or you alone) want to reach. Are you mostly looking to connect to eyeball networks? Enterprise networks? Government networks? If you have some target networks you should do some due diligence to find out how well connected your various options are to the networks that mean the most to you. If possible, I would also recommend talking to other people that are in your data centers, if that's possible. You might find out about hidden vendor-specific gremlins in that location. Regards, Mike On Aug 27, 2013, at 12:02 PM, Eric Louie elo...@yahoo.com wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? routing stability BGP community offerings congestion issues BGP Peering relationships path diversity IPv6 table size Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) much appreciated, Eric Louie
Re: Evaluating Tier 1 Internet providers
On Tue, Aug 27, 2013 at 3:02 PM, Eric Louie elo...@yahoo.com wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? BGP Peering relationships Peering policy. A tier 1 with an open peering policy would get all my money. Even a semi-open policy (bring your network to any of these neutral locations at your cost and we'll peer settlement-free for our regional routes) would be worth encouraging through the purchase of transit services. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Evaluating Tier 1 Internet providers
- time taken to turn around BGP import filter changes So much This... You don't realize how important this is until your nationwide provider takes 8 WEEKS to add one network to your (already set up and working for 20 other networks) peering. Then decides to charge you a fee for the change. Ben Hatton Network Systems Engineer On Tue, Aug 27, 2013 at 1:05 PM, Justin M. Streiner strei...@cluebyfour.org wrote: On Tue, 27 Aug 2013, Eric Louie wrote: Good stuff Justin - Any other criteria that you would use? Joe covered a lot of good stuff in his response. A few providers call themselves Tier 1, though the accuracy of those assertions is often suspect. The truth can be somewhat more complicated... and exactly how much more complicated isn't always clear until Provider X gets de-peered by Provider Y and finds themselves having to negotiate a quick fix, often by cutting a check. I would also ask people here who they have had very good experiences with, regardless of what tier the provider fits into. jms -Original Message- From: Justin M. Streiner [mailto:streiner@cluebyfour.**orgstrei...@cluebyfour.org ] Sent: Tuesday, August 27, 2013 9:17 AM To: nanog@nanog.org Subject: Re: Evaluating Tier 1 Internet providers On Tue, 27 Aug 2013, Eric Louie wrote: Based on various conversation threads on Nanog I've come up with a few criteria for evaluating Tier 1 providers. I'm open to add other criteria - what would you add to this list? And how would I get a quantitative or qualitative measure of it? Define Tier 1 provider. I ask this because it's something that many people don't know what it means, but assume that Tier 1 Tier !=1. routing stability Routeviews.org can shed some light here. BGP community offerings If $provider has a page on www.peeringdb.com, they might publish a list of their BGP communities there. Other places to look would be the provider's whois/IRR entries, and on their respective websites, or the sales/marketing folks might be able to get this information for you. congestion issues There are various internet traffic report / weather report sites that can give you indirect insight into things like. By indirect, I mean that you might be able to infer things like congestion at a specific point based on what you see on those sites. BGP Peering relationships You can look at pages like www.peeringdb.com, and you will typically see if $provider is at an exchange, however the peering relationships that many providers have other providers (locations, speeds, etc) are confidential. path diversity You can ask $provider's sales and marketing folks, but there is no guarantee that you will get an answer (actual routes are considered confidential and proprietary information, despite the fact that a lot of providers' fiber ends up converging in a small handful of routes in some areas - i.e. many of them follow the same set of railroad tracks or cross a river at the same bridge, possibly even in the same conduit) or a correct answer (wave X might be re-groomed onto path Y without a whole lot of customer notification). IPv6 table size Sites like routeviews.org can give you some visibility here. Seems like everyone offers 5 9's service, 45 ms coast-to-coast, 24x7 customer support, 100/1Gbps/10Gbps with various DIR/CIR and burst rates. I'm shopping for new service and want to do better than choosing on reputation. (or, is reputation also a criteria?) Absolutely reputation should be a factor. I would argue that Internet access is largely commoditized anymore (and has been for several years), so the real differentiators are cost and level of service. jms
Re: Evaluating Tier 1 Internet providers
On Tue, 27 Aug 2013, Ben Hatton wrote: - time taken to turn around BGP import filter changes So much This... You don't realize how important this is until your nationwide provider takes 8 WEEKS to add one network to your (already set up and working for 20 other networks) peering. Then decides to charge you a fee for the change. I think after a week I would be tearing my account rep a new one, and then threatening to dump them as soon as the contract was up... 8 weeks? There is absolutely no excuse I would buy for that, though I might give style points if someone told me the dog ate the ticket or something... jms
