BGP per-flow load balancing between eBGP and iBGP learned prefix
Hello, I have a Load Balancer that uses a default route to a VRRP IP hosted between two Juniper MX80 routers. Each MX router has a single BGP feed from the same provider and each session is currently receiving only a default route. I'd like to load balance my outbound traffic across the two links. More specifically I want to balance traffic to a particular prefix across the two links so even if i was receiving full routes trying to manually split prefixes to a second upstream VRRP instance wouldn't help. I believe the typical way to do this would be to run an IGP between the LB and the MX's so that each MX advertises a default route to the LB and let ECMP take care of it. I'm not averse to this long term, but I have a short term need and the LB requires a license to participate in an IGP. Is there another way? I should note that the LB will not let me enter multiple default or identical static routes so I can't try spinning up a second VRRP session and adding a second default route on the LB to it. Is it possible to let BGP do the load balancing? From what I can tell this would be trivial if both of the provider links were terminated on the same router via a few multipath config lines but because they are on different routers I seem to be running into the "eBGP > iBGP" Path selection criteria which prevents the routes from being equal. I appreciate any ideas! regards, -andy
using ARIN IP space outside of ARIN region
We're looking at building into a DC in Europe this year and I wanted to run a few questions by the community and make sure I'm not too far off course. We currently have v4 space from ARIN and operate a multihomed datacenter in the US. This thread from September 2012, though the reverse of my situation, makes it seem like I should be able to work with ARIN to satisfy all of our v4 and v6 needs regardless of where in the world I plan to advertise the space. Obviously this would be really convenient for us since we already have a relationship with ARIN. http://mailman.nanog.org/pipermail/nanog/2012-September/052137.html So my questions are: * Is it ok/recommended to advertise v4 ARIN space (say a /23) to my upstream ISPs in Europe? * Is it ok/recommended to advertise v6 ARIN space to my upstream ISPs in Europe? (I suspect we'll get a /44 block to carve up since we fit in the "more than one but less than 12 sites" category) * Should I use a single AS for both North America and European data centers? It will be the same small team managing them today but it's not like the sites are linked together to form any kind of transit network. Also, is there a European version of RADB with which I should plan to register my prefixes? thanks! - andy
tools and techniques to pinpoint and respond to loss on a path
Hi, Does anyone have any recommendations on how to pinpoint and react to packet loss across the internet? preferably in an automated fashion. For detection I'm currently looking at trying smoketrace to run from inside my network, but I'd love to be able to run traceroutes from my edge routers triggered during periods of loss. I have Juniper MX80s on one end- which I'm hopeful I'll be able to cobble together some combo of RPM and event scripting to kick off a traceroute. We have Cisco4900Ms on the other end and maybe the same thing is possible but I'm not so sure. I'd love to hear other suggestions and experience for detection and also for options on what I might be able to do when loss is detected on a path. In my specific situation I control equipment on both ends of the path that I care about with details below. we are a hosted service company and we currently have two data centers, DC A and DC B. DC A uses juniper MX routers, advertises our own IP space and takes full BGP feeds from two providers, ISPs A1 and A2. At DC B we have a smaller installation and instead take redundant drops (and IP space) from a single provider, ISP B1, who then peers upstream with two providers, B2 and B3 We have a fairly consistent bi-directional stream of traffic between DC A and DC B. Both of ISP A1 and A2 have good peering with ISP B2 so under normal network conditions traffic flows across ISP B1 to B2 and then to either ISP A1 or A2 oversimplified ascii pic showing only the normal best paths: -- ISP A1--ISP B2-- DC A--| |--- ISP B1 - DC B -- ISP A2--ISP B2-- with increasing frequency we've been experiencing packet loss along the path from DC A to DC B. Usually the periods of loss are brief, 30 seconds to a minute, but they are total blackouts. I'd like to be able to collect enough relevant data to pinpoint the trouble spot as much as possible so I can take it to the ISPs and request a solution. The blackouts are so quick that it's impossible to log in and get a trace- hence the desire to automate it. I can provide more details off list if helpful- I'm trying not to vilify anyone- especially without copious amounts of data points. As a side question, what should my expectation be regarding packet loss when sending packets from point A to point B across multiple providers across the internet? Is 30 seconds to a minute of blackout between two destinations every couple of weeks par for the course? My directly connected ISPs offer me an SLA, but what should I reasonably expect from them when one of their upstream peers (or a peer of their peers) has issues? If this turns out to be BGP reconvergence or similar do I have any options? many thanks, -andy
RE: tools and techniques to pinpoint and respond to loss on a path
> From: Blake Dunlap [mailto:iki...@gmail.com] > While any provider will attempt to fix peer / upstream issues as they can, any > SLA you would have is between two points on their private network, not > from point A to point Z that they have no control over across multiple peers > and the public internet itself. makes sense- thanks for confirming > The much more common design is using a single > provider for each thread between sites. Then at least you have an end-to- > end SLA in effect, as well as a single entity that is responsible for the > entire > link in question. > > This sounds like you're trying to achieve private link IGP / FRR level site > to site > failover/convergence across the public internet. Perhaps you should rethink > your goals here or your design? Kind of- I can actually tolerate the blips, but I want to be able to measure and track them in such a way that I know where the loss is occurring. If a particular path is reconverging more often than should be reasonably expected I want to be able to prove it within reason. We also have a customer who happens to host at DC B with the same connectivity. Every time there is one of these blips their alerting fires off a thousand messages and they open a ticket with us. I'd like to be able to show them some good data on the path during the blip so we back a discussion along the lines of "live with it, or pay to privately connect to us". -andy > -Blake > > On Mon, Jul 15, 2013 at 4:18 PM, Andy Litzinger > wrote: > Hi, > > Does anyone have any recommendations on how to pinpoint and react to > packet loss across the internet? preferably in an automated fashion. For > detection I'm currently looking at trying smoketrace to run from inside my > network, but I'd love to be able to run traceroutes from my edge routers > triggered during periods of loss. I have Juniper MX80s on one end- which I'm > hopeful I'll be able to cobble together some combo of RPM and event > scripting to kick off a traceroute. We have Cisco4900Ms on the other end and > maybe the same thing is possible but I'm not so sure. > > I'd love to hear other suggestions and experience for detection and also for > options on what I might be able to do when loss is detected on a path. > > In my specific situation I control equipment on both ends of the path that I > care about with details below. > > we are a hosted service company and we currently have two data centers, > DC A and DC B. DC A uses juniper MX routers, advertises our own IP space > and takes full BGP feeds from two providers, ISPs A1 and A2. At DC B we > have a smaller installation and instead take redundant drops (and IP space) > from a single provider, ISP B1, who then peers upstream with two providers, > B2 and B3 > > We have a fairly consistent bi-directional stream of traffic between DC A and > DC B. Both of ISP A1 and A2 have good peering with ISP B2 so under normal > network conditions traffic flows across ISP B1 to B2 and then to either ISP A1 > or A2 > > oversimplified ascii pic showing only the normal best paths: > > -- ISP A1--ISP B2-- DC A-- > | |--- ISP > B1 - DC B > -- ISP A2--ISP B2-- > > > with increasing frequency we've been experiencing packet loss along the > path from DC A to DC B. Usually the periods of loss are brief, 30 seconds > to a > minute, but they are total blackouts. > > I'd like to be able to collect enough relevant data to pinpoint the trouble > spot as much as possible so I can take it to the ISPs and request a > solution. The blackouts are so quick that it's impossible to log in and get a > trace- hence the desire to automate it. > > I can provide more details off list if helpful- I'm trying not to vilify > anyone- > especially without copious amounts of data points. > > As a side question, what should my expectation be regarding packet loss > when sending packets from point A to point B across multiple providers > across the internet? Is 30 seconds to a minute of blackout between two > destinations every couple of weeks par for the course? My directly > connected ISPs offer me an SLA, but what should I reasonably expect from > them when one of their upstream peers (or a peer of their peers) has > issues? If this turns out to be BGP reconvergence or similar do I have any > options? > > many thanks, > -andy
Advice on v4 NAT for farm of file transfer clients
Hi all, We have a pool of around 100 file transfer clients. They reach out to publicly addressed servers on the net to get and put files. Rather than burn 100 public v4 addresses for the clients, we've traditionally had these guys behind a firewall performing source NAT/PAT overloading about 10 IPs. Recently we've been seeing increases in the amount of throughput to/from the servers through the FW. Within the next 12 mos I expect we'll want to support 10Gbps. Since buying a firewall that supports 10Gbps is fairly expensive I thought i'd seek out alternative ideas before we blindly purchase a bigger firewall. Also, a stateful firewall seems like a bit of overkill for what is actually required. I'm confident we can limit our FTP support to passive connections which should remove the requirement of using a device that supports active FTP (i.e. application inspection). currently we're using a Juniper SRX550 to do this (which replaced an overwhelmed ASA 5520). Avg packet size we see according to the SRX is 1000 bytes. thanks! -andy
Validating possible BGP MITM attack
Hello, we use BGPMon.net to monitor our BGP announcements. This morning we received two possible BGP MITM alerts for two of our prefixes detected by a single BGPMon probe located in China. I've reached out to BGPMon to see how much credence I should give to an alert from a single probe location, but I'm interested in community feedback as well. The alert detailed that one of our /23 prefixes has been broken into /24 specifics and the AS Path shows a peering relationship with us that does not exist: 131477(Shanghai Huajan) 38478(Sunny Vision LTD) 3491(PCCW Global) 14042 (me) We do not peer directly with PCCW Global. I'm going to reach out to them directly to see if they may have done anything by accident, but presuming they haven't and the path is spoofed, can I prove that? How can I detect if traffic is indeed swinging through that hijacked path? How worried should I be and what are my options for resolving the situation? thanks! -andy
Re: Validating possible BGP MITM attack
Hi Steve and Job, Same here- I didn't actually see my prefixes leaked anywhere I could check, but I couldn't check near China where BGPmon's probe was complaining. So I was glad it didn't seem to be spreading, but still concerned that there may have been a large area (China) where my traffic was getting hijacked. The alert did clear after around 18 minutes. Presuming it was a route optimizer and the issue was ongoing, what would be the suggested course of action? reach out to those 2 AS owners and see if they could stop it? Or is it something I just have to live with as a traffic engineering solution they are using and mark the alerts as false positives? thanks! -andy On Thu, Aug 31, 2017 at 10:23 AM, Steve Feldman wrote: > Interesting. We also got similar BGPMon alerts about disaggregated > portions of couple of our prefixes. I didn't see any of the bad prefixes > in route-views, though. > > The AS paths in the alerts started with "131477 38478 ..." and looked > valid after that. Job's suggestion would explain that. > > Steve > > On Aug 31, 2017, at 10:01 AM, Job Snijders wrote: > > Hi Andy, > > It smells like someone in 38478 or 131477 is using Noction or some other > BGP "optimizer" that injects hijacks for the purpose of traffic > engineering. :-( > > Kind regards, > > Job > > On Thu, 31 Aug 2017 at 19:38, Andy Litzinger com> > wrote: > > Hello, > we use BGPMon.net to monitor our BGP announcements. This morning we > received two possible BGP MITM alerts for two of our prefixes detected by a > single BGPMon probe located in China. I've reached out to BGPMon to see > how much credence I should give to an alert from a single probe location, > but I'm interested in community feedback as well. > > The alert detailed that one of our /23 prefixes has been broken into /24 > specifics and the AS Path shows a peering relationship with us that does > not exist: > 131477(Shanghai Huajan) 38478(Sunny Vision LTD) 3491(PCCW Global) 14042 > (me) > > We do not peer directly with PCCW Global. I'm going to reach out to them > directly to see if they may have done anything by accident, but presuming > they haven't and the path is spoofed, can I prove that? How can I detect > if traffic is indeed swinging through that hijacked path? How worried > should I be and what are my options for resolving the situation? > > thanks! > -andy > > > >
Re: Validating possible BGP MITM attack
FYI - I did get a response back from BGPMon- they concur with Job: "Hi Andy, unfortunately we had a peer sending us a polluted BGP views. Most likely using a BGP optimizer that is making up new paths. We've reached out to 131477 and dropped the session with them. This was most likely 131477 making up the paths. And not seen wider on the Internet. We'll work on making sure that cases like this will not cause bgpmon alerts going forward, by detecting these false alerts better." -andy On Thu, Aug 31, 2017 at 7:01 AM, Andy Litzinger < andy.litzinger.li...@gmail.com> wrote: > Hello, > we use BGPMon.net to monitor our BGP announcements. This morning we > received two possible BGP MITM alerts for two of our prefixes detected by a > single BGPMon probe located in China. I've reached out to BGPMon to see > how much credence I should give to an alert from a single probe location, > but I'm interested in community feedback as well. > > The alert detailed that one of our /23 prefixes has been broken into /24 > specifics and the AS Path shows a peering relationship with us that does > not exist: > 131477(Shanghai Huajan) 38478(Sunny Vision LTD) 3491(PCCW Global) 14042 > (me) > > We do not peer directly with PCCW Global. I'm going to reach out to them > directly to see if they may have done anything by accident, but presuming > they haven't and the path is spoofed, can I prove that? How can I detect > if traffic is indeed swinging through that hijacked path? How worried > should I be and what are my options for resolving the situation? > > thanks! > -andy >
validating reachability via an ISP
Hi all, I have an enterprise network and do not provide transit. In one of our datacenters we have our own prefixes and rely on two ISPs as BGP neighbors to provide global reachability for our prefixes. One is a large regional provider and the other is a large global provider. Recently we took our link to the global provider offline to perform maintenance on our router. Nearly immediately we were hit with alerts that our prefix was unreachable and BGPMon alerted that nearly 80 AS's noted our route had been withdrawn. We were not unreachable from every AS, but we certainly were from some of the largest. The root cause is that the our prefix is not being adequately re-distributed globally by the regional ISP. This is unexpected and we are working through this with them now. My question is, how can I monitor global reachability for a prefix via this or any specific provider I use over time? Are there various route-servers I can programmatically query for my prefix and get results that include AS paths? Then I could verify that an "acceptable" number of paths exist that include the AS of the all the ISPs I rely upon. And what would an "acceptable" number of alternate paths be? thanks in advance, -andy
Re: validating reachability via an ISP
Hi Alex, Thanks for the link to the qrator API. The get-all-paths is interesting and the output does seem to show exactly the type of data I'm looking for. Is there some place on the site that lists who Qrator peers with to get your data? I searched through the site and couldn't find it. I'm trying to get an idea of the breadth of data and whether i can comfortably rely on it giving me a global view. Also, how do I authorize programmatic requests to the API? I created a user and can use it to run the queries from the web page, but I don't see how to use it directly with the API. thanks! -andy On Thu, Mar 29, 2018 at 8:23 AM, Alexander Azimov wrote: > Hi Andy, > > You can use Qrator.Radar API: https://api.radar.qrator.net/. > The get-all-paths method will return the set of active paths for selected > prefix. > > PS: I'm sorry if you receive this message two times, something funny is > happening with my emails on NANOG mailing list. > > 2018-03-29 18:09 GMT+03:00 Alexander Azimov : > >> Hi Andy, >> >> You can use Qrator.Radar API: https://api.radar.qrator.net/. >> The get-all-paths method will return the set of active paths for selected >> prefix. >> >> >> 2018-03-29 2:22 GMT+03:00 Andy Litzinger >> : >> >>> Hi all, >>> I have an enterprise network and do not provide transit. In one of our >>> datacenters we have our own prefixes and rely on two ISPs as BGP >>> neighbors >>> to provide global reachability for our prefixes. One is a large regional >>> provider and the other is a large global provider. >>> >>> Recently we took our link to the global provider offline to perform >>> maintenance on our router. Nearly immediately we were hit with alerts >>> that >>> our prefix was unreachable and BGPMon alerted that nearly 80 AS's noted >>> our >>> route had been withdrawn. We were not unreachable from every AS, but we >>> certainly were from some of the largest. >>> >>> The root cause is that the our prefix is not being adequately >>> re-distributed globally by the regional ISP. This is unexpected and we >>> are >>> working through this with them now. >>> >>> My question is, how can I monitor global reachability for a prefix via >>> this >>> or any specific provider I use over time? Are there various >>> route-servers >>> I can programmatically query for my prefix and get results that include >>> AS >>> paths? Then I could verify that an "acceptable" number of paths exist >>> that >>> include the AS of the all the ISPs I rely upon. And what would an >>> "acceptable" number of alternate paths be? >>> >>> >>> thanks in advance, >>> -andy >>> >> >> >> >> -- >> | Alexander Azimov | HLL l QRATOR >> | tel.: +7 499 241 81 92 >> | mob.: +7 915 360 08 86 >> | skype: mitradir >> | mailto: a...@qrator.net >> | visit: www.qrator.net >> > > > > -- > | Alexander Azimov | HLL l QRATOR > | tel.: +7 499 241 81 92 > | mob.: +7 915 360 08 86 > | skype: mitradir > | mailto: a...@qrator.net > | visit: www.qrator.net >
Re: validating reachability via an ISP
Hi Andrew, thanks for sharing your repo, it looks very relevant and I should be able to get it to do what I want with a slight tweak to the regex's. Hopefully the route-views collectors or some other ones I dig up will be able to adequately show at least 3 AS path hops through my regional provider to get to my AS/prefix. I really want to see AS paths where there are at least 2 different AS hops before my regional providers AS. This will help me feel good that the global partners of my regional ISP are re-advertising my prefix to their peers. Otherwise if the AS path only includes the first and second hop upstream I can only infer that the 2nd hop AS is accepting routes from the 1st hop AS (my regional ISP), but not that the 2nd hop AS is actually re-advertising the route to anyone else. thanks! -andy On Thu, Mar 29, 2018 at 7:51 AM, Andrew Wentzell wrote: > On Wed, Mar 28, 2018 at 7:22 PM, Andy Litzinger > wrote: > > Hi all, > > I have an enterprise network and do not provide transit. In one of our > > datacenters we have our own prefixes and rely on two ISPs as BGP > neighbors > > to provide global reachability for our prefixes. One is a large regional > > provider and the other is a large global provider. > > > > Recently we took our link to the global provider offline to perform > > maintenance on our router. Nearly immediately we were hit with alerts > that > > our prefix was unreachable and BGPMon alerted that nearly 80 AS's noted > our > > route had been withdrawn. We were not unreachable from every AS, but we > > certainly were from some of the largest. > > > > The root cause is that the our prefix is not being adequately > > re-distributed globally by the regional ISP. This is unexpected and we > are > > working through this with them now. > > > > My question is, how can I monitor global reachability for a prefix via > this > > or any specific provider I use over time? Are there various > route-servers > > I can programmatically query for my prefix and get results that include > AS > > paths? Then I could verify that an "acceptable" number of paths exist > that > > include the AS of the all the ISPs I rely upon. And what would an > > "acceptable" number of alternate paths be? > > I did something similar a few years ago, by querying routeviews and > validating AS paths using python and pexpect. You could adapt it for > your use pretty easily. The code's here: > > https://github.com/awentzell/check-as-path >
Re: validating reachability via an ISP
Hi Andy, The root cause was they regional ISP was failing to advertise my prefix due to a mistake in their export policy. While I'm glad we were able to figure out the issue I'm generally more interested in figuring out a way that I can programmatically monitor that my ISPs are providing me with good reachability, even when their route isn't necessarily the best/installed route to me. I do have route objects defined in an IRR for this prefix. Perhaps you wouldn't mind explaining to me how route-server or ISP operators generally validate route-objects before accepting routes? Especially in the case where I'm not peering with your route server but my ISP is. Do you query the IRR DB to recurse from the ISP AS to my AS and validate route objects there? thanks! -andy On Thu, Apr 5, 2018 at 12:49 AM, Andy Davidson wrote: > > > > > > On 29/03/2018, 00:22, Andy Litzinger > wrote: > > > >> The root cause is that the our prefix is not being adequately > >> re-distributed globally by the regional ISP. This is unexpected and we > are > >> working through this with them now. > > Hi, Andy — > > Are you failing to advertise it, or are they filtering it on ingress, or > are they failing to send it to their other peers? > > One configuration mishap which is starting to come along more and more > partial or poor reachability caused by route objects which are not > correctly published in the IRRDB. It is going to be essential to make sure > that you have properly recorded IRR route objects in, for instance, RADB. > More BGP speakers properly filter their peers using information that is > published there. Avoid future reachability problems by checking this today! > > Yours, > A friendly route-server operator with strict filtering > > -a > > > > -- > Andy DavidsonAsteroid International BV > https://www.asteroidhq.com@asteroidhq @andyd > -- > Local interconnection. Where you need it. > >
RE: How not to make an error page (was: OT: www.Amazon.com down?)
I've no idea what Amazon uses for Load Balancers, but I'm pretty sure that error message is the default error message served up by a Netscaler LB if no web services are available in the pool... -andy > -Original Message- > From: Kevin Day [mailto:[EMAIL PROTECTED] > Sent: Friday, June 06, 2008 11:40 AM > To: Lasher, Donn > Cc: nanog@nanog.org > Subject: How not to make an error page (was: OT: www.Amazon.com down?) > > > On Jun 6, 2008, at 1:24 PM, Lasher, Donn wrote: > > > Checked, and doublechecked, not just me > > > > www.amazon.com returns: > > > > Http/1.1 Service Unavailable > > > > Anyone have a URL for a network/etc status page, or info on the > > outage? > > Been that way for a while this morning. > > > > -donn > > > > > > Even worse, the page they're displaying is actually a HTTP 200 > response code(OK/no error), with no "Don't cache this" header - which > means their error page is considered cacheable by some browsers/ > proxies. So, you may find users who tried to visit Amazon while they > were down are still seeing it down long after they fix it. > > Lesson to high profile websites: add these to your error pages so you > don't have people complaining you're still down long after you're > fixed. > > * Don't return a 200 response code. Use 500 or 503. Nothing from 2xx > or 4xx. > * Add a "Cache-control: no-cache, must-revalidate, max-age=0" header, > as well as an "Expires: 0" header for good measure. > * If your server is really borked and you can't add headers at all, > add '' to the > section. That's not as good, but helps at least on the browser end. > * If possible, add a timestamp to the page somewhere (even if it's in > an HTML comment) so you can troubleshoot with users still seeing the > error. > > -- Kevin >
AS numbers and multiple site best practices
Are there any best practices or guidelines surrounding whether or not one should use the same or unique AS numbers when advertising via BGP from 2 or more physically separate locations? Each location would be advertising at least their own unique /24. My specific scenario is that we are moving our QA Lab to a datacenter that we will multi-home with two providers via BGP. We also plan to multi-home our corporate office with two providers (not likely to be the same providers) also via BGP. We currently have an AS that is in use for our multi-homed production data center. In the interest of keeping production totally segregated from QA/corp I would prefer to not use our production datacenter AS for our QA Lab or corporate network, but I've had trouble finding any technical reason not to use it. ARIN is asking for a detailed technical explanation to justify my request. Thanks in advance, -andy
RE: AS numbers and multiple site best practices
> > I've had trouble finding any technical reason not to use it. > > What is important to you about having QA and Corporate use separate AS > numbers? Does using the same AS number result in a reduction of > separation? For my part it's mostly a desire to make sure that changes to QA or Corp BGP configs could never impact BGP for our Production datacenter. So far it looks like it may just be a fear of the unknown on my part as I can't think of a good example of how one might actually affect one BGP installation by making changes to another BGP installation purely based on sharing an AS number (clearly you could have impact if you are advertising the same space from both locations).
