IMO, It almost comes down to running two networks, your core backbone, if geographically separated, run BGP, OSPF, using route reflectors, full routing table at each router, then hand off BGP to an internal Mikrotik to do your management for end users, etc. You have to be very procedure oriented when doing a lot of the stuff described in the previous emails, we ingest several Transit connections throughout the US and I think balancing the routes to make everything run efficiently is the biggest challenge, especially being geographically separated.
In the last few years I have seen a lot more mikrotik gear in datacenters I have visited (Just saw a new deployment two days ago in Austin, with several Mikrotiks), i agree with Paul, it is all based on budget, familiarity with existing staff, etc. I can tell you one thing, I sleep better at night knowing the network we have built runs on Brocade. It has its caveats like Cisco, Juniper, etc, but so far from a performance standpoint they have been rock solid for us. We have around a dozen Brocade routers deployed, mix of CER and MLXe. Erich Kaiser The Fusion Network er...@gotfusion.net Office: 630-621-4804 Cell: 630-777-9291 On Sat, Sep 3, 2016 at 6:12 AM, Paul Stewart <p...@paulstewart.org> wrote: > Hey Faisal …. Lots of really great questions! ;) > > > > Route reflectors in BGP configuration are optional but at some point it’ll > make complete sense depending on the size of the network. > > > > I wouldn’t say one is more complicated than the other …. But after > stepping back from this a bit and thinking about it – OSPF is going to seem > easier/simpler to set up though…. Would agree with that for sure. > > > > As you mentioned, a lot of the responses were Cisco/Juniper related vs > Microtik and perhaps it is related to network size, traffic levels, > budgets, services and other factors …. In the world I live in, I see almost > zero Microtik’s where a lot of folks on this list are surrounded in them. > I would believe that many folks on the list primary business is WISP and > then there are some folks (myself included) where WISP is a small part (but > important part) of the business. > > > > It’s two different ways of skinning the cat but dependent on what you want > to accomplish, what network size you are working with, MPLS capabilities, > if you need full Internet tables in parts of the network for downstream > customers… basically, in my opinion, as your network grows and the > services/requirements change then you may find moving from the “OSPF model” > to the “BGP model” necessary … and you may not. J > > > > Thanks, > > Paul > > > > *From:* Af [mailto:af-boun...@afmug.com] *On Behalf Of *Faisal Imtiaz > *Sent:* September 2, 2016 9:07 PM > *To:* af@afmug.com > *Subject:* Re: [AFMUG] (OSPF + ibgp) / formerly Mikrotik OSPF weirdness > > > > Thanks Paul, > > > > So we have established that we can do this via ibgp/route reflectors/ and > communities/filters to manage and control the route distribution, and we > rely on loopback ip's to be known by all routers using OSFP (or some other > underlying dynamic protocol) > > > > We started this conversation by a claim that doing so via ibgp is simpler > or better.... > > > > After flushing out all the details, it is obvious that > > a) it is a bit of an intricate process, filters and communities have to be > planned. > > b) it requires an in-dept knowledge of BGP, or at least a fair amount of > comfort level. > > c) it requires to follow best practices.. > > d) and the configuration requires at least two Route Reflectors and at > least two sessions per router (one to each of the RR). > > > > > > and we have not gotten into traffic engineering .... (influencing the path > over one link vs the other). > > > > now if we compare this to an OSPF Setup... > > > > a) planning and setting up the areas is good to have, most tend not to pay > attention to it. > > b) dealing OSPF properly does require a bit of in depth knowledge, > winging it does not (OSFP is much more forgiving ?) > > c) for things to work well, it is recommended to follow best practices > > d) configuration tends to be simpler > > > > So, would it be better for someone managing lots of routers to spend a bit > of time learning OSPF intricacies and follow the best practices vs trading > this for an ibgp configuration ? > > > > I would also like to point out that the folks who responded to the ibgp > setup, appear to be using Juniper and or Cisco routers.. > > I cannot help in making the observation that folks who are deploying > Juniper or Cisco or even Brocade, tend to do a lot more on their routers > (thus have a fewer boxes ) , while those of us who are deploying Mikrotik > Routers, tend to deploy more in quantity of these (distributing the > functions, rather than trying to do everything on one or two boxes). > > > > Going back to my original question... is this just two different ways of > skinning the cat, and the choice of one vs the other is simply a matter of > personal choice... or is one method truly better than the other method ? if > yes, can someone please share as to why ? > > > > Thanks > > > > :) > > > > > > > > Faisal Imtiaz > Snappy Internet & Telecom > 7266 SW 48 Street > Miami, FL 33155 > Tel: 305 663 5518 x 232 > > Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net > > > ------------------------------ > > *From: *"Paul Stewart" <p...@paulstewart.org> > *To: *af@afmug.com > *Sent: *Friday, September 2, 2016 8:13:52 PM > *Subject: *Re: [AFMUG] (OSPF + ibgp) / formerly Mikrotik OSPF weirdness > > Been on holidays so apologies for posting on what might be older messages… > > > > Communities is one way to do it (filter that only accepts certain > communities). In the Juniper world you can limit it simply by which > “family” you accept on BGP neighbor as well. Some folks also separate > their “Internet routes” from their “internal routes” into separate routing > tables all together > > > > Paul > > > > > > *From:* Af [mailto:af-boun...@afmug.com <af-boun...@afmug.com>] *On > Behalf Of *Bruce Robertson > *Sent:* August 30, 2016 7:23 PM > *To:* af@afmug.com > *Subject:* Re: [AFMUG] (OSPF + ibgp) / formerly Mikrotik OSPF weirdness > > > > Communities. Lemme know if you need more detail on that. I'm a little > pressed for time right now. > > On 08/30/2016 03:23 PM, Faisal Imtiaz wrote: > > I have a follow up question in regards to this... > > > > How do you prevent having ebgp routes being sent to your smaller routers > which are doing ibgp with the Route Reflectors ? > > > > Are you using filters ? or some there method ? > > > > > > Thanks. > > > > Faisal Imtiaz > Snappy Internet & Telecom > 7266 SW 48 Street > Miami, FL 33155 > Tel: 305 663 5518 x 232 > > Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net > > > ------------------------------ > > *From: *"Jesse DuPont" <jesse.dup...@celeritycorp.net> > <jesse.dup...@celeritycorp.net> > *To: *af@afmug.com > *Sent: *Friday, August 26, 2016 11:36:42 PM > *Subject: *Re: [AFMUG] (OSPF + ibgp) / formerly Mikrotik OSPF weirdness > > For me, it was a bit of an experiment, but I have ended up liking it. Yes, > it does add some overhead, but I didn't have to add routers to be the route > reflectors - I just chose two routers which provided good geographic > redundancy balanced with being as well-connected as possible to the rest of > the routers and checked the "route reflect to peers" box. Route reflecting > is really no more intensive than just BGP peering; probably most already > know this, but the only different between a route reflector and a non-route > reflector is that at route reflector is allowed to break the iBGP rule of > not disseminating routes learned from one peer to another peer. > > One of the things I really like about using BGP for access prefixes is > that I don't have to mess with filters or using non-backbone areas and > area-ranges to summarize pools used for things like PPPoE. It's nice that > more recent versions of MikroTik automate adding the U route of a > summarized area-range after the first connected route shows up, but with > BGP, I simply add the prefix to Networks and it's done. > > Another advantage, albeit a "band-aid" one is that if I'm having some link > quality issue that is ultimately causing OSPF to lose adjacency (packet > loss causing dropped Hello's, for example, or some jackass carrier > providing a circuit that upgrades their platform and they don't read the > release notes and multicast gets dropped...), I can deploy a small handful > of static routes to improve stability slightly until I can resolve the > issue (just a small time saver). > > Obviously, none of this functionality REQUIRES the use of BGP and it can > all be done using OSPF. Indeed, while I'm using OSPF + iBGP in my WISP, the > telco I'm also the network architect/engineer at uses only OSPF as the IGP > and we have thousands of internal OSPF routes and dozens of routers in the > backbone area (along with others in non-backbone areas) and it's extremely > stable. I think its easy to misinterpret problems which manifest themselves > as OSPF issues, but are really just OSPF reacting to some other condition; > the canary in the coal mine, if you will. > > <rant> If you're having issues with OSPF losing adjacencies or changing > from full to down or full to init, you've got some problem with the link. > Period. OSPF is not the problem. OSPF has been stable in MikroTiks since > 3.x.</rant> > > *Jesse DuPont* > > Network Architect > email: jesse.dup...@celeritycorp.net > Celerity Networks LLC > > Celerity Broadband LLC > Like us! facebook.com/celeritynetworksllc > > Like us! facebook.com/celeritybroadband > > On 8/26/16 1:16 PM, Faisal Imtiaz wrote: > > So just for the sake of a technical discussion... > > > > In your opinion, what is the merit of such a config (osfp + ibgp) ? > > > > It can be argued that such a config, > > a) Still depends on OSPF functioning. > > b) Layer an additional dynamic protocol on top of it (ibgp) > > c) Requires additional Routers (route reflectors). > > > > If the merit of such an approach is to manage manage OSFP behavior in a > more granular fashion, Why not use the those features as they are > available in OSPF / Best Practices... > > (OSFP best practices, suggest that, don't advertise connected or > static routes, setup all interfaces as passive, and control prefix > advertisements via the network section of OSPF). > > > > OSPF also tends to be the most common denominator (protocol) across > different mfg. Bgp being the 2nd. > > > > Regards > > > > Faisal Imtiaz > Snappy Internet & Telecom > 7266 SW 48 Street > Miami, FL 33155 > Tel: 305 663 5518 x 232 > > Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net > > > ------------------------------ > > *From: *"Jesse DuPont" <jesse.dup...@celeritycorp.net> > <jesse.dup...@celeritycorp.net> > *To: *af@afmug.com > *Sent: *Friday, August 26, 2016 12:03:58 AM > *Subject: *Re: [AFMUG] Mikrotik OSPF weirdness > > Right, PTP and loopback prefixes are distributed with OSPF (and possibly > management subnets for radios) and "access" network prefixes > (customer-facing) are distributed via iBGP. > I have two of my routers configured as BGP route reflectors and all other > routers peer with only these two; this solves the full mesh and provides > redundancy. > > *Jesse DuPont* > > Network Architect > email: jesse.dup...@celeritycorp.net > Celerity Networks LLC > > Celerity Broadband LLC > Like us! facebook.com/celeritynetworksllc > > Like us! facebook.com/celeritybroadband > > On 8/25/16 8:40 PM, David Milholen wrote: > > He may have meant only have the ptp and loopback addresses listed in > networks > > > > > > On 8/25/2016 9:31 PM, Mike Hammett wrote: > > I've heard this concept a few times now. I'm not sure how only using OSPF > for the loopbacks works. > > > > ----- > Mike Hammett > Intelligent Computing Solutions <http://www.ics-il.com/> > <https://www.facebook.com/ICSIL> > <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> > <https://www.linkedin.com/company/intelligent-computing-solutions> > <https://twitter.com/ICSIL> > Midwest Internet Exchange <http://www.midwest-ix.com/> > <https://www.facebook.com/mdwestix> > <https://www.linkedin.com/company/midwest-internet-exchange> > <https://twitter.com/mdwestix> > The Brothers WISP <http://www.thebrotherswisp.com/> > <https://www.facebook.com/thebrotherswisp> > > > <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> > ------------------------------ > > *From: *"Bruce Robertson" <br...@pooh.com> <br...@pooh.com> > *To: *af@afmug.com > *Sent: *Thursday, August 25, 2016 6:28:43 PM > *Subject: *Re: [AFMUG] Mikrotik OSPF weirdness > > I've said it before, and been argued with... this is one of many reasons > why you use iBGP to distribute {customer, dynamic pool, server subnets, > anything} routes, and use OSPF *only* to distribute router loopback > addresses.� All your weird OSPF problems will go away.� My apologies if > I'm misunderstanding the problem, but my point still stands. > > On 08/25/2016 10:22 AM, Robert Haas wrote: > > Alright, this problem has raised it head again on my network since I > started to renumber some PPPoE pools. > > Customer gets a new IP address via PPPoE x.x.x.208/32 (from x.x.x.192/27 > pool). Customer can�t surf and I can�t ping them from my office: > > � > > [office] � [Bernie Router] � [Braggcity Router] � [Ross Router] � > [Hayti Router] � [customer] > > � > > A traceroute from my office dies @ the Bernie router but I am not getting > any type of ICMP response from the Bernie router ie no ICMP Host > Unreachable/Dest unreachable etc � just blackholes after my office router. > > A traceroute from the Customer to the office again dies at the Bernie > router with no type of response. > > � > > Checking the routing table on the Bernie router shows a valid route > pointing to the Braggcity router. It is also in the OSPF LSA�s. > > -- > > Another customer gets x.x.x.207/32 and has no issue at all. > > � > > -- > > Force the original customer to a new ip address of x.x.x.205/32 and the > service starts working again. > > � > > -- > > � > > Now � even though there is no valid route to x.x.x.208/32 in the routing > table � traffic destined to the x.x.x.208/32 IP is still getting > blackholed.. I should be getting a Destination host unreachable from the > Bernie router. > > � > > This is correct the correct response .206 is not being used and there is > no route to it: > > C:\Users\netadmin>ping x.x.x.206 > > � > > Pinging x.x.x.206 with 32 bytes of data: > > Reply from y.y.y.1: Destination host unreachable. > > Reply from y.y.y.1: Destination host unreachable. > > � > > Ping statistics for x.x.x.206: > > ��� Packets: Sent = 2, Received = 2, Lost = 0 (0% loss), > > � > > C:\Users\netadmin>tracert 74.91.65.206 > > � > > Tracing route to host-x.x.x.206.bpsnetworks.com [x.x.x.206] > > over a maximum of 30 hops: > > � > > � 1���� 6 ms���� 6 ms���� 7 ms� z.z.z.z > > � 2���� 6 ms���� 6 ms���� 6 ms� > y.bpsnetworks.com [y.y.y.1] > > � 3� y.bpsnetworks.com [y.y.y.1] �reports: Destination host > unreachable. > > � > > Trace complete. > > � > > This is what I see to x.x.x.208 even though it is not being used and there > is no route to it. > > C:\Users\netadmin>ping x.x.x.208 > > � > > Pinging x.x.x.208 with 32 bytes of data: > > Request timed out. > > Request timed out. > > � > > Ping statistics for x.x.x.208: > > ��� Packets: Sent = 2, Received = 0, Lost = 2 (100% loss), > > � > > C:\Users\netadmin>tracert x.x.x.208 > > � > > Tracing route to host-x.x.x.208.bpsnetworks.com [x.x.x.208] > > over a maximum of 30 hops: > > � > > � 1���� 6 ms���� 6 ms���� 6 ms� z.z.z.z > > � 2���� *������� *������� > *���� Request timed out. > > � 3���� *������� *���� ^C > > � > > -- > > � > > I�ve verified there is no firewall that would affect the traffic � I > even put an accept rule in the forward chain for both the source and > destination of x.x.x.208 and neither increment at all. So the traffic is > not even making out of the routing flow and into the firewall.. > > � > > Any pointers are where to start troubleshooting next? > > > > > > > > -- > > > > > > !DSPAM:2,57c60796289379943469318! > > > > > >
