SAFNOG-3: Agenda Finalized
Hello all. Pleased to inform you that the SAFNOG-3 agenda is now full! You can view the agenda here: http://safnog.org/ We have a very detailed and exciting line-up, sure to appeal to different parts of our community, such as: - Data Centres. - Peering. - IPv4 Address exhaustion. - SDN/NFV. - FTTH. - Cloud Computing. - IPv6. - Internet Security. - DNS. - BGP. - e.t.c. We also have two very powerful keynote speakers delivering critical and vital information at this year's meeting: - Our opening keynote will be presented by Mr. Pascal Hoba, CEO of UbuntuNet Alliance. - Our closing keynote will be presented by Mr. Alan Barrett, CEO of AFRINIC. Bio information about all of this year's speakers will be available in the coming days. Please check the web site regularly for these updates. Please register your attendance for SAFNOG-3, if you have not yet done so. We look forward to seeing you all in Durban, next week! Cheers, Mark Tinka On Behalf of the SAFNOG Management Committee
BGP AS# migration from IOS to IOS-XR
Dear All, Does anybody had to migrate a entire BGP transit AS# from Cisco IOS to IOS-XR ? Our main concern is BGP, especially conversion from route-map, communities, conditional advertisement, template, peer-group up to RPL. Cisco offer a doc how to migrate from IOS to XR of about 40pages, but it's quite old (XR 3.2) and not so interesting. There are also a lot of documentation about XR/RPL, but I was not able to find a good doc about RPL, and how to translate complex IOS config to XR. Any experience on BGP best pratice with XR, af-group, session-group, neighbor-group, communities implementation from existing IOS config ? And how to you manage RPL editing? I mean with IOS you have some completion on TAB keystroke, but as RPL has to be edited within a text editor, you loose this kind of 'help'. Maybe we have to re-think our config from scrash :-). Thank in advance, Best regards, Marcel
Re: BGP AS# migration from IOS to IOS-XR
marcel.duregards--- via NANOG wrote: > Cisco offer a doc how to migrate from IOS to XR of about 40pages, but > it's quite old (XR 3.2) and not so interesting. that doc is still relevant. > And how to you manage RPL editing? I mean with IOS you have some > completion on TAB keystroke, but as RPL has to be edited within a text > editor, you loose this kind of 'help'. You can edit RPL from the command-line too, with tab completion and inline help. > Maybe we have to re-think our config from scrash that is a good option in this situation. RPL is significantly more flexible than what's available on vanilla IOS, and you would benefit from learning RPL, then standing back and looking carefully at what you're doing with route routing policy to see how it can be abstracted into well-structured RPL. There are a number of major new features: RPL functions can call other RPL functions, which you can't really do with route-maps (leading to lots of duplication for similar configuration), and passing variables into RPL functions. You can use these features to build up structured RPL configuration mechanisms which give a lot of flexibility and power. Also, XR is better from the point of view of automation. If it makes sense to build automation into your network, this would provide a good opportunity. Nick
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
Hi, but isn't peer A prefix-out a synonym for peer B prefix-in, that will lead to the same result, e.g. a BGP teardown? I just feel that this will add another factor, that people will not use or abuse: neigh $x max-out infinite What about adding an option to the BGP session that A & B do agree on a fixed number of prefixes in both directions, so Bs prefix-in could be As prefix-out automatically? Jörg On 31 Aug 2017, at 7:01, Alejandro Acosta wrote: What a terrific idea..., simple & useful El 29/8/17 a las 1:41 p.m., Michael Still escribió: I agree a max-prefix outbound could potentially be useful and would hopefully not be too terribly difficult to implement for most vendors. Perhaps RFC4486 would need to be updated to reflect this as a possibility as well?
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
Dear Jörg, On Thu, Aug 31, 2017 at 12:50:58PM +0200, Jörg Kost wrote: > but isn't peer A prefix-out a synonym for peer B prefix-in, that will > lead to the same result, e.g. a BGP teardown? > > I just feel that this will add another factor, that people will not > use or abuse: neigh $x max-out infinite I feel you may be overlooking a key aspect here: Currently all of us rely on our peer's 'inbound maximum prefix limit', and obviously these are not always set correctly. An 'outbound maximum prefix limit' offers networks that care about the rest of the world the option to 'self-destruct' the EBGP session in order to protect others. An 'outbound maximum prefix limit' is a 'permissionless' feature in that you do not require cooperation or support from your peering partner at the other end of the sessio in order to deploy the 'self-destruct to protect' mechanism. If you don't want to use it, then don't. If people configure "neighbor $x max-out infinite" that is fine by me, at least they made a conscience choice, it is no worse than today, and it is clearly documented in the running-configuration what the ramifications of the EBGP session could be. > What about adding an option to the BGP session that A & B do agree on > a fixed number of prefixes in both directions, so Bs prefix-in could > be As prefix-out automatically? I prefer unilateral permissionless mechanisms. Adding new negotiable options to BGP sessions is a lot of work and requires both parties to run software that supports the new feature, whatever it is. Anything that can be done without requiring your peer's cooperation will be more robust. Kind regards, Job
Re: BGP AS# migration from IOS to IOS-XR
Get in touch with your Cisco SE or partner. Cisco SE's have access to a conversion tool that takes in an IOS config and spits out an XR config. It's usually about 80-95% correct. It even shows you sections that are not in use and can be removed. On Thu, Aug 31, 2017 at 5:39 AM, Nick Hilliard wrote: > marcel.duregards--- via NANOG wrote: > > Cisco offer a doc how to migrate from IOS to XR of about 40pages, but > > it's quite old (XR 3.2) and not so interesting. > > that doc is still relevant. > > > And how to you manage RPL editing? I mean with IOS you have some > > completion on TAB keystroke, but as RPL has to be edited within a text > > editor, you loose this kind of 'help'. > > You can edit RPL from the command-line too, with tab completion and > inline help. > > > Maybe we have to re-think our config from scrash > > that is a good option in this situation. RPL is significantly more > flexible than what's available on vanilla IOS, and you would benefit > from learning RPL, then standing back and looking carefully at what > you're doing with route routing policy to see how it can be abstracted > into well-structured RPL. > > There are a number of major new features: RPL functions can call other > RPL functions, which you can't really do with route-maps (leading to > lots of duplication for similar configuration), and passing variables > into RPL functions. You can use these features to build up structured > RPL configuration mechanisms which give a lot of flexibility and power. > > Also, XR is better from the point of view of automation. If it makes > sense to build automation into your network, this would provide a good > opportunity. > > Nick >
Re: Cogent BCP-38
Really surprised that AS174 put in place any anti-spoofing. We are bgp-transit customer of CGNT and received traffic originated from RFC1918 on our public p2p link with them On 15.08.2017 17:36, Ben Russell wrote: > Could someone from Cogent contact me off-list? We are having an issue with > one of our downstream customers who is multi-homed to another carrier. The > end customer is advertising their prefix to both us and the other carrier. > Both us and the other carrier peer with 174. However, if the prefix is > preferred through us and the outbound traffic flows over the other carrier it > is dropped. We suspect uRPF-strict on the other carriers Cogent link. We > are working together with the other carrier and have a ticket open the help > desk seem to be confused. Any help would be appreciated. Thanks > > > Ben Russell > Senior Network Engineer > Stratus Networks > (309)370-3174 > [logo]
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
Hi, but in reality you will factorise and summarize outbound and inbound numbers, create spare room for sessions and failover scenarios and therefore leaks and especially partial leaks can still occur. In another example scenario the BGP process may not only shutdown the session to B, that has run into an outbound warning, but all other sessions to prevent "leaks". Last-resort the router will only judge by the number of the prefixes and therefore could shutdown himself by accident, especially if this router was not the origin. That could be a global headache ;-) Jörg On 31 Aug 2017, at 13:06, Job Snijders wrote: Dear Jörg, On Thu, Aug 31, 2017 at 12:50:58PM +0200, Jörg Kost wrote: but isn't peer A prefix-out a synonym for peer B prefix-in, that will lead to the same result, e.g. a BGP teardown? I just feel that this will add another factor, that people will not use or abuse: neigh $x max-out infinite I feel you may be overlooking a key aspect here: Currently all of us rely on our peer's 'inbound maximum prefix limit', and obviously these are not always set correctly. An 'outbound maximum prefix limit' offers networks that care about the rest of the world the option to 'self-destruct' the EBGP session in order to protect others.
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
I think what this is is just a new (potentially) knob that can be turned. If you don't want to turn it that's your deal, you run your network how you want. There's been no suggestion that there be some explicit default value or even that its turned on by default so behavior won't change unless configured and if you configure it, you are on the hook for knowing how that might affect the behavior of your network. I would expect BGP speakers (router vendors / software devs) to implement this in a way such that it would syslog or otherwise trigger when the number of outbound prefixes reaches a specific percentage (of configured limit) or hard number so that either an engineer could respond or automation take place to do something in response. On Thu, Aug 31, 2017 at 9:21 AM, Jörg Kost wrote: > Hi, > > but in reality you will factorise and summarize outbound and inbound > numbers, create spare room for sessions and failover scenarios and therefore > leaks and especially partial leaks can still occur. > > In another example scenario the BGP process may not only shutdown the > session to B, that has run into an outbound warning, but all other sessions > to prevent "leaks". Last-resort the router will only judge by the number of > the prefixes and therefore could shutdown himself by accident, especially if > this router was not the origin. That could be a global headache ;-) > > Jörg > > > On 31 Aug 2017, at 13:06, Job Snijders wrote: > >> Dear Jörg, >> >> On Thu, Aug 31, 2017 at 12:50:58PM +0200, Jörg Kost wrote: >>> >>> but isn't peer A prefix-out a synonym for peer B prefix-in, that will >>> lead to the same result, e.g. a BGP teardown? >>> >>> I just feel that this will add another factor, that people will not >>> use or abuse: neigh $x max-out infinite >> >> >> I feel you may be overlooking a key aspect here: Currently all of us >> rely on our peer's 'inbound maximum prefix limit', and obviously these >> are not always set correctly. An 'outbound maximum prefix limit' offers >> networks that care about the rest of the world the option to >> 'self-destruct' the EBGP session in order to protect others. >> > -- [stillwa...@gmail.com ~]$ cat .signature cat: .signature: No such file or directory [stillwa...@gmail.com ~]$
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
In a message written on Thu, Aug 31, 2017 at 12:50:58PM +0200, J??rg Kost wrote: > What about adding an option to the BGP session that A & B do agree on a > fixed number of prefixes in both directions, so Bs prefix-in could be As > prefix-out automatically? As others have pointed out, that's harder to do, but there's a different reason it may not be desireable. If a peer sets a limit to tear down the session with no automatic reset, forcing a call to their NOC to get a human to reset it then it may be advantageous to set your side to tear down at N-1 prefixes. That way you can insure restoration at the speed of your NOC, and not at the speed of your peer's. -- Leo Bicknell - bickn...@ufp.org PGP keys at http://www.ufp.org/~bicknell/ signature.asc Description: PGP signature
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
On Thu, Aug 31, 2017 at 11:24 AM, Leo Bicknell wrote: > In a message written on Thu, Aug 31, 2017 at 12:50:58PM +0200, J??rg Kost > wrote: > > What about adding an option to the BGP session that A & B do agree on a > > fixed number of prefixes in both directions, so Bs prefix-in could be As > > prefix-out automatically? > > As others have pointed out, that's harder to do, but there's a > different reason it may not be desireable. > > If a peer sets a limit to tear down the session with no automatic > reset, forcing a call to their NOC to get a human to reset it then > it may be advantageous to set your side to tear down at N-1 prefixes. > That way you can insure restoration at the speed of your NOC, and > not at the speed of your peer's. > Generally controlling your own destiny is preferred, I agree with that. I think also being able to say: "I shouldn't ever send more than 477 routes, let's round up for ops reasons to 1k max" seems like a great way to make your network safer for the rest of the network. Yes, people (as job and others noted) could set 'too high' limits... ok, that's their decision to make. Yes, maybe in the 523 prefixes that leak in my example there could be some affected party... I think it's pretty unlikely that there will be widescale damage from a small number of routes leaking, there are certainly plenty of documented cases of wide scale problems from full table leaks though :) Yes, your sessions might bounce or stay-down... it's probably better to go down on a some peers and have control to get back up on your side, than to cause widescale outages due to a full table leak. i'd be in favor of a output max prefix limit knob.
Validating possible BGP MITM attack
Hello, we use BGPMon.net to monitor our BGP announcements. This morning we received two possible BGP MITM alerts for two of our prefixes detected by a single BGPMon probe located in China. I've reached out to BGPMon to see how much credence I should give to an alert from a single probe location, but I'm interested in community feedback as well. The alert detailed that one of our /23 prefixes has been broken into /24 specifics and the AS Path shows a peering relationship with us that does not exist: 131477(Shanghai Huajan) 38478(Sunny Vision LTD) 3491(PCCW Global) 14042 (me) We do not peer directly with PCCW Global. I'm going to reach out to them directly to see if they may have done anything by accident, but presuming they haven't and the path is spoofed, can I prove that? How can I detect if traffic is indeed swinging through that hijacked path? How worried should I be and what are my options for resolving the situation? thanks! -andy
Re: Validating possible BGP MITM attack
Hi Andy, It smells like someone in 38478 or 131477 is using Noction or some other BGP "optimizer" that injects hijacks for the purpose of traffic engineering. :-( Kind regards, Job On Thu, 31 Aug 2017 at 19:38, Andy Litzinger wrote: > Hello, > we use BGPMon.net to monitor our BGP announcements. This morning we > received two possible BGP MITM alerts for two of our prefixes detected by a > single BGPMon probe located in China. I've reached out to BGPMon to see > how much credence I should give to an alert from a single probe location, > but I'm interested in community feedback as well. > > The alert detailed that one of our /23 prefixes has been broken into /24 > specifics and the AS Path shows a peering relationship with us that does > not exist: > 131477(Shanghai Huajan) 38478(Sunny Vision LTD) 3491(PCCW Global) 14042 > (me) > > We do not peer directly with PCCW Global. I'm going to reach out to them > directly to see if they may have done anything by accident, but presuming > they haven't and the path is spoofed, can I prove that? How can I detect > if traffic is indeed swinging through that hijacked path? How worried > should I be and what are my options for resolving the situation? > > thanks! > -andy >
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
I guess you're looking into something similar to https://tools.ietf.org/html/draft-keyur-idr-bgp-prefix-limit-orf. -- Tassos Jörg Kost wrote on 31/8/17 13:50: > > > What about adding an option to the BGP session that A & B do agree on a fixed > number of prefixes in both directions, so Bs prefix-in could be As prefix-out > automatically? > > Jörg >
Re: Validating possible BGP MITM attack
Interesting. We also got similar BGPMon alerts about disaggregated portions of couple of our prefixes. I didn't see any of the bad prefixes in route-views, though. The AS paths in the alerts started with "131477 38478 ..." and looked valid after that. Job's suggestion would explain that. Steve > On Aug 31, 2017, at 10:01 AM, Job Snijders wrote: > > Hi Andy, > > It smells like someone in 38478 or 131477 is using Noction or some other > BGP "optimizer" that injects hijacks for the purpose of traffic > engineering. :-( > > Kind regards, > > Job > > On Thu, 31 Aug 2017 at 19:38, Andy Litzinger > wrote: > >> Hello, >> we use BGPMon.net to monitor our BGP announcements. This morning we >> received two possible BGP MITM alerts for two of our prefixes detected by a >> single BGPMon probe located in China. I've reached out to BGPMon to see >> how much credence I should give to an alert from a single probe location, >> but I'm interested in community feedback as well. >> >> The alert detailed that one of our /23 prefixes has been broken into /24 >> specifics and the AS Path shows a peering relationship with us that does >> not exist: >> 131477(Shanghai Huajan) 38478(Sunny Vision LTD) 3491(PCCW Global) 14042 >> (me) >> >> We do not peer directly with PCCW Global. I'm going to reach out to them >> directly to see if they may have done anything by accident, but presuming >> they haven't and the path is spoofed, can I prove that? How can I detect >> if traffic is indeed swinging through that hijacked path? How worried >> should I be and what are my options for resolving the situation? >> >> thanks! >> -andy >> >
Re: Validating possible BGP MITM attack
On Thu, Aug 31, 2017 at 1:23 PM, Steve Feldman wrote: > Interesting. We also got similar BGPMon alerts about disaggregated > portions of couple of our prefixes. I didn't see any of the bad prefixes in > route-views, though. > > The AS paths in the alerts started with "131477 38478 ..." and looked > valid after that. Job's suggestion would explain that. > > Looking back at a bunch of historical route leak incidents... they often seem to be this sort of thing :( I think I normally term them; "internap box problems" I think internap doesn't even really sell that product anymore though :( so now I'll call them 'noction problems' instead I guess. lack of outbound route filtering can be painful yo! > Steve > > > On Aug 31, 2017, at 10:01 AM, Job Snijders wrote: > > > > Hi Andy, > > > > It smells like someone in 38478 or 131477 is using Noction or some other > > BGP "optimizer" that injects hijacks for the purpose of traffic > > engineering. :-( > > > > Kind regards, > > > > Job > > > > On Thu, 31 Aug 2017 at 19:38, Andy Litzinger < > andy.litzinger.li...@gmail.com> > > wrote: > > > >> Hello, > >> we use BGPMon.net to monitor our BGP announcements. This morning we > >> received two possible BGP MITM alerts for two of our prefixes detected > by a > >> single BGPMon probe located in China. I've reached out to BGPMon to see > >> how much credence I should give to an alert from a single probe > location, > >> but I'm interested in community feedback as well. > >> > >> The alert detailed that one of our /23 prefixes has been broken into /24 > >> specifics and the AS Path shows a peering relationship with us that does > >> not exist: > >> 131477(Shanghai Huajan) 38478(Sunny Vision LTD) 3491(PCCW Global) 14042 > >> (me) > >> > >> We do not peer directly with PCCW Global. I'm going to reach out to > them > >> directly to see if they may have done anything by accident, but > presuming > >> they haven't and the path is spoofed, can I prove that? How can I > detect > >> if traffic is indeed swinging through that hijacked path? How worried > >> should I be and what are my options for resolving the situation? > >> > >> thanks! > >> -andy > >> > > > >
Re: Validating possible BGP MITM attack
Hi Steve and Job, Same here- I didn't actually see my prefixes leaked anywhere I could check, but I couldn't check near China where BGPmon's probe was complaining. So I was glad it didn't seem to be spreading, but still concerned that there may have been a large area (China) where my traffic was getting hijacked. The alert did clear after around 18 minutes. Presuming it was a route optimizer and the issue was ongoing, what would be the suggested course of action? reach out to those 2 AS owners and see if they could stop it? Or is it something I just have to live with as a traffic engineering solution they are using and mark the alerts as false positives? thanks! -andy On Thu, Aug 31, 2017 at 10:23 AM, Steve Feldman wrote: > Interesting. We also got similar BGPMon alerts about disaggregated > portions of couple of our prefixes. I didn't see any of the bad prefixes > in route-views, though. > > The AS paths in the alerts started with "131477 38478 ..." and looked > valid after that. Job's suggestion would explain that. > > Steve > > On Aug 31, 2017, at 10:01 AM, Job Snijders wrote: > > Hi Andy, > > It smells like someone in 38478 or 131477 is using Noction or some other > BGP "optimizer" that injects hijacks for the purpose of traffic > engineering. :-( > > Kind regards, > > Job > > On Thu, 31 Aug 2017 at 19:38, Andy Litzinger com> > wrote: > > Hello, > we use BGPMon.net to monitor our BGP announcements. This morning we > received two possible BGP MITM alerts for two of our prefixes detected by a > single BGPMon probe located in China. I've reached out to BGPMon to see > how much credence I should give to an alert from a single probe location, > but I'm interested in community feedback as well. > > The alert detailed that one of our /23 prefixes has been broken into /24 > specifics and the AS Path shows a peering relationship with us that does > not exist: > 131477(Shanghai Huajan) 38478(Sunny Vision LTD) 3491(PCCW Global) 14042 > (me) > > We do not peer directly with PCCW Global. I'm going to reach out to them > directly to see if they may have done anything by accident, but presuming > they haven't and the path is spoofed, can I prove that? How can I detect > if traffic is indeed swinging through that hijacked path? How worried > should I be and what are my options for resolving the situation? > > thanks! > -andy > > > >
Re: Validating possible BGP MITM attack
FYI - I did get a response back from BGPMon- they concur with Job: "Hi Andy, unfortunately we had a peer sending us a polluted BGP views. Most likely using a BGP optimizer that is making up new paths. We've reached out to 131477 and dropped the session with them. This was most likely 131477 making up the paths. And not seen wider on the Internet. We'll work on making sure that cases like this will not cause bgpmon alerts going forward, by detecting these false alerts better." -andy On Thu, Aug 31, 2017 at 7:01 AM, Andy Litzinger < andy.litzinger.li...@gmail.com> wrote: > Hello, > we use BGPMon.net to monitor our BGP announcements. This morning we > received two possible BGP MITM alerts for two of our prefixes detected by a > single BGPMon probe located in China. I've reached out to BGPMon to see > how much credence I should give to an alert from a single probe location, > but I'm interested in community feedback as well. > > The alert detailed that one of our /23 prefixes has been broken into /24 > specifics and the AS Path shows a peering relationship with us that does > not exist: > 131477(Shanghai Huajan) 38478(Sunny Vision LTD) 3491(PCCW Global) 14042 > (me) > > We do not peer directly with PCCW Global. I'm going to reach out to them > directly to see if they may have done anything by accident, but presuming > they haven't and the path is spoofed, can I prove that? How can I detect > if traffic is indeed swinging through that hijacked path? How worried > should I be and what are my options for resolving the situation? > > thanks! > -andy >
BGP Optimizers (Was: Validating possible BGP MITM attack)
Dear all, disclaimer: [ The following is targetted at the context where a BGP optimizer generates BGP announcement that are ordinarily not seen in the Default-Free Zone. The OP indicated they announce a /23, and were unpleasantly surprised to see two unauthorized announcements for /24 more-specifics pop up in their alerting system. No permission was granted to create and announce these more-specifics. The AS_PATH for those /24 announcements was entirely fabricated. Original thread https://mailman.nanog.org/pipermail/nanog/2017-August/092124.html ] On Thu, Aug 31, 2017 at 11:13:18AM -0700, Andy Litzinger wrote: > Presuming it was a route optimizer and the issue was ongoing, what > would be the suggested course of action? I strongly recommend to turn off those BGP optimizers, glue the ports shut, burn the hardware, and salt the grounds on which the BGP optimizer sales people walked. It is extremely irresponsible behavior to use software that generates _fake_ BGP more-specifics for the purpose of traffic engineering. You simply cannot expect that those more-specifics will never escape into the global DFZ! Relying on NO_EXPORT is not sufficient: we regularly see software bugs related to NO_EXPORT, and community-squashing configuration mistakes happen all the time. Consider the following: if you leak your own internal more-specifics, at least you are the legitimate destination. (You may suffer from suboptimal routing, but it isn't guaranteed downtime.) However if you generate fake more-specifics for prefixes belonging to OTHER organisations, you essentially are complicit in BGP hijacking. If those fake more-specifics accidentally leak into the DFZ, you are bringing down the actual owner of such prefixes, and depriving people from access to the Internet. Example case: https://mailman.nanog.org/pipermail/nanog/2013-January/054846.html > reach out to those 2 AS owners and see if they could stop it? Yes, absolutely! And if everyone of the affected parties of this localized hijack leak (or should we say 'victims') reaches out to the wrongdoers, they contribute peer pressure to rectify the situation. Just make sure you assign blame to the correct party. :) > Or is it something I just have to live with as a traffic engineering > solution they are using and mark the alerts as false positives? No, this is not something we should just accept. The Default-Free Zone is a shared resource. Anyone using "BGP optimizers" is not only risking their own online presence, but also everyone else's. Using a BGP optimizer is essentially trading a degree of risk to society for the purpose of saving a few bucks or milliseconds. It is basically saying: "The optimizer helps me, but may hurt others, and I am fine with that". An analogy: We don't want transporters of radioactive material to cut corners by using non-existant roads to bring the spent nuclear fuels from A to B faster, nor do we want them to rely on non-robust shielding that works "most of the time". We share the BGP DFZ environment, 'BGP optimizers' are downright pollution contributors. Kind regards, Job ps. If you want to do BGP optimization: don't have the BGP optimizer generate fake BGP announcements, but focus only on manipulating non-transitive attributes (like LOCAL_PREF) on real announcements you actually received from others. Or Perhaps BGP optimizers shouldn't even talk BGP but rather push freshly generated TE-optimized routing-policy to your edge boxes. Or perhaps the 'BGP optimizer' vendors should come to IETF to figure out a safe way (maybe a new AFI/SAFI to manipulate real announcements)... there are ways to do this, without risk to others! p.s. providing a publicly available BGP looking glasses will contribute to proving your innocence in cases like these. Since in many cases the AS_PATH is a complete fabrication, we need to manually check every AS in the AS_PATH to see whether the AS carries the fake more-specific. A public looking glass speeds up this fault-finding process. If you don't want to host a webinterface yourself, please consider sending a BGP feed to the Route Views Project or RIPE RIS, or for something queryable in a real-time fashion the NLNOG RING Looking Glass http://lg.ring.nlnog.net/
Hijacks: AS12506, AS327814, AS44582, AS62135
The following set of interrelated networks appear to be engaged in hijacking various IPv4 address blocks at the present time: AS12506 Inspiring Networks, B.V. (Netherlands) AS44582 Inspiring Networks, B.V. (Netherlands) AS62135 Inspiring Networks, B.V. (Netherlands) AS327814 Echoband, Ltd. (Ghana) The specific routes that are unambiguously being hijacked by each of these networks are as follows: AS12506: 152.108.0.0/16 155.159.0.0/16 196.15.64.0/18 AS327814: 163.198.0.0/18 164.88.0.0/16 168.80.0.0/17 168.80.128.0/17 AS44582: 175.53.0.0/17 175.53.128.0/17 175.54.0.0/17 175.54.128.0/17 AS62135: 160.116.32.0/20 160.116.128.0/20 160.116.240.0/20 160.122.144.0/20 Screenshots of the bgp.he.net prefixes reports for the above networks are archived here: http://i.imgur.com/5HuDRYX.png (AS12506) http://i.imgur.com/YishDCK.png (AS44582) http://i.imgur.com/lgiAKWz.png (AS62135) http://i.imgur.com/IM9Wf5h.png (AS327814) (Note that the set of routes announced by the four networks in question has changed slightly since the last bgp.he.net update -- 30 Aug 2017 14:48 PST. The route for 163.198.0.0/18 has been dropped and the routes for 160.116.128.0/20 and 160.122.144.0/20 have been added.) As seen in previous hijackings, and as is consistant with the general nature of such hijackings, no individual IP addresses within any of the above listed routes have any functioning reverse DNS delegation. Note that AS44582 (Inspiring Networks) and AS62135 (Inspiring Networks) really only have a single upstream connection to the Internet, at least as far as public BGP is concerned, and that is AS12506 (Inspiring Networks). Meanwhile, AS12506 (Inspiring Networks) has only a single BGP upstream, which is AS49544 i3D.net B.V (Netherlands). Therefore, the majority of this hijacking activity is only made possible via the generous help and assistance of AS49544, i3D.net B.V. Inspiring Networks is apparently run by one Maikel Jozef Gerardus Uerlings, : https://labs.ripe.net/Members/maikel_uerlings https://nl.linkedin.com/in/maikel-uerlings-072aaa65 https://twitter.com/maikeluerlings (recently disappeared) https://www.facebook.com/maikel.uerlings http://uerlings.nl/ On February 24, 2013, over four years ago, Mr. Uerlings apparently promised his Facebook friends and fans that his new corporate web site would be "launched soon". As of today however, Mr. Uerlings' corporate web site for Inspiring Networks stil contains only generic/boilerplate "Lorem ipsum" type filler text: https://inspiringnetworks.com/ It would thus appear that Mr. Uerlings has other ways of attracting customers, other than his minimalist placeholder corporate web site. In any case, Mr. Uerlings has apparently gotten some bad press on a couple of occasions, for example the following blog post by some anonymous spammer who felt that Mr. Uerlings didn't actually deliver on his promises of "fresh IPs for mailing": http://maikel-uerlings-inspiring-networks.blogspot.com/ Mr. Uerlings' name also came up in the context of a 2013 attempt by Microsoft to take down a certain botnet: Microsoft v. Botnet United States Court for the Western District of Texas Case: A-13-CV-1014SS http://botnetlegalnotice.com/zeroaccess/files/Summons_Does_1-8.pdf (... Care of: Maikel Uerlings, cust...@serverius.com ...) Other folks also have, or had, a rather unfavorable opinion of Mr. Uerlings also, it would seem: https://www.mywot.com/en/scorecard/uerlings.nl https://www.scamwarners.com/forum/viewtopic.php?p=123180 https://unapprovedpharmacy.com/category/counterfeit-drugs-alert/page/12/ As usual, I wouldn't even mind about any of this hijacking activity if it were not for the fact that at least some porgtion of the hijacked IPv4 space appears to have been populated with snowshoe spammer domains: https://pastebin.com/raw/As9nVCMV I cannot help but wonder if there is something in the water supply in the Netherlands that may be causing so much hijacking activity to originate from that country. I do understand that Netherlands has what I gather is the best connectivity in all of Europe, but even that does not fully explain, I think, the Netherland's disproportionate share of these sorts of events and incidents, in this case involving Inspiring Networks, B.V. and clearly supported by AS49544, i3D.net B.V, also of the Netherlands. Regards, rfg P.s. Don't be fooled by hijackings of IP blocks that were historically allocated by AFRINIC to various corporate entities in the Seychelles Islands. Many of those corporate entities have long since died, and their associated IPv4 blocks have thus been abandoned. Unfortunately, due to the unique and very strict corporate secrecy laws in the Seychelles, it is not possible for any outsider to find out even if these entiries still exist or not, let alone who their corporate officers are or might have been. Thus, literally anybody can come along,
Re: Hijacks: AS12506, AS327814, AS44582, AS62135
Dear Ronald, Thanks for the investigative work, we will look into your report and take the appropriate action. Kind regards, Martijn Schmidt i3D.net / AS49544 PS: if anyone suspects that any of our customers are misbehaving, we would very much appreciate hearing about it directly via ab...@i3d.net as opposed to seeing the information for the first time on a public mailing list. All abuse reports are handled on a case-by-case basis by our own support staff in our headquarters in the Netherlands.
Re: Puerto Rico Internet Exchange
Hi Everyone! Thank you for 10s of great leads regarding PR and Internet EcoSystem. I am wondering if we were to have a Peering Forum in Puerto Rico, when would be the most preferred time? December, February are two options we have for now. Idea would be to create a day where people who are local and involved in network engineering/operations can meet with those in similar functions outside of the island. Thanks once again for amazing support and leads. We look forward to 2018 and the reLaunch. Mehmet On Sat, Aug 12, 2017 at 9:27 PM Mehmet Akcin wrote: > Hey there! > > ... ok this time I am not going to call it PRIX ;) well name doesn't > matter really. Nearly 13 years ago I have attempted to start Puerto rico > Internet exchange in San Juan. I have lived there over 5 years and i just > wanted to really watch videos faster. The project somewhat died when i > moved to LA but now there are few interested party to start an internet > exchange in Puerto rico. The jsland historically had one of the slowest > broadband/internet services which seemed to have improved in recent years > however as of 2017 there still is not an IX in Puerto rico. > > We , 3-4 internet engineers (on island and remote) , want to look into > relaunch of this IX and hopefully find a way to keep local traffic > exchanged at high speeds and low cost. We need expertise, and people who > want to help any way they can. > > We are trying to make this IX a not-for-profit one and we are looking at > opeeating models to adapt which has worked incredibly well like Seattle IX. > > We are hoping the relaunch to happen sometime in 2018. Thanks in advance > hope to share more info and traffic data sometime , soon. Watch this space! > > Mehmet >
Re: BGP Optimizers (Was: Validating possible BGP MITM attack)
I would like to use a BGP optimizer, but I'm too poor. :-\ That said, I'm also an eyeball network, so modifications of my own advertisements are what affects the desired traffic, not so much the outbound routes. I know the BGP optimization industry is weighted towards content networks. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Job Snijders" To: nanog@nanog.org Sent: Thursday, August 31, 2017 3:06:49 PM Subject: BGP Optimizers (Was: Validating possible BGP MITM attack) Dear all, disclaimer: [ The following is targetted at the context where a BGP optimizer generates BGP announcement that are ordinarily not seen in the Default-Free Zone. The OP indicated they announce a /23, and were unpleasantly surprised to see two unauthorized announcements for /24 more-specifics pop up in their alerting system. No permission was granted to create and announce these more-specifics. The AS_PATH for those /24 announcements was entirely fabricated. Original thread https://mailman.nanog.org/pipermail/nanog/2017-August/092124.html ] On Thu, Aug 31, 2017 at 11:13:18AM -0700, Andy Litzinger wrote: > Presuming it was a route optimizer and the issue was ongoing, what > would be the suggested course of action? I strongly recommend to turn off those BGP optimizers, glue the ports shut, burn the hardware, and salt the grounds on which the BGP optimizer sales people walked. It is extremely irresponsible behavior to use software that generates _fake_ BGP more-specifics for the purpose of traffic engineering. You simply cannot expect that those more-specifics will never escape into the global DFZ! Relying on NO_EXPORT is not sufficient: we regularly see software bugs related to NO_EXPORT, and community-squashing configuration mistakes happen all the time. Consider the following: if you leak your own internal more-specifics, at least you are the legitimate destination. (You may suffer from suboptimal routing, but it isn't guaranteed downtime.) However if you generate fake more-specifics for prefixes belonging to OTHER organisations, you essentially are complicit in BGP hijacking. If those fake more-specifics accidentally leak into the DFZ, you are bringing down the actual owner of such prefixes, and depriving people from access to the Internet. Example case: https://mailman.nanog.org/pipermail/nanog/2013-January/054846.html > reach out to those 2 AS owners and see if they could stop it? Yes, absolutely! And if everyone of the affected parties of this localized hijack leak (or should we say 'victims') reaches out to the wrongdoers, they contribute peer pressure to rectify the situation. Just make sure you assign blame to the correct party. :) > Or is it something I just have to live with as a traffic engineering > solution they are using and mark the alerts as false positives? No, this is not something we should just accept. The Default-Free Zone is a shared resource. Anyone using "BGP optimizers" is not only risking their own online presence, but also everyone else's. Using a BGP optimizer is essentially trading a degree of risk to society for the purpose of saving a few bucks or milliseconds. It is basically saying: "The optimizer helps me, but may hurt others, and I am fine with that". An analogy: We don't want transporters of radioactive material to cut corners by using non-existant roads to bring the spent nuclear fuels from A to B faster, nor do we want them to rely on non-robust shielding that works "most of the time". We share the BGP DFZ environment, 'BGP optimizers' are downright pollution contributors. Kind regards, Job ps. If you want to do BGP optimization: don't have the BGP optimizer generate fake BGP announcements, but focus only on manipulating non-transitive attributes (like LOCAL_PREF) on real announcements you actually received from others. Or Perhaps BGP optimizers shouldn't even talk BGP but rather push freshly generated TE-optimized routing-policy to your edge boxes. Or perhaps the 'BGP optimizer' vendors should come to IETF to figure out a safe way (maybe a new AFI/SAFI to manipulate real announcements)... there are ways to do this, without risk to others! p.s. providing a publicly available BGP looking glasses will contribute to proving your innocence in cases like these. Since in many cases the AS_PATH is a complete fabrication, we need to manually check every AS in the AS_PATH to see whether the AS carries the fake more-specific. A public looking glass speeds up this fault-finding process. If you don't want to host a webinterface yourself, please consider sending a BGP feed to the Route Views Project or RIPE RIS, or for something queryable in a real-time fashion the NLNOG RING Looking Glass http://lg.ring.nlnog.net/
Re: BGP Optimizers (Was: Validating possible BGP MITM attack)
Actually, I do remember that one of them would optimize inbound routes, but only billed on outbound usage (as it was content-focused). My in is over 8x my out, so hrm... maybe I'm on to something. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Mike Hammett" Cc: nanog@nanog.org Sent: Thursday, August 31, 2017 8:55:46 PM Subject: Re: BGP Optimizers (Was: Validating possible BGP MITM attack) I would like to use a BGP optimizer, but I'm too poor. :-\ That said, I'm also an eyeball network, so modifications of my own advertisements are what affects the desired traffic, not so much the outbound routes. I know the BGP optimization industry is weighted towards content networks. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Job Snijders" To: nanog@nanog.org Sent: Thursday, August 31, 2017 3:06:49 PM Subject: BGP Optimizers (Was: Validating possible BGP MITM attack) Dear all, disclaimer: [ The following is targetted at the context where a BGP optimizer generates BGP announcement that are ordinarily not seen in the Default-Free Zone. The OP indicated they announce a /23, and were unpleasantly surprised to see two unauthorized announcements for /24 more-specifics pop up in their alerting system. No permission was granted to create and announce these more-specifics. The AS_PATH for those /24 announcements was entirely fabricated. Original thread https://mailman.nanog.org/pipermail/nanog/2017-August/092124.html ] On Thu, Aug 31, 2017 at 11:13:18AM -0700, Andy Litzinger wrote: > Presuming it was a route optimizer and the issue was ongoing, what > would be the suggested course of action? I strongly recommend to turn off those BGP optimizers, glue the ports shut, burn the hardware, and salt the grounds on which the BGP optimizer sales people walked. It is extremely irresponsible behavior to use software that generates _fake_ BGP more-specifics for the purpose of traffic engineering. You simply cannot expect that those more-specifics will never escape into the global DFZ! Relying on NO_EXPORT is not sufficient: we regularly see software bugs related to NO_EXPORT, and community-squashing configuration mistakes happen all the time. Consider the following: if you leak your own internal more-specifics, at least you are the legitimate destination. (You may suffer from suboptimal routing, but it isn't guaranteed downtime.) However if you generate fake more-specifics for prefixes belonging to OTHER organisations, you essentially are complicit in BGP hijacking. If those fake more-specifics accidentally leak into the DFZ, you are bringing down the actual owner of such prefixes, and depriving people from access to the Internet. Example case: https://mailman.nanog.org/pipermail/nanog/2013-January/054846.html > reach out to those 2 AS owners and see if they could stop it? Yes, absolutely! And if everyone of the affected parties of this localized hijack leak (or should we say 'victims') reaches out to the wrongdoers, they contribute peer pressure to rectify the situation. Just make sure you assign blame to the correct party. :) > Or is it something I just have to live with as a traffic engineering > solution they are using and mark the alerts as false positives? No, this is not something we should just accept. The Default-Free Zone is a shared resource. Anyone using "BGP optimizers" is not only risking their own online presence, but also everyone else's. Using a BGP optimizer is essentially trading a degree of risk to society for the purpose of saving a few bucks or milliseconds. It is basically saying: "The optimizer helps me, but may hurt others, and I am fine with that". An analogy: We don't want transporters of radioactive material to cut corners by using non-existant roads to bring the spent nuclear fuels from A to B faster, nor do we want them to rely on non-robust shielding that works "most of the time". We share the BGP DFZ environment, 'BGP optimizers' are downright pollution contributors. Kind regards, Job ps. If you want to do BGP optimization: don't have the BGP optimizer generate fake BGP announcements, but focus only on manipulating non-transitive attributes (like LOCAL_PREF) on real announcements you actually received from others. Or Perhaps BGP optimizers shouldn't even talk BGP but rather push freshly generated TE-optimized routing-policy to your edge boxes. Or perhaps the 'BGP optimizer' vendors should come to IETF to figure out a safe way (maybe a new AFI/SAFI to manipulate real announcements)... there are ways to do this, without risk to others! p.s. providing a publicly available BGP looking glasses will contribute to proving your innocence in cases like these. Since in many cases the AS_PATH is a
Re: BGP Optimizers (Was: Validating possible BGP MITM attack)
Sorry for now taking up 1/4 of this thread My words in the last message don't match what I was thinking, but I think you all get the point. I'm sick, maybe I should be in bed instead of on NANOG. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Mike Hammett" Cc: nanog@nanog.org Sent: Thursday, August 31, 2017 9:02:07 PM Subject: Re: BGP Optimizers (Was: Validating possible BGP MITM attack) Actually, I do remember that one of them would optimize inbound routes, but only billed on outbound usage (as it was content-focused). My in is over 8x my out, so hrm... maybe I'm on to something. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Mike Hammett" Cc: nanog@nanog.org Sent: Thursday, August 31, 2017 8:55:46 PM Subject: Re: BGP Optimizers (Was: Validating possible BGP MITM attack) I would like to use a BGP optimizer, but I'm too poor. :-\ That said, I'm also an eyeball network, so modifications of my own advertisements are what affects the desired traffic, not so much the outbound routes. I know the BGP optimization industry is weighted towards content networks. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Job Snijders" To: nanog@nanog.org Sent: Thursday, August 31, 2017 3:06:49 PM Subject: BGP Optimizers (Was: Validating possible BGP MITM attack) Dear all, disclaimer: [ The following is targetted at the context where a BGP optimizer generates BGP announcement that are ordinarily not seen in the Default-Free Zone. The OP indicated they announce a /23, and were unpleasantly surprised to see two unauthorized announcements for /24 more-specifics pop up in their alerting system. No permission was granted to create and announce these more-specifics. The AS_PATH for those /24 announcements was entirely fabricated. Original thread https://mailman.nanog.org/pipermail/nanog/2017-August/092124.html ] On Thu, Aug 31, 2017 at 11:13:18AM -0700, Andy Litzinger wrote: > Presuming it was a route optimizer and the issue was ongoing, what > would be the suggested course of action? I strongly recommend to turn off those BGP optimizers, glue the ports shut, burn the hardware, and salt the grounds on which the BGP optimizer sales people walked. It is extremely irresponsible behavior to use software that generates _fake_ BGP more-specifics for the purpose of traffic engineering. You simply cannot expect that those more-specifics will never escape into the global DFZ! Relying on NO_EXPORT is not sufficient: we regularly see software bugs related to NO_EXPORT, and community-squashing configuration mistakes happen all the time. Consider the following: if you leak your own internal more-specifics, at least you are the legitimate destination. (You may suffer from suboptimal routing, but it isn't guaranteed downtime.) However if you generate fake more-specifics for prefixes belonging to OTHER organisations, you essentially are complicit in BGP hijacking. If those fake more-specifics accidentally leak into the DFZ, you are bringing down the actual owner of such prefixes, and depriving people from access to the Internet. Example case: https://mailman.nanog.org/pipermail/nanog/2013-January/054846.html > reach out to those 2 AS owners and see if they could stop it? Yes, absolutely! And if everyone of the affected parties of this localized hijack leak (or should we say 'victims') reaches out to the wrongdoers, they contribute peer pressure to rectify the situation. Just make sure you assign blame to the correct party. :) > Or is it something I just have to live with as a traffic engineering > solution they are using and mark the alerts as false positives? No, this is not something we should just accept. The Default-Free Zone is a shared resource. Anyone using "BGP optimizers" is not only risking their own online presence, but also everyone else's. Using a BGP optimizer is essentially trading a degree of risk to society for the purpose of saving a few bucks or milliseconds. It is basically saying: "The optimizer helps me, but may hurt others, and I am fine with that". An analogy: We don't want transporters of radioactive material to cut corners by using non-existant roads to bring the spent nuclear fuels from A to B faster, nor do we want them to rely on non-robust shielding that works "most of the time". We share the BGP DFZ environment, 'BGP optimizers' are downright pollution contributors. Kind regards, Job ps. If you want to do BGP optimization: don't have the BGP optimizer generate fake BGP announcements, but focus only on manipulating non-transitive attributes (like LOCAL_PREF) on real announcements you a
Re: BGP Optimizers (Was: Validating possible BGP MITM attack)
Nanog's quite a nice list to spend your sick day on. ;) On 31/08/2017 19:04, Mike Hammett wrote: Sorry for now taking up 1/4 of this thread My words in the last message don't match what I was thinking, but I think you all get the point. I'm sick, maybe I should be in bed instead of on NANOG. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Mike Hammett" Cc: nanog@nanog.org Sent: Thursday, August 31, 2017 9:02:07 PM Subject: Re: BGP Optimizers (Was: Validating possible BGP MITM attack) Actually, I do remember that one of them would optimize inbound routes, but only billed on outbound usage (as it was content-focused). My in is over 8x my out, so hrm... maybe I'm on to something. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Mike Hammett" Cc: nanog@nanog.org Sent: Thursday, August 31, 2017 8:55:46 PM Subject: Re: BGP Optimizers (Was: Validating possible BGP MITM attack) I would like to use a BGP optimizer, but I'm too poor. :-\ That said, I'm also an eyeball network, so modifications of my own advertisements are what affects the desired traffic, not so much the outbound routes. I know the BGP optimization industry is weighted towards content networks. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Job Snijders" To: nanog@nanog.org Sent: Thursday, August 31, 2017 3:06:49 PM Subject: BGP Optimizers (Was: Validating possible BGP MITM attack) Dear all, disclaimer: [ The following is targetted at the context where a BGP optimizer generates BGP announcement that are ordinarily not seen in the Default-Free Zone. The OP indicated they announce a /23, and were unpleasantly surprised to see two unauthorized announcements for /24 more-specifics pop up in their alerting system. No permission was granted to create and announce these more-specifics. The AS_PATH for those /24 announcements was entirely fabricated. Original thread https://mailman.nanog.org/pipermail/nanog/2017-August/092124.html ] On Thu, Aug 31, 2017 at 11:13:18AM -0700, Andy Litzinger wrote: Presuming it was a route optimizer and the issue was ongoing, what would be the suggested course of action? I strongly recommend to turn off those BGP optimizers, glue the ports shut, burn the hardware, and salt the grounds on which the BGP optimizer sales people walked. It is extremely irresponsible behavior to use software that generates _fake_ BGP more-specifics for the purpose of traffic engineering. You simply cannot expect that those more-specifics will never escape into the global DFZ! Relying on NO_EXPORT is not sufficient: we regularly see software bugs related to NO_EXPORT, and community-squashing configuration mistakes happen all the time. Consider the following: if you leak your own internal more-specifics, at least you are the legitimate destination. (You may suffer from suboptimal routing, but it isn't guaranteed downtime.) However if you generate fake more-specifics for prefixes belonging to OTHER organisations, you essentially are complicit in BGP hijacking. If those fake more-specifics accidentally leak into the DFZ, you are bringing down the actual owner of such prefixes, and depriving people from access to the Internet. Example case: https://mailman.nanog.org/pipermail/nanog/2013-January/054846.html reach out to those 2 AS owners and see if they could stop it? Yes, absolutely! And if everyone of the affected parties of this localized hijack leak (or should we say 'victims') reaches out to the wrongdoers, they contribute peer pressure to rectify the situation. Just make sure you assign blame to the correct party. :) Or is it something I just have to live with as a traffic engineering solution they are using and mark the alerts as false positives? No, this is not something we should just accept. The Default-Free Zone is a shared resource. Anyone using "BGP optimizers" is not only risking their own online presence, but also everyone else's. Using a BGP optimizer is essentially trading a degree of risk to society for the purpose of saving a few bucks or milliseconds. It is basically saying: "The optimizer helps me, but may hurt others, and I am fine with that". An analogy: We don't want transporters of radioactive material to cut corners by using non-existant roads to bring the spent nuclear fuels from A to B faster, nor do we want them to rely on non-robust shielding that works "most of the time". We share the BGP DFZ environment, 'BGP optimizers' are downright pollution contributors. Kind regards, Job ps. If you want to do BGP optimization: don't have the BGP optimizer generate fake BGP announcements, but focus only on manipulating non-transitive attributes (like LOCAL_PREF) on real announcements you
Management softwares
Hi We are a medium ISP. We are looking for Management software solutions. We are interested in finding a solution for billing, invoicing, scheduling, ticketing, provisioning and monitoring, Any suggestions or recommendations will be appreciated? We have been using multiple systems which do not communicate. Our objective is to use a single system or at least reduce the number of systems. Thank you KARIM M.
