Re: DOs and DONTs for small ISP

[Baldur Norddahl]

For some reason our network looks nothing like that. He either needs to
define what a PoP is or define a module that does not have expensive gear
like two redundant routers at every location.

Regards

Baldur

tir. 4. jun. 2019 15.46 skrev Mehmet Akcin :

> This Gem is fantastic by the way,
>
>
> https://nsrc.org/workshops/2015/apricot2015/raw-attachment/wiki/Track1Agenda/01-ISP-Network-Design.pdf
>
> On Tue, Jun 4, 2019 at 5:57 AM Warren Kumari  wrote:
>
>> On Mon, Jun 3, 2019 at 11:34 PM Brandon Martin 
>> wrote:
>> >
>> > On 6/3/19 9:56 AM, Jon Lewis wrote:
>> > > 3) Don't advertise one transit provider's routes to another.  Each
>> should
>> > > be filtering your routes, but you never know.  Come up with, and
>> use
>> > > BGP communities to control route propagation.  As you grow, it
>> sucks
>> > > having to update prefix-list filters in multiple places every time
>> > > something changes...like a new customer with their own IPs.
>> >
>> > To reiterate all this, FILTER EVERYTHING.
>> >
>> > To start with, explicitly specify in a route-map or similar everything
>> > you want to advertise.  I usually create a separate route-map for each
>> > transit/peer and include what I want to advertise via prefix lists (for
>> > my IP space) and/or communities (for downstream BGP-speaking customers
>> > if anticipated).
>>
>> I think a related *principle* is: "Build everything as though you are
>> expecting to scale."
>>
>> This doesn't mean "spend lots of money to buy huge
>> [routers|servers|commercial software|], but rather "when you plan
>> your addressing structure and routing policies and monitoring and
>> device config generation and... keep the in mind the question "If this
>> suddenly takes off, and I hire N more people to run this, can I
>> explain to them how it works? Do I have documentation I can point them
>> at or is it stuck in my head / on the devices? If I need to add
>> another M customers in the next month, can I do that easily?".
>>
>> This is related to the FILTER EVERYTHING -- when you turn up a new
>> customer / peer / transit / whatever, you shouldn't be sitting around
>> trying to figure out how you will write their route-map /
>> policy-options -- this leads to weird one-offs, and quick hacks.
>> Instead you should have policies already largely designed and simply
>> plug in their prefixes (or, better yet, use bgpq3 or similar to build
>> and populate these). Obviously there will be some cases where a new
>> connection does require some special handling, but that *should* just
>> be a plugin/chain in an existing policy-statement. Related to this is
>> how you end up naming things -- I recently found 9 variants of
>> firewall-filters which basically do:
>>
>> filter ACCEPT {
>>term ACCEPT {
>> then accept;
>>   }
>> }
>> named things like: ACCEPT, ACEPT, Accept, Allow, Permit_all,
>> AcceptAll, dontdrop [0].
>>
>> Obviously, there is a tension in the "design for scale" - while it
>> would be great to design a complete automation system so that
>> everything from installing a new customer to a new sites is simply
>> typing 'make ' and having everything pull from a database, at
>> some point you will need to actually build a network, or you'll never
>> have customers :-) Just keep in mind that "Am I building myself into a
>> corner here?". E.g it only takes 10 or 15 minutes to install something
>> like NetBox to keep track of addresses (and prefixes and racks and
>> connections and ...) -- stuffing this in a spreadsheet might save you
>> a few minutes *now*, but will this scale? Can $new_person easily
>> figure it out?
>>
>>
>> W
>> [0]: My personal favorite is:
>> filter Accept_All {
>> term Accept {
>> then {
>> count dropped;
>> reject;
>> }
>> }
>> term filter_ {
>> from {
>> prefix-list {
>> ;
>>}
>> }
>> then accept;
>> }
>> term NEXT {
>> then log;
>> }
>> }
>>
>> Presumably this all made sense to 
>> when they stuck it in at 3AM to deal with some crazy issue, but...
>>
>>
>>
>> >
>> > When you turn on the session, check what you're squawking AND WHAT
>> > YOU'RE FILTERING.  You shouldn't be filtering anything you don't expect.
>> >   Belt + suspenders.
>> >
>> > The same goes for anything you accept.  Obviously for a blended full
>> > transit BGP edge router, you're probably going to accept almost
>> > everything.  But if you only want default + on-net, try to filter using
>> > communities from the peer, etc.  Again, right when you turn on the
>> > session, "sh ip bgp ... filtered" of whatever's equivalent on your
>> > platform.  If you're filtering something you don't expect to be
>> > receiving at all, figure out where the misunderstanding or
>> > misconfiguration lies.
>> >
>> > And of course it goes without saying that, if you've got BGP speaking
>> > customers, you filter the heck out of them.  Use ROAs and/or RPKI if 

Re: DOs and DONTs for small ISP

[William Herrin]

On Wed, Jun 5, 2019 at 5:44 AM William Waites  wrote:
> It's not enough to have monitoring and a ticket system. You need to pay
> attention to them, care for them and feed them. I can't count the number
> of ticket systems full of ancient and irrelevant things or monitoring
> systems that people have forgotten about or don't know how to add new
> stuff to. Even the cycle of,

Some points to consider when monitoring your network:

1. Beware early automation. If you write a generator to go and monitor all
your stuff without addressing how operators will change things one-off
(which is hard to design well) the other operators will find the monitoring
system unusable. Which means they won't update it when stuff is added and
changed. Making it quickly useless.

2. Careful aggregating alarms. That big green or red light is useless. The
operator has to be able to start with the alarm and immediately trace back
to exactly what tests and results bubbled up in to the aggregate and from
there to the malfunctioning component. If you lose this information during
the aggregation process, you're just producing noise.

2. Every alarm must be actionable. When the light goes red, what -exactly-
do you want the operator to do as a result? Don't create an alarm until you
can offer a detailed and specific answer, and link that answer to the alarm
so the operator doesn't have to hunt for it.

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: DOs and DONTs for small ISP

[William Waites]

On 06/03, Mel Beckman wrote:
> I’m constantly amazed at the number of even medium-sized ISPs that have no
> network monitoring. An NMS should go in as the first software component —
> before billing starts and the provider is on the hook to deliver. 
> 
> The second lacking component is a ticket system, which is silly because
> turnkey cloud services are not expensive, and open source solutions abound
> for budget-limited operators.

It's not enough to have monitoring and a ticket system. You need to pay
attention to them, care for them and feed them. I can't count the number
of ticket systems full of ancient and irrelevant things or monitoring 
systems that people have forgotten about or don't know how to add new
stuff to. Even the cycle of,

  10 we need network monitoring
  20 stand up some monitoring system ...
  30 ... time passes, the person leaves etc ...
  40 ... the network monitoring system is forgotten ...
  50 GOTO 10

Cheers,
-w

Re: DOs and DONTs for small ISP

[Randy Bush]

> This Gem is fantastic by the way,
> https://nsrc.org/workshops/2015/apricot2015/raw-attachment/wiki/Track1Agenda/01-ISP-Network-Design.pdf

philip smith

Re: DOs and DONTs for small ISP

[jim deleskie]

triggered :)


On Tue, Jun 4, 2019 at 11:31 AM Bryan Holloway  wrote:

>
> On 6/4/19 9:20 AM, Mark Tinka wrote:
> >
> >
> > On 3/Jun/19 15:41, Fletcher Kittredge wrote:
> >>
> >> Here is your checklist in descending order of importance:
> >>
> >>  1. market opportunity
> >>  2. finding the right partners (see below)
> >>  3. financial
> >>  4. sales and marketing
> >>  5. organizational capacity and HR
> >>  6. legal, regulatory
> >>  7. capital acquisition
> >>  8. security
> >>  9. ...
> >> 10. ...
> >> 11. ...
> >> 12. technical including equipment selection, routing policy,
> >> filtering, etc
> >>
> >
> > 13. Don't run Mikrotik.
> >
> > I'm kidding... I think :-)...
> >
> > Mark.
>
> 14. Go with K56flex, not X2.
>

Re: DOs and DONTs for small ISP

[Bryan Holloway]

On 6/4/19 9:20 AM, Mark Tinka wrote:



On 3/Jun/19 15:41, Fletcher Kittredge wrote:


Here is your checklist in descending order of importance:

 1. market opportunity
 2. finding the right partners (see below)
 3. financial
 4. sales and marketing
 5. organizational capacity and HR
 6. legal, regulatory
 7. capital acquisition
 8. security
 9. ...
10. ...
11. ...
12. technical including equipment selection, routing policy,
filtering, etc



13. Don't run Mikrotik.

I'm kidding... I think :-)...

Mark.


14. Go with K56flex, not X2.

Re: DOs and DONTs for small ISP

[Mark Tinka]

On 3/Jun/19 15:41, Fletcher Kittredge wrote:
>
> Here is your checklist in descending order of importance:
>
>  1. market opportunity
>  2. finding the right partners (see below)
>  3. financial
>  4. sales and marketing
>  5. organizational capacity and HR
>  6. legal, regulatory
>  7. capital acquisition
>  8. security
>  9. ...
> 10. ...
> 11. ...
> 12. technical including equipment selection, routing policy,
> filtering, etc
>

13. Don't run Mikrotik.

I'm kidding... I think :-)...

Mark.

Re: DOs and DONTs for small ISP

[Mehmet Akcin]

This Gem is fantastic by the way,

https://nsrc.org/workshops/2015/apricot2015/raw-attachment/wiki/Track1Agenda/01-ISP-Network-Design.pdf

On Tue, Jun 4, 2019 at 5:57 AM Warren Kumari  wrote:

> On Mon, Jun 3, 2019 at 11:34 PM Brandon Martin 
> wrote:
> >
> > On 6/3/19 9:56 AM, Jon Lewis wrote:
> > > 3) Don't advertise one transit provider's routes to another.  Each
> should
> > > be filtering your routes, but you never know.  Come up with, and
> use
> > > BGP communities to control route propagation.  As you grow, it
> sucks
> > > having to update prefix-list filters in multiple places every time
> > > something changes...like a new customer with their own IPs.
> >
> > To reiterate all this, FILTER EVERYTHING.
> >
> > To start with, explicitly specify in a route-map or similar everything
> > you want to advertise.  I usually create a separate route-map for each
> > transit/peer and include what I want to advertise via prefix lists (for
> > my IP space) and/or communities (for downstream BGP-speaking customers
> > if anticipated).
>
> I think a related *principle* is: "Build everything as though you are
> expecting to scale."
>
> This doesn't mean "spend lots of money to buy huge
> [routers|servers|commercial software|], but rather "when you plan
> your addressing structure and routing policies and monitoring and
> device config generation and... keep the in mind the question "If this
> suddenly takes off, and I hire N more people to run this, can I
> explain to them how it works? Do I have documentation I can point them
> at or is it stuck in my head / on the devices? If I need to add
> another M customers in the next month, can I do that easily?".
>
> This is related to the FILTER EVERYTHING -- when you turn up a new
> customer / peer / transit / whatever, you shouldn't be sitting around
> trying to figure out how you will write their route-map /
> policy-options -- this leads to weird one-offs, and quick hacks.
> Instead you should have policies already largely designed and simply
> plug in their prefixes (or, better yet, use bgpq3 or similar to build
> and populate these). Obviously there will be some cases where a new
> connection does require some special handling, but that *should* just
> be a plugin/chain in an existing policy-statement. Related to this is
> how you end up naming things -- I recently found 9 variants of
> firewall-filters which basically do:
>
> filter ACCEPT {
>term ACCEPT {
> then accept;
>   }
> }
> named things like: ACCEPT, ACEPT, Accept, Allow, Permit_all,
> AcceptAll, dontdrop [0].
>
> Obviously, there is a tension in the "design for scale" - while it
> would be great to design a complete automation system so that
> everything from installing a new customer to a new sites is simply
> typing 'make ' and having everything pull from a database, at
> some point you will need to actually build a network, or you'll never
> have customers :-) Just keep in mind that "Am I building myself into a
> corner here?". E.g it only takes 10 or 15 minutes to install something
> like NetBox to keep track of addresses (and prefixes and racks and
> connections and ...) -- stuffing this in a spreadsheet might save you
> a few minutes *now*, but will this scale? Can $new_person easily
> figure it out?
>
>
> W
> [0]: My personal favorite is:
> filter Accept_All {
> term Accept {
> then {
> count dropped;
> reject;
> }
> }
> term filter_ {
> from {
> prefix-list {
> ;
>}
> }
> then accept;
> }
> term NEXT {
> then log;
> }
> }
>
> Presumably this all made sense to 
> when they stuck it in at 3AM to deal with some crazy issue, but...
>
>
>
> >
> > When you turn on the session, check what you're squawking AND WHAT
> > YOU'RE FILTERING.  You shouldn't be filtering anything you don't expect.
> >   Belt + suspenders.
> >
> > The same goes for anything you accept.  Obviously for a blended full
> > transit BGP edge router, you're probably going to accept almost
> > everything.  But if you only want default + on-net, try to filter using
> > communities from the peer, etc.  Again, right when you turn on the
> > session, "sh ip bgp ... filtered" of whatever's equivalent on your
> > platform.  If you're filtering something you don't expect to be
> > receiving at all, figure out where the misunderstanding or
> > misconfiguration lies.
> >
> > And of course it goes without saying that, if you've got BGP speaking
> > customers, you filter the heck out of them.  Use ROAs and/or RPKI if you
> > can to automatically generate filter lists.  Encourage your upstreams to
> > do the same if they're filtering you (and they probably are, or at least
> > should be, if you're new).  Remember that you are responsible for every
> > route you advertise, at the end of the day, even if you only advertised
> > it because a downstream network made a boo-boo and you didn't 

Re: DOs and DONTs for small ISP

[Warren Kumari]

On Mon, Jun 3, 2019 at 11:34 PM Brandon Martin  wrote:
>
> On 6/3/19 9:56 AM, Jon Lewis wrote:
> > 3) Don't advertise one transit provider's routes to another.  Each should
> > be filtering your routes, but you never know.  Come up with, and use
> > BGP communities to control route propagation.  As you grow, it sucks
> > having to update prefix-list filters in multiple places every time
> > something changes...like a new customer with their own IPs.
>
> To reiterate all this, FILTER EVERYTHING.
>
> To start with, explicitly specify in a route-map or similar everything
> you want to advertise.  I usually create a separate route-map for each
> transit/peer and include what I want to advertise via prefix lists (for
> my IP space) and/or communities (for downstream BGP-speaking customers
> if anticipated).

I think a related *principle* is: "Build everything as though you are
expecting to scale."

This doesn't mean "spend lots of money to buy huge
[routers|servers|commercial software|], but rather "when you plan
your addressing structure and routing policies and monitoring and
device config generation and... keep the in mind the question "If this
suddenly takes off, and I hire N more people to run this, can I
explain to them how it works? Do I have documentation I can point them
at or is it stuck in my head / on the devices? If I need to add
another M customers in the next month, can I do that easily?".

This is related to the FILTER EVERYTHING -- when you turn up a new
customer / peer / transit / whatever, you shouldn't be sitting around
trying to figure out how you will write their route-map /
policy-options -- this leads to weird one-offs, and quick hacks.
Instead you should have policies already largely designed and simply
plug in their prefixes (or, better yet, use bgpq3 or similar to build
and populate these). Obviously there will be some cases where a new
connection does require some special handling, but that *should* just
be a plugin/chain in an existing policy-statement. Related to this is
how you end up naming things -- I recently found 9 variants of
firewall-filters which basically do:

filter ACCEPT {
   term ACCEPT {
then accept;
  }
}
named things like: ACCEPT, ACEPT, Accept, Allow, Permit_all,
AcceptAll, dontdrop [0].

Obviously, there is a tension in the "design for scale" - while it
would be great to design a complete automation system so that
everything from installing a new customer to a new sites is simply
typing 'make ' and having everything pull from a database, at
some point you will need to actually build a network, or you'll never
have customers :-) Just keep in mind that "Am I building myself into a
corner here?". E.g it only takes 10 or 15 minutes to install something
like NetBox to keep track of addresses (and prefixes and racks and
connections and ...) -- stuffing this in a spreadsheet might save you
a few minutes *now*, but will this scale? Can $new_person easily
figure it out?


W
[0]: My personal favorite is:
filter Accept_All {
term Accept {
then {
count dropped;
reject;
}
}
term filter_ {
from {
prefix-list {
;
   }
}
then accept;
}
term NEXT {
then log;
}
}

Presumably this all made sense to 
when they stuck it in at 3AM to deal with some crazy issue, but...



>
> When you turn on the session, check what you're squawking AND WHAT
> YOU'RE FILTERING.  You shouldn't be filtering anything you don't expect.
>   Belt + suspenders.
>
> The same goes for anything you accept.  Obviously for a blended full
> transit BGP edge router, you're probably going to accept almost
> everything.  But if you only want default + on-net, try to filter using
> communities from the peer, etc.  Again, right when you turn on the
> session, "sh ip bgp ... filtered" of whatever's equivalent on your
> platform.  If you're filtering something you don't expect to be
> receiving at all, figure out where the misunderstanding or
> misconfiguration lies.
>
> And of course it goes without saying that, if you've got BGP speaking
> customers, you filter the heck out of them.  Use ROAs and/or RPKI if you
> can to automatically generate filter lists.  Encourage your upstreams to
> do the same if they're filtering you (and they probably are, or at least
> should be, if you're new).  Remember that you are responsible for every
> route you advertise, at the end of the day, even if you only advertised
> it because a downstream network made a boo-boo and you didn't filter it.
>
> Filters are useful on your IGP, too, but there's so many ways to set all
> that up that it's a bit more difficult to come up with nearly universal
> best practices.  Generally speaking, be careful with redistribution,
> never distribute BGP into IGP or vice versa unless you have a really,
> really good reason to, and consider filters between both IGP
> areas/regions or protocols (e.g. RIP 

Re: DOs and DONTs for small ISP

[Jared Mauch]

> On Jun 3, 2019, at 3:12 PM, Warren Kumari  wrote:
> 
> On Mon, Jun 3, 2019 at 2:55 PM Fletcher Kittredge  wrote:
>> 
>> 
>> I would respectfully point out that my point about the importance of finding 
>> the right partners. For you, sounds like it was good to have opportunity to 
>> get out of this venture.
> 
> 
> Oh, goodness yes -- however, I *still* have barely working Internet
> access at that location (it's basically a weekend home) -- I've
> somewhat started down the Jared Mauch "buy used vibratory plow, trench
> (house is off a dirt road, neighbors won't have an issue with right of
> way if I provide them free bits!), install fiber" path. My driveway is
> @1/4 mile, the dirt road is 0.6 miles and at the end of the road there
> is (what used to be) Quest fiber. 

Few tips:

1) Some incumbents don’t know their fiber, so sales reps don’t know how
to quote it but a reseller can.  This might be a good route.  Then again
I am having trouble with a firm right now so will be poking around at
NANOG to solve my problem.

2) Don’t tell all my secrets yet.  I also now own a directional drill
and have a tariff on file with the state.  (It wasn’t expensive once you
find the right firm).

3) Fusion splicers and OTDRs are super cheap these days if you’re doing
simple work.

4) Labor is always the expensive part.

5) More to come, maybe by the October NANOG I’ll be ready with my talk.

- Jared

Re: DOs and DONTs for small ISP

[Brandon Martin]

On 6/3/19 9:56 AM, Jon Lewis wrote:

3) Don't advertise one transit provider's routes to another.  Each should
    be filtering your routes, but you never know.  Come up with, and use
    BGP communities to control route propagation.  As you grow, it sucks
    having to update prefix-list filters in multiple places every time
    something changes...like a new customer with their own IPs.


To reiterate all this, FILTER EVERYTHING.

To start with, explicitly specify in a route-map or similar everything 
you want to advertise.  I usually create a separate route-map for each 
transit/peer and include what I want to advertise via prefix lists (for 
my IP space) and/or communities (for downstream BGP-speaking customers 
if anticipated).


When you turn on the session, check what you're squawking AND WHAT 
YOU'RE FILTERING.  You shouldn't be filtering anything you don't expect. 
 Belt + suspenders.


The same goes for anything you accept.  Obviously for a blended full 
transit BGP edge router, you're probably going to accept almost 
everything.  But if you only want default + on-net, try to filter using 
communities from the peer, etc.  Again, right when you turn on the 
session, "sh ip bgp ... filtered" of whatever's equivalent on your 
platform.  If you're filtering something you don't expect to be 
receiving at all, figure out where the misunderstanding or 
misconfiguration lies.


And of course it goes without saying that, if you've got BGP speaking 
customers, you filter the heck out of them.  Use ROAs and/or RPKI if you 
can to automatically generate filter lists.  Encourage your upstreams to 
do the same if they're filtering you (and they probably are, or at least 
should be, if you're new).  Remember that you are responsible for every 
route you advertise, at the end of the day, even if you only advertised 
it because a downstream network made a boo-boo and you didn't filter it.


Filters are useful on your IGP, too, but there's so many ways to set all 
that up that it's a bit more difficult to come up with nearly universal 
best practices.  Generally speaking, be careful with redistribution, 
never distribute BGP into IGP or vice versa unless you have a really, 
really good reason to, and consider filters between both IGP 
areas/regions or protocols (e.g. RIP coming into OSPF) as well as on 
redistributions of static/connected to prevent simple typos on a static 
route or interface configuration from taking down more than just local 
stuff.


It's way, way easier to remove or relax filters later if they prove more 
of an operational hazard than asset than it is to add or tighten them if 
they prove insufficient.

--
Brandon Martin

Re: DOs and DONTs for small ISP

[Warren Kumari]

On Mon, Jun 3, 2019 at 2:55 PM Fletcher Kittredge  wrote:
>
>
> I would respectfully point out that my point about the importance of finding 
> the right partners. For you, sounds like it was good to have opportunity to 
> get out of this venture.


Oh, goodness yes -- however, I *still* have barely working Internet
access at that location (it's basically a weekend home) -- I've
somewhat started down the Jared Mauch "buy used vibratory plow, trench
(house is off a dirt road, neighbors won't have an issue with right of
way if I provide them free bits!), install fiber" path. My driveway is
@1/4 mile, the dirt road is 0.6 miles and at the end of the road there
is (what used to be) Quest fiber. Unfortunately the fiber was
originally run for Mount Weather
(https://en.wikipedia.org/wiki/Mount_Weather_Emergency_Operations_Center)
and no-one I've asked seems to know who controls it now, and or swears
that it doesn't exist (although the local Miss Utility / VA811 will
come mark it if you call).

Some dark nights, when trying to use the network while everyone is
using NetFlix, and the RTT for my SSH sessions starts exceeding
1,500ms I strongly consider walking to the end of the road with a
nice, sharp shovel, and then talking to the splicers when they come
repair the damage :-)

W

>
> On Mon, Jun 3, 2019 at 2:40 PM Warren Kumari  wrote:
>>
>> On Mon, Jun 3, 2019 at 1:09 PM Fletcher Kittredge  wrote:
>> >
>> >
>> > Here is your checklist in descending order of importance:
>> >
>> > market opportunity
>> > finding the right partners (see below)
>> > financial
>> > sales and marketing
>> > organizational capacity and HR
>> > legal, regulatory
>> > capital acquisition
>> > security
>> > ...
>> > ...
>> > ...
>> > technical including equipment selection, routing policy, filtering, etc
>> >
>> > It is a stone cold lock that the success of your new ISP will governed by 
>> > factors other than technical. Your most important task is to find 
>> > competent  financial and marketing people you can respect and trust. If 
>> > the market opportunity exists and you find them, you will succeed. If you 
>> > don't, all the technical excellence in the world won't help you. The road 
>> > is littered with technically excellent companies that failed.
>>
>> Indeed, but you *also* need to have some technical clue. Two or three
>> years ago a friend and I tried to start a local wireless ISP -- I was
>> doing this purely as a "My home Internet access sucks, and I'll
>> happily donate time, equipment, IP space and some startup capital to
>> fix this" play -- unfortunately it turns out that he and I had very
>> different ideas on, well, basically everything. I wanted an actual
>> architecture / design, and diagrams and routin' and such. He was much
>> more of "We don't need a list of IPs, if I ping it and can't reach it
>> it must be free" / "routing is too hard, let's just put it all in a
>> switch and... um... NAT!". I wanted a plan, and was willing to put in
>> the time and effort to build Ansible / Puppet / an NMS / AAA, etc, he
>> was more seat-of-the-pants.
>>
>> But yes, even if we had good technology this would have failed - there
>> was no real business plan (other than "The current provider is really
>> bad, if we build something else, people will be breaking down the door
>> to sign up"), no real marketing plan (see previous), etc.
>>
>> He was also a bit of a gun nut, and so would arrive at customers with
>> a (holstered) firearm belted on -- even in Virginia this was not a
>> winning business move.
>>
>> Starting a successful ISP is this day and age is hard - make sure
>> that, if you do it, you and whoever you are doing this with are
>> compatible, are both committed, and have similar views on things...
>>
>> W
>>
>>
>> >
>> >
>> >
>> > On Mon, Jun 3, 2019 at 8:05 AM Mehmet Akcin  wrote:
>> >>
>> >> hi there,
>> >>
>> >> I know there are folks from lots of small ISPs here and I wanted to 
>> >> check-in on asking few advice points as I am involved building an ISP 
>> >> from green-field.
>> >>
>> >> Usually, it's pretty straight forward to cover high-level important 
>> >> things, filters, routing policies, etc.but we all know the devil is in 
>> >> the details.
>> >>
>> >> I am putting together a public DOs and DONTs blog post and would love to 
>> >> hear from those who have built ISPs and have recommendations from Billing 
>> >> to Interconnection, Routing policy to Out of the band  & console setup, 
>> >> Software recommendations, etc. Bottom line is that I would like to 
>> >> publish a checklist with these recommendations which I hope will be 
>> >> useful for all.
>> >>
>> >> thanks in advance for your help and recommendation.
>> >>
>> >> Mehmet
>> >>
>> >>
>> >
>> >
>> > --
>> > Fletcher Kittredge
>> > GWI
>> > 207-602-1134
>> > www.gwi.net
>>
>>
>>
>> --
>> I don't think the execution is relevant when it was obviously a bad
>> idea in the first place.
>> This is like putting rabid weasels in your pants, and later 

Re: DOs and DONTs for small ISP

[Fletcher Kittredge]

I would respectfully point out that my point about the importance of
finding the right partners. For you, sounds like it was good to have
opportunity to get out of this venture.

On Mon, Jun 3, 2019 at 2:40 PM Warren Kumari  wrote:

> On Mon, Jun 3, 2019 at 1:09 PM Fletcher Kittredge 
> wrote:
> >
> >
> > Here is your checklist in descending order of importance:
> >
> > market opportunity
> > finding the right partners (see below)
> > financial
> > sales and marketing
> > organizational capacity and HR
> > legal, regulatory
> > capital acquisition
> > security
> > ...
> > ...
> > ...
> > technical including equipment selection, routing policy, filtering, etc
> >
> > It is a stone cold lock that the success of your new ISP will governed
> by factors other than technical. Your most important task is to find
> competent  financial and marketing people you can respect and trust. If the
> market opportunity exists and you find them, you will succeed. If you
> don't, all the technical excellence in the world won't help you. The road
> is littered with technically excellent companies that failed.
>
> Indeed, but you *also* need to have some technical clue. Two or three
> years ago a friend and I tried to start a local wireless ISP -- I was
> doing this purely as a "My home Internet access sucks, and I'll
> happily donate time, equipment, IP space and some startup capital to
> fix this" play -- unfortunately it turns out that he and I had very
> different ideas on, well, basically everything. I wanted an actual
> architecture / design, and diagrams and routin' and such. He was much
> more of "We don't need a list of IPs, if I ping it and can't reach it
> it must be free" / "routing is too hard, let's just put it all in a
> switch and... um... NAT!". I wanted a plan, and was willing to put in
> the time and effort to build Ansible / Puppet / an NMS / AAA, etc, he
> was more seat-of-the-pants.
>
> But yes, even if we had good technology this would have failed - there
> was no real business plan (other than "The current provider is really
> bad, if we build something else, people will be breaking down the door
> to sign up"), no real marketing plan (see previous), etc.
>
> He was also a bit of a gun nut, and so would arrive at customers with
> a (holstered) firearm belted on -- even in Virginia this was not a
> winning business move.
>
> Starting a successful ISP is this day and age is hard - make sure
> that, if you do it, you and whoever you are doing this with are
> compatible, are both committed, and have similar views on things...
>
> W
>
>
> >
> >
> >
> > On Mon, Jun 3, 2019 at 8:05 AM Mehmet Akcin  wrote:
> >>
> >> hi there,
> >>
> >> I know there are folks from lots of small ISPs here and I wanted to
> check-in on asking few advice points as I am involved building an ISP from
> green-field.
> >>
> >> Usually, it's pretty straight forward to cover high-level important
> things, filters, routing policies, etc.but we all know the devil is in the
> details.
> >>
> >> I am putting together a public DOs and DONTs blog post and would love
> to hear from those who have built ISPs and have recommendations from
> Billing to Interconnection, Routing policy to Out of the band  & console
> setup, Software recommendations, etc. Bottom line is that I would like to
> publish a checklist with these recommendations which I hope will be useful
> for all.
> >>
> >> thanks in advance for your help and recommendation.
> >>
> >> Mehmet
> >>
> >>
> >
> >
> > --
> > Fletcher Kittredge
> > GWI
> > 207-602-1134
> > www.gwi.net
>
>
>
> --
> I don't think the execution is relevant when it was obviously a bad
> idea in the first place.
> This is like putting rabid weasels in your pants, and later expressing
> regret at having chosen those particular rabid weasels and that pair
> of pants.
>---maf
>


-- 
Fletcher Kittredge
GWI
207-602-1134
www.gwi.net

Re: DOs and DONTs for small ISP

[Justin Wilson]

We have a fledgling web-site with information at http://startawisp.info/ 



Justin Wilson
j...@mtin.net

https://j2sw.com
https://blog.j2sw.com


> On Jun 3, 2019, at 10:35 AM, Mehmet Akcin  wrote:
> 
> https://startyourownisp.com/  this is also 
> pretty good, just thought i would share..
> 
> On Mon, Jun 3, 2019 at 7:19 AM Mikael Abrahamsson  > wrote:
> On Mon, 3 Jun 2019, Mehmet Akcin wrote:
> 
> > I am putting together a public DOs and DONTs blog post and would love to 
> > hear from those who have built ISPs and have recommendations from 
> > Billing to Interconnection, Routing policy to Out of the band & console 
> > setup, Software recommendations, etc. Bottom line is that I would like 
> > to publish a checklist with these recommendations which I hope will be 
> > useful for all.
> 
> The "ISP Essentials" publication is still valid in a lot of cases. It 
> might not cover everything but gets a lot of the basics right.
> 
> ftp://ftp.securecomp.net/EBook/Cisco%AE%20ISP%20Essentials.pdf 
> 
> 
> -- 
> Mikael Abrahamssonemail: swm...@swm.pp.se 

Re: DOs and DONTs for small ISP

[Warren Kumari]

On Mon, Jun 3, 2019 at 1:09 PM Fletcher Kittredge  wrote:
>
>
> Here is your checklist in descending order of importance:
>
> market opportunity
> finding the right partners (see below)
> financial
> sales and marketing
> organizational capacity and HR
> legal, regulatory
> capital acquisition
> security
> ...
> ...
> ...
> technical including equipment selection, routing policy, filtering, etc
>
> It is a stone cold lock that the success of your new ISP will governed by 
> factors other than technical. Your most important task is to find competent  
> financial and marketing people you can respect and trust. If the market 
> opportunity exists and you find them, you will succeed. If you don't, all the 
> technical excellence in the world won't help you. The road is littered with 
> technically excellent companies that failed.

Indeed, but you *also* need to have some technical clue. Two or three
years ago a friend and I tried to start a local wireless ISP -- I was
doing this purely as a "My home Internet access sucks, and I'll
happily donate time, equipment, IP space and some startup capital to
fix this" play -- unfortunately it turns out that he and I had very
different ideas on, well, basically everything. I wanted an actual
architecture / design, and diagrams and routin' and such. He was much
more of "We don't need a list of IPs, if I ping it and can't reach it
it must be free" / "routing is too hard, let's just put it all in a
switch and... um... NAT!". I wanted a plan, and was willing to put in
the time and effort to build Ansible / Puppet / an NMS / AAA, etc, he
was more seat-of-the-pants.

But yes, even if we had good technology this would have failed - there
was no real business plan (other than "The current provider is really
bad, if we build something else, people will be breaking down the door
to sign up"), no real marketing plan (see previous), etc.

He was also a bit of a gun nut, and so would arrive at customers with
a (holstered) firearm belted on -- even in Virginia this was not a
winning business move.

Starting a successful ISP is this day and age is hard - make sure
that, if you do it, you and whoever you are doing this with are
compatible, are both committed, and have similar views on things...

W


>
>
>
> On Mon, Jun 3, 2019 at 8:05 AM Mehmet Akcin  wrote:
>>
>> hi there,
>>
>> I know there are folks from lots of small ISPs here and I wanted to check-in 
>> on asking few advice points as I am involved building an ISP from 
>> green-field.
>>
>> Usually, it's pretty straight forward to cover high-level important things, 
>> filters, routing policies, etc.but we all know the devil is in the details.
>>
>> I am putting together a public DOs and DONTs blog post and would love to 
>> hear from those who have built ISPs and have recommendations from Billing to 
>> Interconnection, Routing policy to Out of the band  & console setup, 
>> Software recommendations, etc. Bottom line is that I would like to publish a 
>> checklist with these recommendations which I hope will be useful for all.
>>
>> thanks in advance for your help and recommendation.
>>
>> Mehmet
>>
>>
>
>
> --
> Fletcher Kittredge
> GWI
> 207-602-1134
> www.gwi.net



-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

Re: DOs and DONTs for small ISP

[Jared Mauch]

On Mon, Jun 03, 2019 at 01:48:33PM +, Mel Beckman wrote:
> I’m constantly amazed at the number of even medium-sized ISPs that have no 
> network monitoring. An NMS should go in as the first software component — 
> before billing starts and the provider is on the hook to deliver. 

often people are using tools like quickbooks to start, these don't 
support integration with networking tools.  You see tools like Sonar or 
powercode in use.  Some of this is changing with newer tools like UNMS and UCRM 
in some spaces, but often these are vendor locked or don't integrate well.

> The second lacking component is a ticket system, which is silly because 
> turnkey cloud services are not expensive, and open source solutions abound 
> for budget-limited operators. 

The number of people who can't do sysadmin functions is high.  there's 
a reason SaaS is a thing, but the costs are often enough to force someone to 
roll their own.  Take something like powercode with a $1/subscriber fee which 
adds up quickly.

> The third component failure is security, including weak and default (!) 
> passwords, failure to use real certificates, and the complete lack of 2FA or 
> MFA. Security also requires data surveillance, in the form of net flow 
> analysis.

Much of this is because hardware has defaults that aren't sane or lack 
some ZTP or provisioning that you can do.  How do you do this with UBNT, Tik or 
other cost optimized hardware?

> The “two guys and a router” business model must be upgraded with more 
> planning and a cohesive operating plan.

Most large networks are run with small teams, while usually more than 2 
it's often not more than 10 to do the arch + eng work necessary.  If you have 
more, they're often doing installer work not actual eng work.

- Jared

> > On Jun 3, 2019, at 5:05 AM, Mehmet Akcin  wrote:
> > 
> > hi there,
> > 
> > I know there are folks from lots of small ISPs here and I wanted to 
> > check-in on asking few advice points as I am involved building an ISP from 
> > green-field.
> > 
> > Usually, it's pretty straight forward to cover high-level important things, 
> > filters, routing policies, etc.but we all know the devil is in the details. 
> > 
> > I am putting together a public DOs and DONTs blog post and would love to 
> > hear from those who have built ISPs and have recommendations from Billing 
> > to Interconnection, Routing policy to Out of the band  & console setup, 
> > Software recommendations, etc. Bottom line is that I would like to publish 
> > a checklist with these recommendations which I hope will be useful for all. 
> > 
> > thanks in advance for your help and recommendation.
> > 
> > Mehmet
> > 
> > 

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: DOs and DONTs for small ISP

[Fletcher Kittredge]

Here is your checklist in descending order of importance:

   1. market opportunity
   2. finding the right partners (see below)
   3. financial
   4. sales and marketing
   5. organizational capacity and HR
   6. legal, regulatory
   7. capital acquisition
   8. security
   9. ...
   10. ...
   11. ...
   12. technical including equipment selection, routing policy, filtering,
   etc

It is a stone cold lock that the success of your new ISP will governed by
factors other than technical. Your most important task is to find
competent  financial and marketing people you can respect and trust. If the
market opportunity exists and you find them, you will succeed. If you
don't, all the technical excellence in the world won't help you. The road
is littered with technically excellent companies that failed.



On Mon, Jun 3, 2019 at 8:05 AM Mehmet Akcin  wrote:

> hi there,
>
> I know there are folks from lots of small ISPs here and I wanted to
> check-in on asking few advice points as I am involved building an ISP from
> green-field.
>
> Usually, it's pretty straight forward to cover high-level important
> things, filters, routing policies, etc.but we all know the devil is in the
> details.
>
> I am putting together a public DOs and DONTs blog post and would love to
> hear from those who have built ISPs and have recommendations from Billing
> to Interconnection, Routing policy to Out of the band  & console setup,
> Software recommendations, etc. Bottom line is that I would like to publish
> a checklist with these recommendations which I hope will be useful for all.
>
> thanks in advance for your help and recommendation.
>
> Mehmet
>
>
>

-- 
Fletcher Kittredge
GWI
207-602-1134
www.gwi.net

Re: DOs and DONTs for small ISP

[Mehmet Akcin]

https://startyourownisp.com/ this is also pretty good, just thought i would
share..

On Mon, Jun 3, 2019 at 7:19 AM Mikael Abrahamsson  wrote:

> On Mon, 3 Jun 2019, Mehmet Akcin wrote:
>
> > I am putting together a public DOs and DONTs blog post and would love to
> > hear from those who have built ISPs and have recommendations from
> > Billing to Interconnection, Routing policy to Out of the band & console
> > setup, Software recommendations, etc. Bottom line is that I would like
> > to publish a checklist with these recommendations which I hope will be
> > useful for all.
>
> The "ISP Essentials" publication is still valid in a lot of cases. It
> might not cover everything but gets a lot of the basics right.
>
> ftp://ftp.securecomp.net/EBook/Cisco%AE%20ISP%20Essentials.pdf
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>

Re: DOs and DONTs for small ISP

[Mikael Abrahamsson]

On Mon, 3 Jun 2019, Mehmet Akcin wrote:

I am putting together a public DOs and DONTs blog post and would love to 
hear from those who have built ISPs and have recommendations from 
Billing to Interconnection, Routing policy to Out of the band & console 
setup, Software recommendations, etc. Bottom line is that I would like 
to publish a checklist with these recommendations which I hope will be 
useful for all.


The "ISP Essentials" publication is still valid in a lot of cases. It 
might not cover everything but gets a lot of the basics right.


ftp://ftp.securecomp.net/EBook/Cisco%AE%20ISP%20Essentials.pdf

--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: DOs and DONTs for small ISP

[Jon Lewis]

On Mon, 3 Jun 2019, Mehmet Akcin wrote:


hi there,

I know there are folks from lots of small ISPs here and I wanted to check-in on 
asking few advice points as I am involved building an ISP from green-field.

Usually, it's pretty straight forward to cover high-level important things, 
filters, routing policies, etc.but we all know the devil is in the details. 

I am putting together a public DOs and DONTs blog post and would love to hear 
from those who have built ISPs and have recommendations from Billing to 
Interconnection, Routing policy to Out of
the band  & console setup, Software recommendations, etc. Bottom line is that I 
would like to publish a checklist with these recommendations which I hope will be 
useful for all. 

thanks in advance for your help and recommendation.


Probably the #1 thing I've seen messed up is BGP config.

1) Nail up your routes using network statements and static null
   routes.  Don't rely on redistribute connected to advertise what's
   configured on an ethernet interface.  You probably shouldn't be using
   redistribute at all unless you "know what you're doing" with it.

2) Don't advertise your v4 IP space as a collection of /24s if you have a
   larger aggregate block, unless you have good reason to do so...and if
   you do, you should probably still advertise the aggregate unless
   there's a good reason not to.

3) Don't advertise one transit provider's routes to another.  Each should
   be filtering your routes, but you never know.  Come up with, and use
   BGP communities to control route propagation.  As you grow, it sucks
   having to update prefix-list filters in multiple places every time
   something changes...like a new customer with their own IPs.

--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: DOs and DONTs for small ISP

[Mehmet Akcin]

thank you, great start, just want to jump start to technical part AFTER
deciding to start the ISP, and focus on operational, technical, and
administrative things though. Question is no longer whether starting ISP is
a good idea or not (but i agree this is needed and was already completed)
thanks again.

On Mon, Jun 3, 2019 at 6:42 AM Fletcher Kittredge  wrote:

>
> Here is your checklist in descending order of importance:
>
>1. market opportunity
>2. finding the right partners (see below)
>3. financial
>4. sales and marketing
>5. organizational capacity and HR
>6. legal, regulatory
>7. capital acquisition
>8. security
>9. ...
>10. ...
>11. ...
>12. technical including equipment selection, routing policy,
>filtering, etc
>
> It is a stone cold lock that the success of your new ISP will governed by
> factors other than technical. Your most important task is to find
> competent  financial and marketing people you can respect and trust. If the
> market opportunity exists and you find them, you will succeed. If you
> don't, all the technical excellence in the world won't help you. The road
> is littered with technically excellent companies that failed.
>
>
>
> On Mon, Jun 3, 2019 at 8:05 AM Mehmet Akcin  wrote:
>
>> hi there,
>>
>> I know there are folks from lots of small ISPs here and I wanted to
>> check-in on asking few advice points as I am involved building an ISP from
>> green-field.
>>
>> Usually, it's pretty straight forward to cover high-level important
>> things, filters, routing policies, etc.but we all know the devil is in the
>> details.
>>
>> I am putting together a public DOs and DONTs blog post and would love to
>> hear from those who have built ISPs and have recommendations from Billing
>> to Interconnection, Routing policy to Out of the band  & console setup,
>> Software recommendations, etc. Bottom line is that I would like to publish
>> a checklist with these recommendations which I hope will be useful for all.
>>
>> thanks in advance for your help and recommendation.
>>
>> Mehmet
>>
>>
>>
>
> --
> Fletcher Kittredge
> GWI
> 207-602-1134
> www.gwi.net
>

Re: DOs and DONTs for small ISP

[Mel Beckman]

I’m constantly amazed at the number of even medium-sized ISPs that have no 
network monitoring. An NMS should go in as the first software component — 
before billing starts and the provider is on the hook to deliver. 

The second lacking component is a ticket system, which is silly because turnkey 
cloud services are not expensive, and open source solutions abound for 
budget-limited operators. 

The third component failure is security, including weak and default (!) 
passwords, failure to use real certificates, and the complete lack of 2FA or 
MFA. Security also requires data surveillance, in the form of net flow analysis.

The “two guys and a router” business model must be upgraded with more planning 
and a cohesive operating plan.

 -mel 

> On Jun 3, 2019, at 5:05 AM, Mehmet Akcin  wrote:
> 
> hi there,
> 
> I know there are folks from lots of small ISPs here and I wanted to check-in 
> on asking few advice points as I am involved building an ISP from green-field.
> 
> Usually, it's pretty straight forward to cover high-level important things, 
> filters, routing policies, etc.but we all know the devil is in the details. 
> 
> I am putting together a public DOs and DONTs blog post and would love to hear 
> from those who have built ISPs and have recommendations from Billing to 
> Interconnection, Routing policy to Out of the band  & console setup, Software 
> recommendations, etc. Bottom line is that I would like to publish a checklist 
> with these recommendations which I hope will be useful for all. 
> 
> thanks in advance for your help and recommendation.
> 
> Mehmet
> 
> 

Re: DOs and DONTs for small ISP

[Cummings, Chris]

Mehmet, I think this is a cool idea, perhaps a good format for the 
documentation would be something along the lines of an “awesome list”? 
(https://github.com/sindresorhus/awesome)

Chris

From: NANOG  on behalf of Mehmet Akcin 

Date: Monday, June 3, 2019 at 07:06
To: nanog 
Subject: DOs and DONTs for small ISP

hi there,

I know there are folks from lots of small ISPs here and I wanted to check-in on 
asking few advice points as I am involved building an ISP from green-field.

Usually, it's pretty straight forward to cover high-level important things, 
filters, routing policies, etc.but we all know the devil is in the details.

I am putting together a public DOs and DONTs blog post and would love to hear 
from those who have built ISPs and have recommendations from Billing to 
Interconnection, Routing policy to Out of the band  & console setup, Software 
recommendations, etc. Bottom line is that I would like to publish a checklist 
with these recommendations which I hope will be useful for all.

thanks in advance for your help and recommendation.

Mehmet

DOs and DONTs for small ISP

[Mehmet Akcin]

hi there,

I know there are folks from lots of small ISPs here and I wanted to
check-in on asking few advice points as I am involved building an ISP from
green-field.

Usually, it's pretty straight forward to cover high-level important things,
filters, routing policies, etc.but we all know the devil is in the details.

I am putting together a public DOs and DONTs blog post and would love to
hear from those who have built ISPs and have recommendations from Billing
to Interconnection, Routing policy to Out of the band  & console setup,
Software recommendations, etc. Bottom line is that I would like to publish
a checklist with these recommendations which I hope will be useful for all.

thanks in advance for your help and recommendation.

Mehmet