Even though we know it's technically possible, service providers aren't going to overprovision power backup unless there is a business reason to do so. Some state PUCs have minimum battery run times -- I'm sure service providers who provide telephone service are meeting that because their certificate depends on that. After that, it's just providing enough services to remain comparatively competitive.
Frank -----Original Message----- From: Ralph E. Whitmore, III [mailto:ral...@interworld.net] Sent: Sunday, August 05, 2012 11:54 PM To: nanog@nanog.org Subject: RE: Verizon FiOS - is BGP an option? What I think most people are objecting to is that most of the issue with maintain service is not related to technical capabilities, but related to the cost of providing these support services impacting the profit margins of the large monopolistic carriers. Verizon with their FIOS offerings, at least in my area is CO based and all optical, so all I have to provide is power to my own FIOS terminal which is easy to do, floated on batteries (not the POS that VZ provided but a real battery bank) and I stay online. We have been through 10 outages at least 10 hours in duration and one 3 day outage all courtesy of SCE who can't figure out how to replace the 57 year old wires that keep breaking from corrosion here in So. Calif. As a subscriber, I paid for the copper Wire that VZ installed, I paid for the Copper that Edison installed, I pay to maintain both of these services on a 7x24x365 basis. And each and every month, SCE and VZ take a mandatory deduction out of my bill specifically to replace the copper every 50 years (the design life of the infrastructure). Edison squanders that money and just patches the infrastructure with no regard for the customers, and VZ replaces the copper with FIOS so they don't have to allow any competition from anyone else and demos the copper when they are done so no one else can use it. All of these companies fail to understand that they were granted a license and handed the keys to provide a public service and we expect them to perform that job rain or shine 7x24x365. Hurricanes (while not a problem where I am) are a known problem in many parts of the company and it is their job to maintain service despite the hurricanes. These fiber huts/RSU's were installed to minimize VZ's (insert your favorite carrier here) cost of maintenance for their network . This way, they can increase their profits by laying off more workers and hiring more subcontractors. So be it, that is their business model. What people/PUC/ Regulatory bodies fail to follow up on is that just because they are allowed to install Fiber Huts/RSU's the customers should expect the same level of service and redundancy that is provided by a brick and mortar CO built to the ATT/Bellcore standards for stability and reliability. I am all for the carriers pushing the edge closer to the customers, but it should not be allowed to occur at a substandard level. They certainly aren't offering a discount for substandard service received by some. My customers get 99.999% reliability from my infrastructure, I expect the carriers to do at least as well, obviously that doesn't happen. All my Roadside cabinets have a DC plant that is engineered to hold the facility for at least 18 hours based on the equipment in the box. All my facilities have an external Transfer switch and a generator plug. A single 5kw generator, with one cable and padlock, with one tank of fuel (generally propane or diesel)(about 5 gallons) will run the hut for 8-10 hours, fully recharge the DC plant inside buying you another 18 hours on battery (therefore 28 additional hours before fuel is needed) and to boot have electronic monitoring to tell me if there is another AC failure of the generator during that 8-10 hours. One contractor, god forbid we wouldn't do this with actual employees of the company, with a pickup truck and a small trailer should be able to support between 12-20 Fiber Huts /RSU's in a single shift. If your site is bigger than a 5KW site then you should have generator backup onsite in the first place. If I can do this Verizon/ATT/Centurylink should have no trouble supporting their facilities in a similar fashion. I am a little dinky company with 12 employees, but Verizon( your carrier name here) If you want to use Wisper Watt 25KW generators, more power to you, they will just pass the costs on to us anyways in the next rate hike. The argument about lack of fuel availability is total bunk as well as I can't believe that any carrier doesn't already have 1000's of gallons onsite already to support their CO's, trucks, ect. if you need more you can order a 5000gallon tank like everyone else in the real world does, you can probably have it delivered tomorrow afternoon if you place the order tomorrow morning before noon. Lack of power at an Fiber hut/RSU is negligence at best on a carriers part, not an unsolvable technical problem My uncle who retired from Bell Atlantic/VZ after 30 years was in a rural area of Virginia and now works as a contractor for emergency restoration after storms. He was called in to work on Storm restoration he along with 20 other crews, were told report to the staging yard and wait for an assignment. It took three days to get to the staging area, They waited 4 days in the yard before being assigned work and was then told that they couldn't work more than 12 hours in a day, because VZ would only pay for a max of 4 hours OT for any personnel in a 24 hour period. They worked for 4 days and were then sent home as they were no longer needed. To the best of my recollection people were without service for weeks if not months. The safety wackos in the world will say well we don't want to tax our employees and more than 12 hours isn't safe. Ok, I will not argue the validity of their argument, personally I disagree with it, but this is an emergency response not a shareholders meeting, but sending the crews home weeks before all service is restored doesn't resonate well. IMHO Restoration was not about serving the people who were out of service, but rather spending at little as they could get away with without regard for the people who pay for the infrastructure and pay their bills. While we are at it, my uncle installed the RSU he is serviced from at his house before he retired from Bell Atlantic/VZ, they engineered only 4 hours of battery backup for their site near his house, the fiber hut/RSU is approx. 8'x8'x5' and it has only 4 -105AH batteries installed in it, there are three vacant shelves for more batteries and one totally vacant compartment left in the cabinet for battery expansion. Given the rural nature of his service RSU, the nearest stocking yard for VZ is better than an hour's drive from his RSU. Given the Union contract gives the workers 2 hours to show up at the yard from the time they are contacted ( which could take hours to do in the first place if his RSU is down) there is almost no way that you can callout a tech and make it to the site before the battery back is exhausted and Dominion Power is highly un-likely to arrive onsite in less than 12 hours from the time of call. So this site is doomed to go down from the start. A single house, my house, your house, despite what we think are not critical infrastructure to the carriers, but the fiber huts that service them certainly is and needs to be supported better than they are now. Back to the original topic, BGP works great over FIOS if you have a business account with static IP's and a Cooperative Colo. I have IPv4 and IPv6 running at my house and office (we microwave the Home's business FIOS to the office) through a GRE tunnel. It's the cheapest 35x35Mb connection around for about $200.00 with 32 static IP's around TWC wants about $2k for the same service. It took a little bit of fiddling to get the MTU right across the tunnel (ip tcp adjust-mss is your friend) I could upgrade the service for another $100.00 to 150Mb x 65Mb, but the wireless link won't support that without a large expenditure and for 12 people it's not needed. Boy that was a lot, glad I had time on a Sunday night to write this after fueling my generator. Ralph -----Original Message----- From: Frank Bulk [mailto:frnk...@iname.com] Sent: Sunday, August 05, 2012 7:41 PM To: 'William Herrin'; Andy Koch Cc: nanog@nanog.org Subject: RE: Verizon FiOS - is BGP an option? I think the term "critical" is being used in different senses in this discussion. Are people's lives critical? Yes, but the regulations for wired and wireless infrastructure don't require service providers to expend any and all costs to maintain connectivity. And we don't have ambulances and fire trucks at every corner and hospitals in every subdivision. Despite the almost incalculable value of life, there are still limitations on all the services provided to residences. Would I like to have the same uptime at my home as we have in the CO? or data center? Sure, but collectively we aren't willing, nay, able, to pay that price. Frank -----Original Message----- From: William Herrin [mailto:b...@herrin.us] Sent: Saturday, August 04, 2012 11:15 PM To: Andy Koch Cc: nanog@nanog.org Subject: Re: Verizon FiOS - is BGP an option? <snip> On Sat, Aug 4, 2012 at 10:26 PM, Nathan Eisenberg <nat...@atlasnetworks.us> wrote: >> Residences aren't critical infrastructure, no matter how angry the >> owners get. > > 911 access isn't a critical service? Fire and security panels aren't critical services? > If basic life safety and property protection aren't critical services, > I'm not sure what is. Whether each individual's residence contains critical infrastructure is a decision best left up to that individual. By necessity that makes the upstream aggregation components critical infrastructure. No different than it was for POTS 20 years ago. The Internet isn't just a toy any more. It's the primary communications channel in to many folks homes and well on its way to becoming the primary channel period. Regards, Bill Herrin -- William D. Herrin ................ her...@dirtside.com b...@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
