Re: Rogers 2022 network outage, Xona Report

2024-08-02 Thread Jean-Francois Mezei
On 2024-08-02 21:39, Jean-Francois Mezei wrote:

> Following process, redacted portions of the XONA Partners report have
> been published.
> 
> https://crtc.gc.ca/otf/eng/2022/8000/c12-202203868.htm


I have some question on terminology: (pardon my newbieness, just wanting
to be pedantic on terminology).



"Rogers staff removed the Access Control List policy filter from the
configuration of the distribution routers. This consequently resulted in
a flood of IP routing information into the core network routers, which
triggered the outage."

The report mentions Rogers uses IS-IS as interior routing protocol buit
i'll use the more generic OSPF bellow.


Questions:

1- I had always heard of routers facing the Internet (and thus doing
BGP) as "edge" or "border".   Is the term "distribution router" common
in the industry?


2- When a border/edge router receives some 980,000 route entries from
the transit provider, aren't those packets addressed to the IP address
of that edge router with port 179 and going to the router's internal BGP
process instead of being routed?   (report makes it look those packets
were left to run wild and propagate onto the intranet due to lack of ACL).

Would the rules that define which BGP routes are to be converted to OSPF
and then propagated to OSPF peers on the intranet be called an "Access
Control List"?  If not, what would they be called? (routing policy filter?)

(I have always though of ACL as a packet routing rule, not of route
building one).

Would it be fair to state that a large ISP network would use BGP to OSPF
route propagation to load balance upload-heavy site?  BGP1 advertises
itself to OSPF-A B and C as the router to talk to for packets desined to
upload-heavy site,  while BGP2 does the same for internal routers OSPF-D
E and F?

(Just trying to understand the scope of route information that Rogers's
BGP routers would want to send to the internal routing protocol.

Thanks in advance for any precisions on the above. Just want to make
sure I puch right when I make requests for disclosure of the redacted
portions.



Rogers 2022 network outage, Xona Report

2024-08-02 Thread Jean-Francois Mezei
After a July 2022 outage that caused the whole Rogers network to go down
for an extended period (bring down the single homed Interac payment
system with it across Canada), political pressure caused CRTC  to have a
process to look into it. Part of it was the commissioning of a report by
experts.

Following process, redacted portions of the XONA Partners report have
been published.

https://crtc.gc.ca/otf/eng/2022/8000/c12-202203868.htm

(The 2024-07-31 entry from Rogers Com unications Canada, ZIP file
contains the PDF report as well as Rogers letters justifying its
redactions).



Re: Qu??bec Sales tax

2018-03-28 Thread Jean-Francois Mezei
On 2018-03-28 17:45, Alain Hebert wrote:
>      Same deal as Paypal and EBay.

Paypal and EBay have not worked fevereshly to avoid a presence in
Canada. They have presence and already handle the taxes.

>      Netflix dropping their services in CDN/QC only serve  
> attempt at making yet another market grab.

Netflix has worked VERY hard  to avoid having a presence in Canada to
avoid not only taxation, but also regulation from CRTC in broadcasting,
having to contribute to various funds etc.

The danger here is that it may feel that losing its QC customers is
worth the price of maintaining the illusion it has no presence in Canada.

There is also a class action lawsuit for Netflix because it did not
follow the Québec Consumer procection law when it raised its rates a
year or two ago. Class action lawsuits can be very expensive.


And to bring his back to the network/ISP level: this is why it is very
important that ISPs remain "common carriers" who do not control or have
responsibility over content so that they don't get dragged into all
those issues.


>          ( And with all the hardware already deployed locally at the 
> many exchanges ... )

Netflix owns NO, NONE, NADA, ZERO hardware in Canada. It has no offices
in Canada.  Gifting the network appliances to ISPs means Netflix does
not own the hardware and thus maintains its "no presence here".


The second Netflix has physical presence here, the existing tax laws
kicks in and Netflix must collect federal and provincial taxes.



Re: Québec Sales tax

2018-03-27 Thread Jean-Francois Mezei
On 2018-03-27 18:28, Eric Dugas wrote:
> On the IP geoloc subject, we (EBOX) actually have multiple pools for QC-based 
> and ON-based customers.

You may all have different IP pools, but are they registered such that
geolocation services show them with different provinces, or do they all
point to your

OrgName:EBOX
OrgId:  QUEBE-50
Address:1225 St-Charles Ouest, Suite 1100
City:   Longueuil
StateProv:  QC
PostalCode: J4K 0B9
Country:CA

address ?

(there were complains in the past of some of your ON customers unable to
access Sport Check and being directed to Sport Experts store web sites
due to geolocation.


Re: Qu??bec Sales tax

2018-03-27 Thread Jean-Francois Mezei
On 2018-03-27 18:21, Ken Chase wrote:
> If Netflix has no physical presence in Quebec, what the lever are they going
> to use to force this? A lawsuit in  in the
> US? What court is going to entertain a foreign jurisdiction's tax claim in
> their court? And how would that be then enforced?


Or Netflix could just call the QC govt's bluff and just stop serving QC
customers and see QC voters rebel against the government if they lose
Netflix.

The "no presence in Canada" has had other impacts *Netflix refusing to
comply to a CRTC request for infornation claiming CRTC doesn't regulate
foreign companies with no presence in Canada).  (Ironically, Netflix has
hired lobysists who are quite active in Canada).

The jurisdiction aspect is an important one. Basically, the current
structure favour people buying services outside their own country to
avoid tax. And it isn't just in Canada. (For instance, the small
Canadian streaming services have to charge tax to their customers, but
Netflix doesn't. (google/apple do because they have physical presence here).

BTW, and this is a serious Orwelian thing, foreign providers who
register will be required to give the QC government their customer lists
so that the QC govt can find any tax cheaters who pretend to live in
different province and charhe penalies of $100 or more to those
individuals).

So the QC govt will claim some fairly serious extra-territorial powers.




Québec Sales tax

2018-03-27 Thread Jean-Francois Mezei
Not quite networking but probably relevant.

The Canadian province of Québec just introduced a new budget with
basically the intent to force foreign digital companies who sell
services to Québekers to collect the local value added sales tax and
remit those to the QC government.

The goal is to capture tax from Netflix who has so far escaped taxation
in Canada by having no legal/physical presence in Canada, no cache
servers of its own etc. Netflix does not currently collect province
information from customers (or any address info for that matter).

They based many of their arguments on an OECD study (which ironically
the Canadian federal government says is not completed yet (as excuse for
not proceeding with similar tax).

So foreign digital services will be required to require subscibers enter
AND VALIDATE their address so that they have an accurate province field
(validation remains to be finalized), and IF they sell more than $30,000
to Québec residents, will be required to self register with QC
government to collect local sales tax (and remit to QC government).

The Québec budget expects that validation of address will be based on IP
address geolocation or custoemrs send paper bills to prove place of
residence.

(Although requiring full address/phone number and sendint this to credit
card network for authorization might constitute a better means to
validate address).

I suspect the big winners will be VPN services in the USA :-)

Because many ISPs span multiple provinces, IP geolocation generally
points to their HQ address, not necessarily the province of the
subscriber. (This is especially true for DSL in bell Canada wholesale
where currently a single point of connection between Bell and ISP allows
full reach of all of its DSL territory in QC/ON. For Cable, ISPs require
different IP pools for Rogers in Ontario and Vidéotron in Ontario (with
a couple of exceptions where Vidéotron has service in a couple fo
Ontario towns). In Western Canada, things are harder as Shaw serves BC,
AB, SASK and MB.


Question about great firewall of China

2018-03-23 Thread Jean-Francois Mezei
Asking in a sanity check context.


As you may have heard, Bell Canada has gathered a group called Fairplay
Canada to force all ISPs in Canada to block web sites Fairplay has
decided infringe on copyright.   (ironically, Fairplay is copyright by
Apple, and used without permission :-)

Canada has hundreds of separate ISPs, each using a combination of one or
more transit providers (and there are many that have POPs in Canada).

(so the following question makes it relevant to the NA in NAnog).

1-

Does anyone have "big picture" details on how China implements its
website blocks?

Is this implemented in major trunks that enter China from the outside
world?  Is there a governmenmt onwed transit provider to whom any/all
ISPs must connect (and thus that provider can implemnent the blocks), or
are the blocks performed closer to the edges with ISPs in charge of
implementing them ?

I assume they are some blocked ports, and fake authoritative DNS zone
files to redirect sites like bbc.co.uk to something else?  Would DPI, on
a national scale work to look at HTTP and HTTPS transactions to kill TCP
sessione to IPs where the HTTP transaction has a banned work (such as
"Host: www.bbc.co.uk"



2-

Bell Canada used to use DPI on 1gbps Ellacoya on its wireline Internet
to detect and slow bittorrent flows down to dialup speeds. When it
started to upgrade its core network to support FTTH in 2010, the upgrade
of the BRAS routers to 10GBPS ports would have required Bell buy a
totally new fleet of DPI boxes and keep buying whenever there were
capacity upgrades. The math favoured increasing capacity instead of
limiting use via DPI throttling, especially since traffic growth was
with youtube and netflix , not bittorrent.


fast forward 7-8 years to today: Is the deployment of dedicated DPI,
capable of wire speed control of individual flows be economically
feasable for wireline internet services? (DOCSIS and FTTH speeds).

When Rogers and Comcast wanted to slow Netflix, underprovisioning links
from the Netflix appliances/CDN is much cheaper than deploying DPI. Just
curious if there is still an apetite for DPI for wireline ISPs that
deploy at modern DOCSIS/FTTH speeds.


Does the rapid move from HTTP to HTTPS render DPI for wire speed live
control useless?  ( I realise that blind collection of netflow data to
be batch processed into billing systems to implement zero rating schemes
is possible with normal routers and may not require dedicated DPI.


3-

In the case of the USA with ISPs slated to become AOL-like information
providers, is there an expectation of widespread deployment of DPI
equipment to "manage" the provision of information, or is the
expectation that the ISPs will focus more on using netflow to impact the
billing system and usage limits?

4-

Or is DPI being deployed anyways to protect the networks from DDOS
attacks, so adding website blocking would be possible?




Average number of ports on OLT cards

2018-03-02 Thread Jean-Francois Mezei
Quick question: (sanity check).

For a deployment happening now by an incumbent telco (aka: serving large
number of homes), how many GPON ports would it want per each OLT card ?

or more precisely, what sort of range is there for the number of ports
for such a deployment?

(The CRTC in Canada is asking for costing info for 4 port cards, so
wondering if this could be squewing the cost per port if cards today are
generally deployed with say 8 or 16 ports).

As an example of where I am coming from:  Bell Canada claimed that
Juniper E320s  costed $x and had 1 gbps of capacity, which means $x per
gbps of capacity, when in fact, the actual real life capacity with PPPoE
going to L2TP links was about 80gbps, which means $x/80 per gbps, so
significant difference in cost per gbps.



Re: listserv hosed? [Was: Fwd: nanog.org mailing list memberships reminder]

2018-02-01 Thread Jean-Francois Mezei
On 2018-02-01 22:59, Paul Ferguson wrote:

> Started getting a series of these just now from the past. :-)


Same here. The  821 headers show Received: to be "now", while the RFC
822 headers have a Date of first of  where Month started in
August 2017.

Suspect something got reset and the list server is just catching up with
the monthly reminders.


Re: Blockchain and Networking

2018-01-23 Thread Jean-Francois Mezei
On 2018-01-23 08:17, Jimmy Hess wrote:

> The promise of blockchain is fraud-resistant recordkeeping, database
> management,  AND
> resource management maintained by a distributed decentralized network which
> eliminates or reduces the extent to which there are central points of trust
> involved in
> the recordkeeping,


Current distributed "block chain" systems are architecturally insecure,
but with the requirement of computationally intensive "proof of work",
reduce risk of someone succesfully tampering a block to near 0.

However, to put things in perspective: Hydro Québec recently revealed
that it was not interested in "bitcoin mining" operations in Québec
which consume inordinate amount of power without producing anything of
value.



> Under the current system,  they retain an Unwarranted level of trust,  for
> example:  ARIN  Could  Delete an IP address allocation or an AS number
> allocation  after it was assigned,because  someone else told them to,

A recent case in Canada had the supreme court order Google to remove a
domain name from worldwide searches (so extra territorial court powers).

(rogue company stole product design freom real company, refused to
appear in court, so real company went to court asking Google to remove
that rogue compamy from searches).

The correct way would have been to get warrant on the registrar to take
the domain name out of the onwer's hands. Or go after the web site
provider. When the legal system starts to go after the wrong
people"/process to enforce law, you get problems.

There may be impulse to make the Internet "government proof", but this
will simply shift government actions to more inappropriate but still
avaialble methods of trying to enforce the law.




> For example:  A DNS Registrar or TLD Registry could make a change to the DS
> Key or remove
> the DS Key and confiscate a domain to intercept traffic, without even the
> permission
> of the original registrant.

Choose your registrar/regitry who will only take actiosn with valid
court orders and otherwise protect your privacy.




Re: End of 2017 hurricane season

2018-01-22 Thread Jean-Francois Mezei
On 2017-12-01 01:15, John Souvestre wrote:

> The #(provider name)sucks tweets on twitter in South Florida and South 
> Texas have essentially stopped. I assume this means that providers 
> have repaired almost all Hurricane Harvey and Hurricane Irma damage.


In an area touched by freezing rain,  homes are not damaged and people
expect services back within hours and issue "#provider sucks" tweets if
they don't.


But in an area where homes are rendered uninhabitable, it takes a while
before residents can return to their address (either new or repaired
home) and then ask for services to be restored.  The advantage of this
is this is more of a staggered restauration process so less load on
utilities who have more time to repair the shared infrastructure before
dealing with individual homes.


Re: End of 2017 hurricane season

2018-01-22 Thread Jean-Francois Mezei
On 2017-11-30 22:34, Sean Donelan wrote:

> The #(provider name)sucks tweets on twitter in South Florida and South 
> Texas have essentially stopped. I assume this means that providers 
> have repaired almost all Hurricane Harvey and Hurricane Irma damage.


Sorry for delay in this old topic.

In early december, I bicycled the florida keys.

Some of the old railway bridges have been rehabilitated and turned into
bike paths.

One of these, Toms Harbor Cut Bridge, between Duck Key and Conch key
(between Marathon and Isla Morada)
(24.7833 -80.9057)

There was an AT&T fibre optic cable clearly temporarily  hung from the
concrete railing through the bike bridge, and then hung in free air for
some distance before going underground.

This was in an area with much damage.

picture:
http://www.vaxination.ca/temp/virb0095.jpg
(and yes, bike was leaning against the cable :-(


So just because service has been restored from the point of view of
remote network managers does not mean that the reconstruction is
complete. You may find much temporary setups.

When one side of the road has been partly washed out by the ocean,
chances are that underground cables were damaged if they were on that side.


To put things in perspective:

In an area that is evacuated, such as the Keys, it takes time before a
resident that had been assumed evacuated never returns and is then
switched to "missing" (at which point his/her/their home has already
been bulldozed and debris moved to side of road for pickup). This means
specialized workers to comb through debris to look for human remains.
This is long after the media has left. (and yes, I saw that in the
Keys). Initial reports of loss of life are totally meaningless.

Meanwhile Key West was mostly spared and most of the sunken boats had
already been removed. So the average tourist wouldn't really see what
Irma did there. It was business as usual with cruise ships etc.  The
major damage was in the middle keys where media never put much attention.




Definition of ISP vs Transit provider

2017-11-22 Thread Jean-Francois Mezei
The FCC is about to reclassify "Broadband Internet Access Service" as an
information service instead of Telecommunications Service. This
prombpted the following question which isn't about the FCC action per say.

This is about how does one define Transit provider vs ISP ?

Cogent for instance acts as a transit provider to other networks but
also sells connectivity to companies.

Peer1 in Canada used to sell "transit" to a then small emerging ISP, but
as its sole transit provider, provided the BGP management as well as
peering at Torix.  Is the service to the ISP still called "transit" ?

Or would ISP be defined as the organsation which assigns IPs to end
users via PPPoE of DHCP ?

One could argue that a network which assigns 4 or less IPs per customer
would be an ISP. But what about IPv6 where the ISP could give each end
user a /64 ?

Just curious to see if there are agreed upon definitions from the
network operators's point of view.

I note that large companies tend to do everything from transit, to
residential ISP, business ISP, libraries, airports etc. For Bell Canada,
it is almost all under AS577. So separating what is telecom and what is
information becomes more "interesting".









As a point of reference this is what I *think* the FCC defines as an ISP:

##
23. Broadband Internet access service also does not include virtual
private network (VPN) services, content delivery networks (CDNs),
hosting or data storage services, or Internet backbone services (if
those services are separate from broadband Internet access service),
consistent with past Commission precedent.69 The Commission has
historically distinguished these services from “mass market” services,
as they do not provide the capability to transmit data to and receive
data from all or substantially all Internet endpoints.70 We do not
disturb that finding here.

24. Finally, we observe that to the extent that coffee shops,
bookstores, airlines, private end- user networks such as libraries and
universities, and other businesses acquire broadband Internet access
service from a broadband provider to enable patrons to access the
Internet from their respective establishments, provision of such service
by the premise operator would not itself be considered a broadband
Internet access service unless it was offered to patrons as a retail
mass market service, as we define it here.71 Likewise, when a user
employs, for example, a wireless router or a Wi-Fi hotspot to create a
personal Wi-Fi network that is not intentionally offered for the benefit
of others, he or she is not offering a broadband Internet access
service, under our definition, because the user is not marketing and
selling such service to residential customers, small business, and other
end-user customers such as schools and libraries.
##

The full 210 proposed FCC decision is at:
https://apps.fcc.gov/edocs_public/attachmatch/DOC-347927A1.pdf



Re: Broadcast television in an IP world

2017-11-20 Thread Jean-Francois Mezei
On 2017-11-20 17:14, Masataka Ohta wrote:

> It is merely that third parties should pay ISPs offering multicast
> service for them. Amount of payment should be proportional to
> bandwidth used and area covered.

Since multicast benefits the ISP the most, why should the ISP charge the
content provider for multicast?

The content provider (lets say local TV station that broadcasts the
Superbowl) can just unicast to the ISP a single stream, and give the
ISPs some pizza sized box (lets call it an "Appliance") and that box
then provides unicast delivery to each customer watching the Superbowl.

The ISP only wins in reduced transit/peering load, but not on the load
on its distribution network.

And with the switch to on-demand programming, one wonders if the cost of
setting up multicast all the way from the "border" to every bit of CPE
equipment is worth it if it is only truly beneficial for the Superbowl
and a couple of Hollywood awards ceremonies per year.



Re: Broadcast television in an IP world

2017-11-17 Thread Jean-Francois Mezei
On 2017-11-17 18:56, shawn wilson wrote:
> Besides Netflix, does anyone else offer CDN boxes for their services?


This is where local TV stations are different as they are already
present in the market they serve. They can connect locally, transit-free
to the local ISPs.

(and buy transit only for those outside of the local ISP's footprint).

Of course, when CBS sells rights to a local TV station based on its
antenna footprint, going OTT changes that as it allows a Burlington VT
station to serve people in California in another affiliate's exclusive
territory for that network.

Which is why the TV stations might require "working" geolocation to be
able to serve a Comcast customer in Burlington VT but not a Comcast
customer in Wilmington Delaware (assuming COmcast serves both for sake
of discussion).

Without this, we'll see CBS offer a nationwide SVOD service (oh wait,
they already do), and leave local TV station to have web based
newscasts since other programming will be through CBS All Access (which,
being a national service uses CDN services to get near to people).

Either way, I see TV content moving to the web which means the numebvr
of hours currently spent watching via OTA or Cable are moving to IP
networks.

An IPTV service such as Bell's already pushes that "cable TV" content
through its last mile IP infrastructure, so the main difference is loss
of multicast when programming originates outside the "BDU/MVPD"
environment. But with more and more people watching TV "on demand", the
advantages of multicast dimisnish (except for sports) because mroe a d
more programmin is watched withg unicast, at which point no different
from Netflix, Youtube etc.





Re: Broadcast television in an IP world

2017-11-17 Thread Jean-Francois Mezei
On 2017-11-17 16:37, Luke Guillory wrote:
> Have you seen what the OTA guys charge for retrans rights? They don't want to 
> do this, 


Fair point. Coming from Canada, OTA stations, because are freely
available, can't charge distributors (BDUs (MVPDs in USA) so their
revenues are purely from advertising.

So that changes the equation. If going OTT allows them to shut down
their OTA transmitters (and not pay for conversion to ATSC3) it could
result in lower operating costs.

In canada, BDU subsriptions are down and if the trend continues, NOT
making programming available on the net means you miss the boat.


In the USA, perhaps OTA stations could go to subscription model pn
Internet to replace the MVPDs revenues and end retrans disputes.?


Broadcast television in an IP world

2017-11-17 Thread Jean-Francois Mezei

Once ISPs became able to provide sufficient speeds to end users, video
over the internet became a thing.

This week, the FCC approved the ATSC3 standard.

What if instead of moving to ATSC3, TV stations that broadcast OTA
became OTT instead?  Could the Internet handle the load?

Since TV stations that are OTA are "local", wouldn't this create an
instant CDN service for networks such as CBS/ABC/NBS/FOX/PBS since these
networks have local presence and can feed ISPs locally?

And while a small ISP serving Plattsburg NY would have no problem
peering with the WPTZ server in Plattsburg, would the big guys like
Comcast/Verizon be amenable to peering with TV stations in small markets?

Some of them would also be selling transit to the TV station (for
instance, to serve its Canadian audience, WPTZ would need transit to go
outside of Comcast/Frontier and reach canadian IP networks).

But a local TV station whose footprint is served by the local ISPs may
not need any transit.

The PAY TV servives, if HBO is any indication will also move OTT, but be
served in the more traditional way, with a central feed of content going
to a CDN which has presence that is local to large ISPs (or inside ISPs).


We the traditional BDU (canada) MVPD (USA) is abandonned by the public
and TV stations , PAY TV services and SVOD services such as Netflix are
all on the Internet, would this represent a huge change in load, or
just incremental growth, especially if local TV stations are served locally?


Just curious to see if the current OTA and Cable distribution models
will/could morph into IP based services, eliminating the "cable TV" service.


Re: Calgary <-> Toronto 100% Canadian Fibre Resiliency on failover

2017-11-01 Thread Jean-Francois Mezei
On 2017-11-01 03:16, Jacques Latour wrote:
> JF, c¹est bon ça!
> 
> This is good point JF, according to
> http://www.acwr.com/economic-development/rail-maps/canadian-national we
> seem to have a single rail on top of Lac Superior.

Both CN and CP (still) have their own tracks. CP along shore of Lake
Superior, CN further north.

a more accurate map of CN where they have track rights or own tracks:

http://cnebusiness.geomapguide.ca/?MAP=WL

The black lines indicate other railways (such as CP and short lines)


As CP went on a "anthing but be a railway" policy between early 1980s
and a shoreholder revolt a couple years ago, not sure how many telecoms
would have wanted their fibre along CP tracks that CP might tear up.
(CP had requested permission to shut Thunder Bay to Sudbury in the 1980s
- it was refused).

It isn't clear to me what happens to fibre when a railway abandons and
removes tracks. For instance, Rigaud to Ottawa on CP. Ottawa to Sudbury
was sold to short line, and Ottawa-Mattawa tracks removed in 2012. CN
had long ago removed its Ottawa-Sudbury tracks.

So, from Ottawa to Winnipeg, unless a carrier follows rural road 17 (the
trans canada in Ontario) from Ottawa to Sudbury, you are essentially
stuck with the one track out of Ottawa to Smith Falls. There, the CP to
Belleville, or continue to Brockville and CN to Belleville.

Belleville to Toronto, the CP and CN tracks basically follow each other,
sometimes distant enough to have separate crossings, often share same
rail crossing barriers.

And from Pickering to downtown, it's basically the Metrolinx tracks (Go
Train) or go around on freight lines and then down to downtown (and
follow same route up towards Sudbury).

>From a "diversity" point of view, I guess you have to look at frequency
of backhoe events on railway right of way. Since railways also have
their own signaling fibre in the conduits, I suspect they have very few
"oops, forgot there were conduits below tracks" events.

Also, whether train derailments often affect fibre under tracks. CN had
a few derailments along its Sudbury-Winnipeg line last year and there
were no news of major telecom disruptions. Is it because of carriers
having diversity in routes or because the fibre under tracks is rarely
affected by derailments?

So would having carrier-A and carrier-B burried same tracks be
considered dangerous ? (along a road, I suspect the backhoe risks are
higher since individual home owners have driveways to road and could use
a backhoe without calling anyone).  But along tracks, farmers would not
use backhoes or other equipment over track ballast.

Also: there is a big difference between a highway like the 401/417 and a
road line the 17. For major highways, any upgrades/construction will
involve the govt informing the carriers who would have burried cables
along highway.

But along rural roads like the 17, municipalities often are in charge of
a strech of highway, and individual homeowners or businesses have their
driveway to the road and may not call to locate cables before having fun
with their backhoe.


Re: Google DNS intermittent ServFail for Disney subdomain

2017-10-25 Thread Jean-Francois Mezei
On 2017-10-25 13:05, Matthew Pounsett wrote:

> I'm also led to wonder how much worse it would be if all those CPE were
> open recursives instead of open forwarders.  I'd like to see CPE
> manufacturers' decision making and processes improved BEFORE we start
> encouraging them to go around ISPs' DNS servers or the large public
> recursive clouds.


A while back, the Québec government, wanting to protect its gambling
monopoly, decided to force ISPs to block a list of gambling sites (list
drawn up by the gambling monopoly to block outside competitors).

Recently, Bell Canada went to government suggesting the government setup
a internet web site block list to prevent canadians from accessing
pirating web sites.

And of course, in the USA, the upcoming decision to drop Title II for
ISPs may result in large ISPs quickly starting to play tricks on DNS
(redirecting traffic to their own properties etc).

While all this is in its infancy and may not happen, this could have
serious impact on the architecture of DNS with large swaths of customers
bypassing their ISP's DNS services.

But it is more likely that everyone would be going to 8.8.8.8 instead of
running their own recursive server. But if the "free" DNS servers also
start to play games or charge money, then CPE equipment may start
including a full bind recursive server and bypass everything.

This is why it is important for network folks to educate politicians to
not play with the internet.


Re: Puerto Rico: Lack of electricity threatens telephone and internet services

2017-10-19 Thread Jean-Francois Mezei
On 2017-10-19 18:18, Wayne Bouchard wrote:
> Well, the problem as I understand it is that the infrastructure was
> not all that great to begin with. Much of it was damaged in the first
> storm and when this second one came through, what remained basically
> disappeared.


Being hit with a Cat 5 hurricane/cyclone in a caribeean island that
hasn't been a direct hit from severe storms in decades will cause
extensive damage no matter what state its infrastructure was in before.

Vegetation that does not regular storms to "prune" it will grow to a
point where it will cause major damage when a big storm hits.

And a caribbean island who has never been "rich" will not have had, as a
priority, increasing building codes to widthstand hurricanes. Building
codes get updated after a big devastating hurricane, whether it is for
Darwin in 1974 (Tracy) or ones like Andrew in Florida.

It's easy for a state the size of Texas to send all of its electrical
utility trucks to the Houson area to repair damage. But they too would
be stretched thin if all of Texas had been leveled.

If buildings were not built to widthstand a 5 or a 4, then the building
itself becomes destructor of infrastructure as its materials become high
speed projectiles throuwn at other buildings and especially
teleohone/electrical lines.

I went through a category 4 (Olivia, Australia 1996). While the town and
building I was in (Karatha) were built to new standards and had little
damage, I witnessed the power of it, and I can totally understand Puerto
Rico being destroyed.

I know a politician with tendancy to skew facts points to Puerto Rico
having had terrible infrastructure. But consider that Darwin, a "rich"
town" was wiped out in 1974 by Tracy.

https://www.youtube.com/watch?v=B89wBGydSvs

Tracy was a 4. Maria was a 5.
(note the alert sound at start of video still sends shivers down my
spine because it was the same as I heard before Olivia hit).

The population was evacuated by 747s because there was nothing there to
support it. The road link to is (Stuart Highway) is so long that Darwin
is tantamount to an island. (especially since Stuart wasn't fully paved
back then).


Also note: in Florida, the utilities positioned all their equipment in
safe places so it could survive storm and be deployed when needed. But
what happens when there is no safe place, or the safe places become
isolated because roads become impassable?


It is one thing when a state has some areas with high level of
destruction. But when the whole state is destroyed, it is a truly
different situation because its economy is also destroyed. Florida
Power still has plenty of revenues from undamaged areas to pay for the
repairs in damaged areas. The Utility in Puerto Rico doesn't. (and if it
was finacially weak before, it makes things worse).

When you see other states' utilities coming to help in a highly damaged
area, don't think for a minute they do this for free. The local utility
stll gets a bill at the end of the day for the work done. If the Puerto
Rico company has no cash to pay, don't exopect other utilities to send
crews.



Re: Puerto Rico: Lack of electricity threatens telephone and internet services

2017-10-19 Thread Jean-Francois Mezei
On 2017-10-19 03:00, Sean Donelan wrote:

> not intended for long-term, continuous use.  The generators will need 
> maintenance and likely experience unscheduled failures the longer they're
> used.

Permanent duty diesel generators exist.  Many northern communities in
Canada run on them as their 7/24 power source.

It *shouldn't* have taken long after Maria for locals to know how much
damage there had been to electrical grid and that if it's gonna take
months to fix, you're gonna need constant duty generators.

What isn't clear to me is whether everything still depends on FEMA/army
help, or whether business is able to function autonomously and get their
own generators without the army confiscating them to be delieved to a
hospital instead.

And if you're a telco who is deprived of revenues because almost all
your customers are without power, do you spend your own money and effort
to try to get a permanent duty diesel generator to maintain your central
office, or do you wait for government to install one for you ?

It is one thing to be benevolent and wanting to have your network
backbone up, but financial realities of the cost of running a business
without revenues will eventually hit you when the disaster lasts for
months instead of days.


Re: Northern California fires and telecomm outages

2017-10-17 Thread Jean-Francois Mezei
On 2017-10-17 17:43, Mike wrote:

> trying to address. We notified ATT of our need and only days later, 
> without followup response, it was suggested that we somehow don't have 
> the legal authority to install microwave

My reaction would have been to call local mayor, mayor in twon where you
try to setup the microwave base, and then if that fails FEMA and tell
them that you can restore Internet but AT&T is blocking you. They would
have the means to give a call to ATT and get them to stop blocking you.

This is an emergency and AT&T would look very very bad if this were to
become public.


Re: California fires: smart speakers and emergency alerts

2017-10-16 Thread Jean-Francois Mezei
re: alerts


last march, Montréal had a nasty winter storm which resulted in a
stretch of highway wheree all exits were blocked for hours (the
government had inquiry on what happened). Cars stuck in there in middle
of night for 6 hours.

Once police woke up, it would have been extremely helpful if they could
have broadcasted an alert to all cars in that area, giving them
instruction on how to turn around and exit "backwards").

Similarly, in Atlanta, when a piece of highway collapsed, such alerts
might have been helpful to all those drivers stuck and unable to proceed
(and needing to turn around). But this has to be very targetted to one
antenna, not an area.

The problem is that people get annoyed by alerts that don't concern them
and if they turn it off, then it defeats the purpose for "real" alerts.


Last year, where Fort McMurray was hit by forest fires, Canada did not
yet have emergency alerts enabled. Twitter and radio were the "official"
evacuation orders. (and there were mistakes, underestimating it,
mistakes in handling traffic etc).

A telling video in case you hadn't seen it:
https://www.youtube.com/watch?v=aC2iPvXAggM


Communications systems become extremely important in such emergency
events because of the time critical nature. For instance, in Fort
McMurray, one neighbourhood had only road out and it was already in teh
fire so people evacuating had to go through it. Yet, at intersection
with highway, the first responders were slowing traffic exiting from
Beacon hill to let highway traffic through, unaware of what was going on
on that one exit from beacon Hill neighbourhood (bad neighbourhood
design BTW).  Had they stopped highway, they could have evacuated
neighbourhood quickly instead of forcing cars to be stuck in traffic
with fire all around them.

And as a sign of the times, many home cameras ran and kept sending
surveillance video to some service provider servers as the house burned
down until power cut or camera burned. (and some of the evacuated people
were able to get cable company to check iof theyr modem was still
"there" as a means to find out if their home had burned or not.

And while authorities refused to release real information on what areas
were damaged or not, Google released "before/after" satellite images so
people could check if their home was still there of not.  (the
information age defeating politicians fears of releasing information).


on lighter note: this past summer while on an Amtrak train south of
Wilmington, interesting experience to see everyuone's phone beep at
roughly same time in train car due to flash flood alert, followed by
skies opening up and dumping an ocean on the train.


Re: California fires: smart speakers and emergency alerts

2017-10-13 Thread Jean-Francois Mezei
Note: Google Maps shows various alerts applicable to the region you are
looking at in maps.

So, assuming its Speaker is geolocated, Google would know if an alert is
applicable to its location and be able to send it to the unit.


Re: Calgary <-> Toronto 100% Canadian Fibre Resiliency on failover

2017-10-13 Thread Jean-Francois Mezei
On 2017-10-13 17:20, Clinton Work wrote:
> 
> My understanding is that nobody has a 2nd diverse fiber route north of
> the great lakes from Winnipeg to Toronto.   Every provider makes use of
> a fiber route south of the great lakes thru the US in order to provide
> diversity.

But if provider 1 has its 1 fibre on the CN line and provider 2 has its
1 fibre along CP line (or road), then you can get diversity by getting
bandwidth from both.


> The following map shows that the CN rail and CP Rail lines across over
> each other at multiple times from Winnipeg to Toronto. 

At Rennie MB, the CN line has a bridge over the CP line. Between Sudbury
and Toronto, you may have to live with the crossings. But I suspect they
are bridged too (with some interchange points near Sudbury).

Ideally, there would be some link leftover from when there were tracks
between Ottawa and Sudbury. Tracks remain between Mattawa and Sudbury.
(Ottawa-Mattawa removed circa 2012).  Bell Canada still wants to serve
those areas even if tracks no longer present.



Note: road has interesting side effects. A new bridge on highway 17
"broke" when it got too cold: the stay cables on suspension bridge
contracted and ended up lifting bridge deck by about 1m above ground
level. So any fibre conduits would have been severed as it crossed from
ground to bridge.


Re: Temp at Level 3 data centers

2017-10-13 Thread Jean-Francois Mezei
On 2017-10-13 14:10, Roy wrote:
> 
> 
> The IBM 308x and 309x series mainframes were water cooled. 


The bank I worked for had just installed one. A big change were noise
levels, the thing was really quiet. But servicing now required a plumber
too. (there was a separate cabinet for the water pumps as I recall.)

But in all cases, the issue is how long you can survive when your "heat
dump" is not available. If nobody is removing heat from your water loop
it will eventually fail too.


In the end, it is a lot easier to provide redundancy for HVAC in one
large room than splitting the DC into small suites that each have their
1 unit. Redundancy there would require 2 units per suite. And the
problem with having AC units that are capable of twice the load (in case
other one fails) is that it increases the on-off cycles and thus reduces
lifetime (increases likelyhood of failure).



Re: Calgary <-> Toronto 100% Canadian Fibre Resiliency on failover

2017-10-13 Thread Jean-Francois Mezei
Answer from Allstream (aka Zayo)

A combination: Tor-Ott-Mtl N route is CP & S route is CN. From Tor-Wpg
its mostly CN on the N route and the S goes thru various US routes.



So Allstream would get you out west via the more northern CN line from
Toronto.

So you would need to find someone who has fibre along the CP line.

(note: Ottawa to North Bay, the tracks have been removed a couple years
ago, not sure if there is any fibre left.

What is interesting is Allstream saying Tor-Ott-Mtl route is on CP. CP's
transcontinental line from Montreal-Ottawa-Sudbury no longer exists.
(Rigaud to Ottawa is Trans Canada Trail now).  But CP still has its
Smoth Falls to Montreal line. (and at the DOrion commuter train station
(CP tracks) there are "do not dig" signs from Allstream.



Re: Temp at Level 3 data centers

2017-10-12 Thread Jean-Francois Mezei
back in the arly 1990s, Tandem had a computer called "Cyclone". (these
were mission critical, fault tolerant machines).

The reason for "Cyclone" name was that the cabinets had huge fan
capacity, and that was to deal with air conditioning failure by
increasing the air flow over the electronics to still keep then "comfy"
despite high data centre air temperature. (with the aim of having the
Tandem continue to run despite HVAC failure).

With dense  computers packed in 1U, you just can't have that excessive
airflow to cope with HVAC failure with tiny 1" fans.

The other difference is data centre density.  Bank computer rooms were
sparse compared to today's densely packed racks. So lots of space
relative to heat sources.

The equivalent today would be the football field size data centres from
the likes of Google with high ceilings and hot air from one area with
failed HVAC to rise to ceiling and partly be taken out by the others.

But when you are talking about downdown co-lo with enclosed suites that
are packed to the brim, failure of HVAC results in quick temp increases
because the heat has nowhere to spread to, and HVACs from adjoining also
enclosed suites can't provide help.

So when a tennant agrees to rent rack space in an small enclosed suite,
it should be considerewd that the odds of failure due to heat are
greater (and perhaps consider renting rack space in different suites to
provide some redundancy).




Re: replacing compromised biometric authenticators

2017-10-12 Thread Jean-Francois Mezei
On 2017-10-12 16:58, Rich Kulawiec wrote:

> (3) because they facilitate coerced risk transference away from the
> people who are actually responsible (and are paid to be so) to the
> people who shouldn't be responsible (and aren't paid to be)


I think biometrics are seen as a means to reduce the possible
errors/corruption of a security guard by shifting responsibility to a
computer.

When you have multiple tennants, the DC can't assume all tennants will
keep all access cards secure so has to protect tennant 2 from tennant 1
having cards stolen by some crook intent on damaging tennant 2's cards.

A security guard matching face to picture on card AND picture in his
computer for that card can be very good, and woudl eliminate card
counterfeiting (with match against the DC's database of images) but
would not eliminate security guard making mistakes and allowing people
whose face does not match (corruption or lazyness).


This is very different from a data centre owned by a single tennant who
has full control over staff and knows who is and isn't staff and
authorized to go in.





Re: Calgary <-> Toronto 100% Canadian Fibre Resiliency on failover

2017-10-11 Thread Jean-Francois Mezei
BTW,

a web site showing list of registered cellular towers in Canada:

http://www.ertyu.org/steven_nikkel/cancellsites.html

In areas where the 17 or 11 stray from railroad, you could cobine that
map with Street View to try to spot towers to see if they are on
microwave or not.

If I were to cycle the route again, I would be able to spot signs of
fibre along the road. (do not dig, or orange tags on telephone pole
lines where they exist).

There may be stretches where there is fibre along the 17 (between
Sudbury and White River, there are no through tracks, and a few towns,
so one assumes some fibre has been laid).

The folks at ViaNet.ca have laid FTTH in towns such as Chapleau, so they
would know what fibre trunks exist in the region.


Re: Calgary <-> Toronto 100% Canadian Fibre Resiliency on failover

2017-10-11 Thread Jean-Francois Mezei
On 2017-10-11 11:40, Jacques Latour wrote:
> Does anyone know if there's fibre resiliency between Calgary and Toronto over 
> the Great lakes, I thinking redundancy could be achieved by using two paths 
> one following the railroad and the other following the Trans-Canadian 
> highway.  Does anyone know if there is fibre following the Trans-Canadian 
> highway and who owns it?


More than likely one around lake Superior on CP Rail tracks, and the
other along the CN tracks further north.

Zayo in Canada is formerly CNCP telecommunications, and they are likely
first to have fibre along tracks.

Since the Trans Canada highway in that part of Ontario is actually a 2
lane rural road, I am not sure people would have laid fibre along it
knowing the progressive work to widen it might require frequent
relocation of the fibre.


Re: Hurricane Maria: Summary of communication status - and lack of

2017-10-10 Thread Jean-Francois Mezei
On 2017-10-10 00:47, Sean Donelan wrote:
> 
> The Puerto Rico government has posted threee maps of cellular coverage and 
> GPS coordinates of Cells on Wheels (COWs) in service.
> 
> http://www.status.pr/Maps/
> 
> It still looks grim in Puerto Ricofrom a telecommunications perspective. 

I found the coverage maps to be better than I would have expected.

> Claro, the ILEC but second in terms of mobile phone marketshare behind 
> AT&T,

Do AT&T and T_Mobile have much actual infrastructure or do they tend to
roam or network share with Claro? From what I read, Sprint/Verizon ride
on the  Open Mobile network as it is the only one still providing CDMA
signal.

Of course a round coverage dot on the map which provide CDMA is of
little use to those who have GSM phones (and vice versa).  (I guess LTE
roaming between Open Mobile and the GSM guys would work?)




> has started to more fully explain what "restored" means, and that 
> it doesn't mean everything as before the hurricane.  It is minimum 
> telecommunications.

I would expect that priority is in expanding coverage, not capacity.
Light up the one tower at the top of the hill with sub 1ghz band to have
farthest reach. But that means lots of people on same tower, hence lower
capacity per person.

The maps still show lots of individual round circles.



>  Claro has been more willing to talk about the 
> situation in Puerto Rico, which is why I've referencing Claro a lot more 
> than other carriers.

If AT&T and T_Mobile rely a lot of Claro, this could explain a lot why
Claro is the one speaking out since it is its radios/antennae that are
being lighted up and it would be the one with the info on work/progress.

(and I suspect AT&T and T-Mo would provide crews to helop Claro restore
basic service first).



Re: Hurricane Maria: Summary of communication status - and lack of

2017-10-06 Thread Jean-Francois Mezei
I have not ound the official announcements, but the press is reporting
that the FCC has granted Google rights to fly 30 of its "Loon" high
altitude ballons to provide cellular cervice in Puerto Rico for up to 6
months.

(From my readings, there are glorified relays of ground based signals
(which I assume some antennas have to be oriented to face up towards the
balloons).

The Loon will use spectrum allocated to the carriers they relay (and got
their OK)

Altitude 20km. (so not sure they need 30 balloons, 1 probably suffices
to cover all of PR).

I suspect more concrete info will be coming.


Re: Hurricane Maria: Summary of communication status - and lack of

2017-10-05 Thread Jean-Francois Mezei
got curious about the FCC's definition of "cell site" in the Maria
outages reports in Puerto Rico.



In the Oct 4 report: Arecibo is reported as having 68 cell sites served,
65 being out. (95.2% outage)


The FCC has an "ASR" (Antenna Structure Registration) search for cell
sites, and this points to actual masts (which I assume need some permit
above certain height).

For ARECIBO, there are 31 entries,
1 dismantled,
4 granted
2 cancelled

That leaves 24 "constructed".

These registrations do not mention which carrier(s) uses the mast.  And
include some owners such as Caribbean Broadcasting Corporation which
isn't likiely being used for cellular.

For all of Puerto Rico, it reports 930 ASR registrations. (haven't done
the parsing to see how many are "Constructed" vs Cancelled, granted,
dismantled). Lets assume 900 for sake of discussion.


So the ~1600 quoted by another organisation would have to include more
than just registered antenna masts.

Except for water towers, what other structures would be amenable to
having multiple carrier's antennas?

What is also not clear from such statistics is the fact you could have a
town with an high antenna broadcasting 850 to the whole area, and then
lots of DAS antennas at telephone pole height in the town at 1900 or 1700.

Having the 850 up and running at the top of the hill might cover the
whole town, even if it would represent only 1 of say 50 cell sites in
the area.


Similarly, covering a windy road in a canyon might be done with lots of
DAS anetnnas on telephone poles along the way. They may all be down, but
would normally serve 0 population, so is this number of "down" antennas
relevant?


During the 1998 ce storm in Québec, Hydro Québec was overwhelmed and
asked cities to identify priority sites inside their territories.

It's fancy "point to where the break is based on where everyone reports
an outage" software was useless because many breaks continued to happen
after power had been lost.

So it had to start from where there was power and work its way, fixing
breaks along the line towards those priority sites. (and once done, fan
out from there to power the non priority areas).

In many rural areas, this involved planting new poles for long
distances, rebuilding from scratch. (And only once the poles are up can
the telco restring its wiring).


What the media doesn't show after a disaster is what is still standing,
what is still working.  It could be that a large portion of telephone
poles are still standing and intact and only require minor individual
fixes. Or it could be that large swaths ave seen the poles toppled and
new ones needed with new power and telco wiring done from scratch.

Statistics may look bad showing 100,000 without power. But if it is a
single break by a branch it is easy to fix compared to having 1000
breaks by 1000 branches. So again, statistics don't give the full story
on the real extent of damage.



Re: Hurricane Maria: Summary of communication status - and lack of

2017-10-02 Thread Jean-Francois Mezei
On 2017-10-02 02:58, Wayne Bouchard wrote:
> Well, that's why recovery efforts in broad scale events like this have
> to go from a central point to pushing a perimiter farther and farther
> out. Create a habital, functional zone where workers can return to
> both to organize and recouperate and then go back out and push farther
> afield. 


Logic yes. But...

I have read stories of sick people in shelters dying because of lack of
electricity, lack of O2. Stories of FEMA sending water/food for only
half of population of a village.

This is where telecom plays a role.

If the shelter had comms, it could have told mayor "we need generator,
we need 5 tabks of O2 for sick people". Mayor could have sent request to
FEMA ASAP. My **guess** is that by the time FEMA got the requests, it
was too late  and people died.

In hindsight, every village should have been given a sat-phone BEFORE
the hurricane,

Ajit Pai complained about iPhone not having FM radio. But it is more
important for reverse communication from villages to headquarters/FEMA
to be able to transmit urgent needs, status reports, how much food/water
needed etc.

I suspect that if such comms had happened right off the bat, they would
have known that waiting for roads to be cleared wasn't sufficient and
taken a different philosophy for immediate help.

I think that disaster planners have made wrong assumptions about
cellular and terrestrial communications being robust enough to survive
cyclones.


Re: Hurricane Maria: Summary of communication status - and lack of

2017-10-01 Thread Jean-Francois Mezei
On 2017-10-02 00:32, Javier J wrote:

> I hope they do. There doesn't seem to be a shortage of FEMA, Army, etc
> personnel on the ground or a shortage of truck drivers in the US willing to
> help. If 80% of Truck drivers that pick up containers from the ports can't
> make it, then this needs to be supplemented any way possible to get things
> moving.


When disaster is in focused area (Like Houston), truck drivers can
easily return to functional cities after delivering goods to the diaster
zone (so not a strain on food/lodging in diaster zone).

If you bring truck drivers (and telecom, electrical etc) workiers into
Puerto Rico, they can't go home every night, so become a strain on
shelter/food resources.

And you can't "steal" your local workers if they are busy pickup up
their belongings from collapsed homes, waiting in long queues for food
and caring for their families.

In 1998 Ice Storm, Bombardier in Montréal had full power and got a lot
of bad publicity when it threatened to fire employees who didn't show up
for work. Seesm like mamnagement lived in areas that had power and
didn't realise how life changes when you have no power,  queue up for
wood provided by city etc. (and that is nothing compared to what people
on Puerto Rico are dealing with).



Re: Hurricane Maria: Summary of communication status - and lack of

2017-10-01 Thread Jean-Francois Mezei
On 2017-10-01 23:09, Jason Baugher wrote:
> The more I read about this, the more disturbed I get. On the one hand, we
> keep hearing that the trucks aren't moving because roads are impassable.

Note: media NEVER shows places that are up and running, only shows
disaster zones, so one may not get full story by looking at media.

Just saw a report on Al Jazeera. 2 sisters trying to get to their father
who lives up in the hills. They show some main roads now open, but they
get to a "road closed" by a huge landslide (with diggers working to
clear it) and have to walk from there, including fording rivers. They
eventially get to their dad who is still alive.

If there are many cell towers on top of hills where the roads are
blocked by landslides, trees, restoration would take a long time before
ground crews get to clear those remote roads that might be considered
low priority.

(and it isn't clear that a helicopter could land there either).

> Do FEMA and the National Guard have the authority to commandeer the trucks
> and deliver the containers themselves? The telcom companies aren't going to
> be able to do much by way of repairs without supplies.

Where telecom wiring is underground, it may be easier to light the links
back up. But where it is aerial, they would have to wait for the
electric utility to fix the poles before stringing new wiring. Not clear
how much of aerial plant needs rebuild, or mere fixes.


After Sandy, Verizon saw the state of corrosion in lower Manhattan and
decided to not fix the copper and string  fibre instead. If enough of
the copper plant is destroyed, would Claro (or govt) consider stringing
FTTH instead of stringing copper?


Re: Hurricane Maria: Summary of communication status - and lack of

2017-09-29 Thread Jean-Francois Mezei
On 2017-09-29 23:07, Sean Donelan wrote:

> I don't know what FCC and PRTRB are counting:
> 
>  286 working cell sites out of 2671 (according to FCC report)
>  96 working cell sites out of 1600 (according to PR Telecommunications 
> Regulatory Board report)

I had noticed the different numbers too. My speculation:

The 1600 may refer to antenna  sites, whereas the 2671 may be the sum of
the number of sites reported by each carrier (think a mast supporting
antennas from multiple carriers).


Assuming my logic is correct, the 96/1600 statistic may be of more use
in a "can I dial 911" point of view.  Having multiple carriers "up" at
the same tower doesn't increase geographic footprint where some coverage
exists.

>From a disaster management point of view, in a town where each carrier
has its own tower, deciding which one to light up first could be
interesting. (aka carriers getting together to compare state of antennas
in town and somehow elevating that info to whoever controls the
generators (army corps of engineers who are "foreigners" with no local
knowledge).



Re: Hurricane Maria: Summary of communication status - and lack of

2017-09-28 Thread Jean-Francois Mezei
FYI:
White House announces that the US Army Corp of Engineers is in charge of
power in Puerto Rico, and were given priorities to hospitals and other
emergency services. No mention of telecom being part of those
priorities.  Initial push is installing temporary power generation.
They are not yet working on fixing the electrical grid.

(44 of 69 hospitals now have power).

(Note: FEMA has decided to stick to road deliveries, not air drops for
supplies).




Re: Hurricane Maria: Summary of communication status - and lack of

2017-09-27 Thread Jean-Francois Mezei
On 2017-09-27 17:44, Sean Donelan wrote:

> After a week without power, all the stationary batteries throughout the 
> telecommunications network are likely completely drained.

from the point of view of cell sites, wouldn't battery autonomy be
measured in hours rather than days?  I could see some site having
autonomy in days due to permanent generator, and when fuel runs out so
does the cell site.

> I'm not ignoring the status competitive and smaller USVI and PR 
> communication providers, its just difficult to find official statements 
> from them.  If you have status about them, let me know.

One aspect often forgotten is that people have homes (or what is left of
them) families and the need to find food/water which can involve
standing in line for hours in a day and they may not be able to show up
for work. larger companies can usually find enough employees not so
hindered, but smaller outfits may not be able to remain functional due
to not enough staff able to work.

Smaller outfits may not have the ability to get petrol for their trucks
to go out oand fix things. (whereas the big guys have the credentials to
get petrol form authorities/army.







Re: Hurricane Maria: Summary of communication status - and lack of

2017-09-25 Thread Jean-Francois Mezei
On 2017-09-24 17:13, Sean Donelan wrote:

> I'm not sure what clearances they are waiting for.  If they are already in 
> Puerto Rico, self-sufficient, and respect curfews and other emergency 
> responders, they should be able to start local restoration and recovery 
> activities.

Priority is to restore communications to emergency responders, restore
power to hospitals and other critical infrastructure). So workers that
clear roads, remove dangling electrical wires would prioritize fixing of
that critical infrastructure.  That road you need cleared to get to your
fixed wireless antenna will wait.

Similarly, I get the impression that all cargo capacity into the island
is still controlled to prioritize essentials. So those spare circuit
board you need to fix a router have to wait.

Also, with residences overwhelmingly without power, fixing the "normal"
ISP business won't do much when nobody can use it. It is best to focus
on wi-fi in central locations such as shelters, and cellular for first
responders and others.

There are good reasons local governments work out disaster plans because
they need to identify in advance what gets priority after a disaster.



Re: Application Layer Gateways

2017-09-23 Thread Jean-Francois Mezei
What you do with the CPE  "firewall" settings depends on what sort of
ISP you are. Do you cater to geeks or aunts/grand mothers?

Whatever you do, I would suggest that you document in a place that is
easy for customers to find exactlyt what apps/protocols are open/closed
with the settings you've decided on (especially if it deviates from any
documentation available on the net for that device)


You could consider configuring it by default to protect the aunts and
grand mothers, but make sure geeks get the info on how to easily open
ports for their apps.

Also depends on what you block at the network level. If you block all
incoming calls to port 25, then blocking it at the CPE router won't add
much resilience against attacks as it is already blocked.




Re: Puerto Rico just lost internet?

2017-09-21 Thread Jean-Francois Mezei

During the 1998 ice storm, Hydro Québec stated its infrastructure had
not been built to widthstand this once in 100 year event.  Reporters did
some research and the next day asked him if there was a trend in
increased freezing rain events. "I'll have to look into it".

The next day, the HQ CEO came back at the daily press conference to
confirm a gradual increase in last 20 years in freezing rain events, and
after looking at situation, HQ would change standards for its
infrastructure to widstand more frequent freezing rain events.

In Ontario, the govt passed new stronger standards for utility poles
which while granfathering existing ones, required the new standards
apply before you can add one more wire to a pole. This seemed innocusous
until telcos (Bell and smaller ones) started to want to add fibre to
poles, where, in many cases, poles had to be replaced at $30k a shot,
and original owner retained onwership of new pole paid by the telco.

During the same event, Bell Canada, whose disaster plans were
overwhelmed by the extent of power outages didn't have enough mobile
generators to keep every outdoor plant's batteries charged all the time.
As a result many areas suffered rolling POTS and cellular blackouts
until a truck could there there with a generator. Because of the extent
of the event, Bell couldn't bring spare generators from the next town
over because that town was also in short supply of generators.

When the nature of disruptive weather events changes (or become more
frequent), utilities needs to adapt by adding more resiliency to
physical infrastructure and being prepared with more spare hardware to
cope with the aftermath.

Hurricanes have the advantage of giving a few days warning and
predictions are becoming more accurate. In the case of Irma, utilises
have the time to pre-position trucks/equipment so they can kick in as
soon as winds/flooding go down.

In the case of Hydro Québec, their own statistics showed significant
long term increase in freezing rain events, so easy to justify spending
money to upgrade infrastrtucture.

In the case of recent hurricanes, it is still debatable whether those
were unusual events (since many towns had not experienced such striong
weather for over 50 years) or whetgher frequencty of such events was
going to increase.

This would affect how telcos plan how resilient their infrastructure
needs to be.


Re: USA local SIM card

2017-09-18 Thread Jean-Francois Mezei
On 2017-09-18 19:01, Nathan Anderson wrote:

> The larger issue for you with T-Mobile might be their previous (and ongoing?) 
> use of AWS bands (split 1700MHz uplink/2100 downlink) for 3G service, which 
> very few phones sold outside of the U.S. support.  They have been working 
> nationwide to reallocate their AWS licenses to LTE, turn off 2G service on 
> PCS (1900MHz) bands,


T-Mobile isn't shutting down its 2G for now. AT&T did and T-Mobile hoped
to get IoT business for people whose devices are stuck on 2G. (Think
parking meters etc).

However, T-Mobile carved a big chunk of its 1900s to support 3G
(UMTS/HSPA+) leaving little for 2G.

> Very very few (if any) prepaid plans, either from the carriers themselves or 
> MVNOs, have roaming in Canada

AT&T's prepaid plans at $45 and $65 provide full roaming into Canada for
voice and Data, using your US "bucket".  The one at $30 and the daily
"pay as you go" don't.

T-Mobile prepaid has roaming in Canada for its $75 plan, as well as it
$45/$55 plans with a $5 surcharge.

because AT&T and T-Mo have reciprocal roaming with the Canadian
carriers, they can afford to offer roaming because it costs them next to
nothing. MVNOs would have to pay higher roaming fees so less likely to
include roaming in Canada.

> If you shop for other MVNOs, be very careful to get clarification on what 
> carrier's network they use before you>fork over any cash.

Sprit is very common for MVNOs which means you can't use it with a
standard handset.


Re: IOS new versions and network load

2017-09-18 Thread Jean-Francois Mezei
On 2017-09-18 08:48, Mike Hammett wrote:

> It looks very difficult to manage, given the DNS TXT records and domain 
> search fields. If it was as simple as entering the supported IP ranges, it'd 
> be a lot easier to implement. 


I would have to read the stuff again, but my understanding is:

caching server starts.
caching server registers with Apple, gives it its local IP, as well as
the IP ranges that it manages.

When a client wants something, it first reaches out to an Apple server.
That server decides which content server is nearest to the client, and
if there is a caching server in the same network, will give the client
the IP address to access that local caching server. (and this is where
there is NAT friendliness , as other have pointed out, designed mostlty
for enterprise).

The business about TXT records is to allow real IPs with multiple ranges
to be used. I *assume* that it is the caching server which reads those
records upon startup and then transmits it to Apple when it "logs in" as
a caching server. You can have up to 24 chained TXT records to list all
the IP blocks you can service.





Re: IOS new versions and network load

2017-09-17 Thread Jean-Francois Mezei
On 2017-09-17 19:37, Eduardo Schoedler wrote:

> Server is an app now, any MacOS can have it running.

But do carriers/ISPs really want to deal with a rack unfriendly Mac Mini
or iMac at a carrier hotel?  If the Server App could run on Linux, or if
OS-X could boot on standard servers, perhaps, it it seems to be a very
bad fit in carrier/enterprise environments.

> Implementation will be a little tricky, because you need your
> customers to look a record in your domain.


I've tried reading some about it.
The cache server app registers with Apple its existence and the IP
address ranges it serves

When a client wants to download new IOS version, Apple checked and finds
that the client's IP is served by the caching server whose "local" IP is
a.b.c.d (akaL the inside NAT IP address). Tells client to get version of
software from that IP address.

The DNS TXT records are used by the Caching Server to get the list of IP
blocks it can serve.  (not needed in the target small office
environments where everyone is on same subnet and the caching server can
tell the apple serves the one subnet it seves).



Re: IOS new versions and network load

2017-09-17 Thread Jean-Francois Mezei
On 2017-09-17 18:41, Eduardo Schoedler wrote:
> https://www.peeringdb.com/net/3554

Peering would reduce an ISP's reliance on transit provider (and thus
load on transit providers) hut still present same problem on the ISP's
internal network.

Also, doesn't Apple use a CDN such as Akamai or L3 to deliver content
like that?

> "We do have another option to consider -
> http://www.apple.com/osx/server/features/#caching-server";

Considering Apple has been out of the server business since 2010, Would
ISPs really bother installing/configuring (and finding a spot on a rack
shelf ) for a Mac Mini only to reduce load once a year ?



Re: USA local SIM card

2017-09-17 Thread Jean-Francois Mezei
On 2017-09-17 16:40, Max Tulyev wrote:

> 1. My phone is not LTE but 3G GSM/UMTS capable (all bands,
> 850/900/1700/1900/2100). Will it work? Is 3G coverage good enough in New
> York and Orlando for VoIP calls (SIP, Viber, Skype)?

3G coverage is a superset of LTE coverage. (aka: carriers still have
some areas that have 3G but not LTE).

AT&T has 850 and 1900 in 3G. the AWS (1700/2100) is for LTE only.
AT&T has shutdown 2G, but T-Mobile still has it.

In Canada, Rogers still has 2G, but Bell/Telus never had 2G. (they went
from CDMA to 3G circa 2010).


T-Mobile does not have 850. It has AWS (1700/2100) and 1900 in the above
list. Originally, it had 1900 only (2G). When it acquired 1700, it
deployed 3G on it. But because the big US carriers deemed 1700 to be for
LTE once it arrived, very few handset manufacturers provided support for
3G on 1700, especially during the days when handsets coudl only support
3 or 4 frequencies. Many of the hansets custom ordered by T-Mobile and
the 3 small new Canadian carriers replaced 1900 support in 3G with 1700
support.  (so when Rogers got the Mobilicity customers, many of them had
handsets that could not support 3G services in 1900 so Rogers had to get
a package to upgrade those customers).



T-Mobile has subsequently refarmed 1700 to support LTE, and split its
1900 to support 2G and 3G. It has since acquired some 700 for LTE
service, but this is no help to you. However, as a 3G-only user on
t-mobile, you are limited to 1900 which has shorter propagation from
antennas. So consider that the T-Mobile coverage maps may be built with
700mhz propagation in mind, so you would not get as much coverage on a
1900 only sertvice.   There may still be areas where 3G is on 1700, but
propagation is similar to 1900. (assuming your handset can support 3G on
AWS (1700/2100).

Note that AWS (1700/2100) is not used outside North America, even if
frequemncies such as 2100 are. Carefully check your handset's specs.




> 2. Is there public or private IP address? IPv6?

I can't answer this. During my bike trip, I choose AT&T because it is
the service which cuases me the least amount of waiting to post a tweet
or check emails.  Getting the IP address on an iPhone isn't easy so I
didn't waste any time doing this.



IOS new versions and network load

2017-09-17 Thread Jean-Francois Mezei
A couple years ago, Apple unleashed an IOS update which made the news
because network operators reported serious congestion on their networks
as everyone and their uncle tried to download the gig+ package at 11:00 PDT.

Was the problem solved simply by Apple staggering the announcement of
downloads? or were there distribution network changes also made to
reduce the load?


In Canada, during net neutralirty hearings, it was revealed that
cellular carriers zero rated over the air updates.  I know my iPhone
gets updates without me asking for them, only getting a "update ready to
install" while on a long cycling ride (aka: must have used cellular data).

Does anyone know whether this is pushed by Apple who has gotten the OK
form individual carriers, or is it pushed by carriers (with Apple's OK)
in a low priorioty stream that doesn't cause congestion on cellular
network? (carriers delivering content in "push mode" would change their
role).



Re: USA local SIM card

2017-09-17 Thread Jean-Francois Mezei
Addituinal notes:

When setting up AT&T prepaid, at one point you need to insert the SIM
into your handset in order to receive a confirmation code (your login
password).

I know this process works while the handset is in Canada. Even though
service is not yet activated on this SIM, the SIM can still receive SMS
from AT&T via the Rogers network.  I *assume* this would work from other
countries as well but can't garantee. Rogers and AT&T tend to have
fairly compatible and very "connected" systems for roaming as AT&T
Wireless(original company) used to own a big chunk of Rogers Wireless.

(this step is required for you to login to your new account, deposit
funds via credit card and choose your package).



Re: USA local SIM card

2017-09-17 Thread Jean-Francois Mezei
BTW, AT&T's prefered roaming partner in Canada is Rogers.

In other words, if you have an AT&T SIM card, it will try to log in
first via Rogers. I assume it also roams with Bell/Telus as second
choices but have not been able to test it.



Re: USA local SIM card

2017-09-17 Thread Jean-Francois Mezei
On 2017-09-17 13:07, Max Tulyev wrote:


AT&T's $45 prepaid pans and its more expemsive sibbling (I think $65)
allow over 6GB of data at LTE speeds, and the rest is unlimited but at
2G speeds (I think).


The AT&T plans at the $45 and higher levels allows data and voice
roaming into Canada, as long as your usage in Canada represents less
than 50% of total use.

The AT&T plan allows you to remove video throttling (the T-Mobile plan
doesn't and has more severe net neutrality violations).

If you obtain a SIM card from eBay, there is a hard to find web access
to set it up (normal AT&T web site forces you to buy a SIM card which
AT&T won't deliver outside of USA).

https://www.att.com/prepaid/activations/#/activate.html

In my case, I choose AT&T because I tested T-Mobile a few years ago
along the route taken and found too many areas without service,
interestingly, one area where in 1998-1999, I had service with Omnipoint
on a 1900 only phone (Fort Edward NY).

Note on T-Mobile: its coverage map expects you to be on postpaid plans
which includes areas where you're allowed to roam on AT&T, but not
necessarily if on prepaid, so hard to tell if you will really get
service based on its maps.

Also note: AT&T on an iPhone gets to disable the "manual" seach for
available carriers, so you can't test in a town if T-Mobile would also
be available. You can insert you own SIM card just to scan for networks
and with roaming disbaled, you won't encurr any charges by home carrier.


Re: Moving fibre trunks: interruptions?

2017-09-02 Thread Jean-Francois Mezei
On 2017-09-01 18:38, Ricky Beam wrote:

> Buried stuff requires a great deal of planning, permitting, and insurance. 

Are cables in railway right of way considered "burried stuff" from the
point of view of all the regulatory approvals since it is on private
land (railway's) ?

I take it that it is the railway which burries a new cable in its
ballast (since it knows where other cables are burried, has to handle
cable crossing its bridges etc)?

In the specific case of Turcot in Montréal, the government was in charge
of cleaning the land, removing any obstructions (such as a major sewer
collector which had to be moved) etc, and even drained and compressed
the ground before handing it over to CN to build its tracks.  So CN got
a clean slate, ready to lay tarp, ballast and tracks (and later string
fibre).

(ironically, that land used to belong to CN and was the Turcot rail yards).



Re: Moving fibre trunks: interruptions?

2017-09-01 Thread Jean-Francois Mezei
On 2017-09-01 16:12, Alain Hebert wrote:
>  Being somehow familiar with how things operate when it involve 
> Quebec Govt and the Fed Govt...  Expect hell.  Pray for purgatory.  
> Rejoice if it takes less than 3 months.


In this particular case, the government is giving CN new land, and once
construction crews for the highway/interchange have moved on, segments
are opened for CN to bring its crews to install tracks, portals,
signals, track service road etc.

The main contract gives CN responsability to handle the telecom under
its tracks, so I assume that once CN is given access to the full length
of new right of way, it will coordinate with the various telecom
companies that rent space under its tracks to do the move.

The move is expected in summer 2018. (during next winter, the last
remaining elevated structures that block the new CN right of way will be
torn down, allowing CN to then finish the work starting in spring. (it
does not lay tracks in winter).






Moving fibre trunks: interruptions?

2017-09-01 Thread Jean-Francois Mezei

A large highway interchange is being rebuilt in Montréal (Turcot) and
this requires that the CN mainline tracks out of downtown be moved a few
hundred metres to the north for a couple of kilometres until it rejoins
the existing alignment.

Part of the contract involves the cost of moving the fibre trunks along
with the tracks. (old alignment will become commercial properties).


So they have new cable that goes through the new alignment and joins the
old one at both ends.  So they'll have hundreds of strands to splice.

When doing that type of work, how much downtime can be expected for each
strand?

Would they typically use patch panels in central offices to move a
customer to a spare strand while they splice their assigned strand to
use the new cable segment (and then move traffic back to that assigned
strand?). Or would they switch customers around to new strands and
update their documentation on which customer is on which strand?

Or do they do nothing at patch panels in COs and just take whatever time
it is needed to have crews at both ends of the work site splice each
strand at same time (I assume about 5 minutes outage for each strand?)


Would they normally involve the customer advising them of upcoming
outage? Would the folks working trackside be limited to overnight hours
to make outages less significant, or do they work around the clock ?



Re: Hurricane Harvey - Network Status (FCC)

2017-08-28 Thread Jean-Francois Mezei
On 2017-08-27 20:58, Tim Jackson wrote:
> KHOU's local transmitter (Missouri City I think is where it's at) seems to
> be back on the air, but with all production from WFAA out of Dallas.


KHOU had a tweet with video showing the water flooding into their
offices/studios and staff having to leave.

https://twitter.com/sallykhou11/status/901805513905668096

I guess this is where disaster tolerance/recovery plans really kick in.


Re: Last Week's Canadian Fiber Cut

2017-08-24 Thread Jean-Francois Mezei
On 2017-08-16 18:29, Christopher Morrell wrote:
> Let’s not forget that all POTS and cell service was offline during the
> outage - even for local and 911 service.


It would be interesting to know how incumbent telco services within
Aliant territory became dependent on a link to central Canada. Whenh on
dials 911 in Moncton, does it require some database inquiry to some
database in Toronto, failing which, the call can't go through?




(Aliant, which used to be separate maritime telcos was bought lock stock
and barrel by Bell Canada a couple years ago, so likely rationlized some
services to save costs, so anything that became dependent on some
Toronto server would stop working)

The CRTC asked Bell for a report on what happened, but told media that
report may not be made public.


Re: US/Canada International border concerns for routing

2017-08-21 Thread Jean-Francois Mezei
On 2017-08-09 10:11, Hiers, David wrote:
> That is what our lawyers are starting to figure out, too.  Very glad to see 
> them converging on the tribal wisdom.

late to discussion.

You might get some organisations which require you to provide
intra-canada routes for privacy reasons. But at the moment there are no
laws that require it.

Also, you need to consider that the way the Internet is designed, should
a Montréal-Toroonto link go down, traffic will automatically reroute
Montréal-New-York-Chicago-Toronto. So it becomes hard to *guarantee*
intra-Canadian routes. (such arrangements do exist for military type of
classified private networks).


It is consumer pressure and advocacy groups who are raising the issue of
intra-Canada routing. (Patriot Act in USA gets NSA to listen to any/all
intl traffic, and Canada-USA-Canada traffic is considered such by USA).

But from a regulatory poimt of view, the most one could expect would be
a requireement to openly peer at exchanges where a netowrk has a
presence. (as opposed to garanteeing intra-canada routes). And even that
isn't on horizon at the moment.

Note that normal businesses want to peer because it reduces costs.  The
old incumbents such as Bell work on a monopoly mentality of forcing
people to buy transit from them, so allowing peering is against their
philosophy of forcing yo to buy transit. (and if you don't buy from
them, you then have to buy extra capacity to USA to connect to them).

Some US transit providers, after having been here for a while, start to
get their own intra-Canada links (such as Montréal to Toronto) where
traffic warrants. reduced latency is likely the biggest winner in this.




Re: Consumer networking head scratcher

2017-03-01 Thread Jean-Francois Mezei
On 2017-03-01 11:28, Ryan Pugatch wrote:

> At random times, my Windows machines (Win 7 and Win 10, attached to the
> network via WiFi, 5GHz) lose connectivity to the Internet. 

> For what it's worth, the router is a Linksys EA7300 that I just picked
> up.


Way back when, I have a netgear router. It ended having a limit on its
NAT translation table, and when I had too many connections going at same
time (or not yet timed out), I would lose connection. There was an
unofficial patch to the firmware (litterally a patch in code that
defined table size) to increase that table to 1000- as I recall.

Does the Linksys have a means to display the NAT translation table and
see if maybe connections are lost when that table is full and lots of
connections have not yet timed out ?



Re: backbones filtering unsanctioned sites

2017-02-16 Thread Jean-Francois Mezei
On 2017-02-16 14:59, Sadiq Saif wrote:

> From -
> https://arstechnica.com/tech-policy/2017/02/a-court-order-blocked-pirate-sites-that-werent-supposed-to-be-blocked/


Many thanks.

pardon my ignorance here, but question:

For an outfit such as Cogent which acts not only as a transit provider,
but also edge provider to large end users, can it easily implement such
a court order to block only edge interfaces and not to its transit
infrastructure?

(aka: propagate null routes for 104.31.19.30 only to interfaces that
lead to end users, but leave core/GBP aspects without the block.)

Or is BGP and any internal routing protocols so intermingled that it
becomes hard to manage such blocks ?

The difficulty for network to block traffic becomes an important
argument when trying to convince governments that blocking should not be
done. (ex: Québec government wanting to block access to gambling sites
except its own).



Re: backbones filtering unsanctioned sites

2017-02-14 Thread Jean-Francois Mezei
On 2017-02-14 08:27, Jared Mauch wrote:
> So risk avoidance on the part of the 100k other sites hosted by CF is now a 
> conspiracy? 


Cogent is a backbone network that is international in scope.  When China
tells a network to block the BBC that block happens only in China.

If the USA wants to be like China and start blocking web sites it
doesn't like, then it should only affect traffic in the USA.

Google is a content company. Removing a company from its search results
is a content issue, not a telecom issue.

Cogent blocking an IP is a telecom issue and at least in canada should
this be brought up at CRTC, would raise a Section 36 violation.

And if transit providers start to block content, especially if they do
not warn their ISP customers (so thei can warn their retail customers),
then this is really not correct.


In Canada, the supreme court has ruled, from different slants all
reaching tghe conclusion that a neutral carrier is not responsible for
the content that travels through its pipes. The second that carrier
starts to exert control over content, it loses that immunity.

Cogent blocking content affects traffic outside of the USA.


Re: backbones filtering unsanctioned sites

2017-02-13 Thread Jean-Francois Mezei


Cogent seems to have been very very silent on the issue.

Could this be because they got some police/NSA/FBI letter requiring
confindentiality and requiring Cogent to snoop on all traffic to
104.31.19.30 , and along with agreeing to comply, blocked all the
requested traffic which means that their cooperation yield logs of what
IP has made a SYN to 104.31.18.30 but since that SYN went nowhere,
contains no other information, so the agency gets its logs as requested,
but with no actionable information in them ?

That would explain the block AND Cogent being coy/silent on issue.

This could be a "protect users" move even though on the surface Cogent
appears to be the bad guy.

The other question is whether other major backbone providers got the
same order and complied without telling ayone nor taking any action to
block.

In my case, the ISP I used has local peering with Cloudfare, so not
affected. Not sure what percentage of users have local transit-free
connections.




Re: backbones filtering unsanctioned sites

2017-02-10 Thread Jean-Francois Mezei
Since 104.31.19.30 is an anycast IP, is it possible that this isn't
related to PirateBay but more related to Cogent  having a dispute with
Cloudfare ?

It is counter intuitive for a transit provider to refuse
business/traffic, but then again, Cogent has been involved in counter
intuituve disputes in the past.

I note that this has been going on since last night (at least).


It hasn't been resolved, nor has Cogent issued a statement about it (or
has it ?)




Re: Benefits (and Detriments) of Standardizing Network Equipment in a Global Organization

2016-12-29 Thread Jean-Francois Mezei

When doing business in 100 countries, what if vendor A has support in 80
of those countries, and vendor B has good presence in the last 20 ? What
if you require a vendor that has presence in all countries and this
limits your RFPs to a single vendor ?

Does your company run semi autonomous subsidiaries in each country with
its own IT/networking staff ? Who buys local connectivity, HQ in USA or
the local subsidiary ?

So, if you maintain links to 100 countries around the globe, do you want
central management ? can it provide localised support in local languages
and local times zones from head office ? Or would that stretch it beyond
reasonable capacity and you start to need support from different
locations anyways ?

Does HQ staff have legal knowledge or all local regulations? Do they
have experience bribing officials where bribing is part of business?

What happens when country X has special legal requirements, and country
Y has conflicting requirememnts that prevent uniform deployment ?

It wasn't that long ago that US equipment with encryption couldn't be
exported everywhere because encryption was considered a military secret.

Consider that in today's environment, it isn't so ludicrous to suggest
that a country may require that the equipment has a "backdoor".  So it
is best to allow that country to have its own separate equipment with
minimal management abilities to/from that country to prevent that
country's government from interfering with your opps in other countries.

It may seem more efficient to manage everything centrally but...

I'll use an airline analogy:

Southwest airlines was quite succesful with a single plane type. Common
training for pilots, all planes maintained from a central hangar, common
spare parts etc. If it had 100 planes, and all were maintained from one
hangar capable of maintaining 100 planes, this was much better than
having 50 737s, and 50 A 320s, requiring 2 separate hangars, each used
at only 50% capacity.

BUT, if you have 200 planes, then the second batch of planes could be
Airbus, and maintained in their own hangar, resulting in both hangars
being used at 100% capacity.

The point is that beyond a certain size, the advantages of having
everything common are not as important, and having dfferent equipment
gives you more leverage when negotiating, as well as isolates
bugs/viruses to only part of your network.



Another aspect is of innovation. When HQ standardizes with one vendor
and it is all centralliy managed, it becomes really hard to introduce
new technologies because your systems are cast into concrete. If you
give each country some autonomy for local equipment, they may be
experimenting with different vendors and could find that some new vendor
is much better than the one used at HQ, and that experience could then
percolate up to headquarters (instead of everything decided at HQ and
percolating down to each branch office/subsidiary).


At the end of the day, it all depends on how autonomous each subsidiary
is around the world.

This is quite different from having 100 branches in the USA, each
getting physical connection from the same fibre vendor and each
operating under same laws, and minimal time zones (still 5 hours between
New York and Hawaii though).




Re: Canada joins the 21st century !

2016-12-23 Thread Jean-Francois Mezei
On 2016-12-23 10:37, Seth Mattinen wrote:

> It would certainly suck to be an ISP in Canada and be forced to fund 
> your competitors. Or does Canada not have any small privately run ISPs 
> like we do in the US?

We not only have smaller ISPs, but also a wholesale framework where ISPs
can purchase access to last mile from the incumbents (telco and cable).

The current plan (which will be defined in a subsequent proceeding)
calls for any ISPs with more than 10 million in revenues to contribute
to the fund, the amount being a percentage of their revenues.


The percentage will be adjusted annually to cause contributions to the
fund to total the desired amount  ($100 million on year 1, increasing by
$25m until iut reaches $200).

And yeah, this means ISPs contribute money which will be used by a
competitor to deploy in rural areas.

Of course, since we're talking about billions to get all Canadians
connected, this is peanuts.


Canada joins the 21st century !

2016-12-22 Thread Jean-Francois Mezei
This is more of an FYI.

Yesterday, the CRTC released a big decision on broadband. In 2011, the
same process resulted in CRTC to not declare the Internet as "basic
service" and to set speed goals to 1990s 5/1.

Yesterday, the CRTC declared the Internet to be a basic service (which
enables additional regulatory powers) and set speed goals to 50/10.

Note that this is not a definition of broadband as the FCC had done, it
one of many criteria that will be weighted when proposal to get funding
is received. But hopefully, it means the end of deployment of DSL.


Also, as a result of declaring it a basic service, the CRTC enables
powers to force ISPs to contrtibute to a fund that will be used to
subsidize deplooyment in rural areas.

It plans to collect $100 million/year, increasing by $25m each year to
top at $200m which will then be distributed to companies who deploy
internet to unserved areas.

By setting the speed standard to 50/10, it basically marks any territory
not served by cableco as underserved since telco's copper can't reliably
deliver those speeds.


Nothing happens for now because a "follow up" process is needed to
decide how the funding mechanism will work (what portions of a companies
revenues are counted to calculated its mandated contribution to fund)
and how the process of bidding for subsidies will work. That could take
1 to 2 years.

Also in the decision is the phasing out of the equivalent programme for
POTS which saw telephone deployed everywhere. The difference is that the
POTS program had an "obligation to serve" whereas the internet doesn't.


Multicast IPTV on DOCSIS

2016-12-16 Thread Jean-Francois Mezei

Today, Rogers (in Canada) announced it was ditching it long running
project to move to an IPTV platform it had been developping and will
adopt Comcast X1.  (some $500 million writeoff).

Telco IPTV systems use multicasting all the way to the customer's LAN
and generally use the Microsoft/Ericcson MediaRoom product suite and
compatible STBs.

Could a cableco have deployed Mediaroom on a DOCSIS system? Does docsis
support multicasting ? I assume it would run on a separate DOCSIS group
of NTSC channels on the coax and likely require 2 modems in the home.

I am curious on why Rogers would have spent so much time trying to
develop its own system.

With regards to Comcast X1, is this a very conventional TV on coax
system, with a bunch of switched video channels to cater to on-demand
services to invividual homes ?

Or does X1 use some form of IP connection to deliver "data" for the
on-demand content ? (while linear TV still provided in traditional
digital channels bundled into an NTSC channel ?

BTW, one of the original arguments for Rogers is that cable STBs had
proprietary software that was slow to evolve and get new features
approved/tested, so they lagged behind the IPTV STB software which
didn't require certificatioN/approval by the STB hardware vendors (cisco
etc).

Has any Cable system converted to IPTV ?









Re: Wanted: volunteers with bandwidth/storage to help save climate data

2016-12-16 Thread Jean-Francois Mezei
On 2016-12-16 10:58, Rich Kulawiec wrote:
> This is a short-term (about one month) project being thrown together
> in a hurry...and it could use some help.  


How much data are we talking about here?  A few floppy disks ? a couple
of megabytes ? gigabytes ? terabytes ? petabytes ?

Have you considered giving "courtesy copies" to other environmental
organistaions such as Environment Canada, Australian Bureau of
Meteorology etc ?





Re: Voice channels (FTTH, DOCSIS, VoLTE)

2016-11-21 Thread Jean-Francois Mezei
On 2016-11-21 21:56, joel jaeggli wrote:

> Not really the air interface uses OFDMA coding scheme, so it is both
> divided into sub-carriers from 1.4 to 20mhz wide which are then also
> scheduled accordingly.

I have read in a number of places that 1 * 20mhz yields much more
capacity than 2 * 10mhz for LTE. but...

On the other hand, just read something on
> https://www.nxp.com/files/wireless_comm/doc/white_paper/3GPPEVOLUTIONWP.pdf


and it states:
##
Unlike single carrier systems described above, OFDM communication
systems do not rely on increased symbol rates in order to achieve higher
data rates. This makes the task of managing ISI much simpler.
***OFDM systems break the available bandwidth into many narrower
sub-carriers and transmit the data in parallel streams.***
 Each subcarrier is modulated using varying levels of QAM modulation,
e.g. QPSK, QAM, 64QAM or possibly higher orders depending on signal
quality. Each OFDM symbol is therefore a linear combination of the
instantaneous signals on each of the sub-carriers in the channel.
Because data is transmitted in parallel rather than serially, OFDM
symbols are generally MUCH longer than symbols on single carrier systems
of equivalent data rate.
##


At page 8:
##
In OFDMA, users are allocated a specific number of subcarriers for a
predetermined amount of time. These are referred to as physical resource
blocks (PRBs) in the LTE specifications. PRBs thus have both a time and
frequency dimension.
##

At page 9, a table shows that a PRB is 180KHz, and that if you have
20mhz of spectrum, you have 100 PRBs.

And more importantly:
##
A PRB is the smallest element of resource allocation assigned by the
base station scheduler.
##


Intertingly, the data I have read in that document points to performance
that is linear with more spectrum, no mention that 1 block of 20mhz
yields more capacity than 2 blocks of 10mhz.


So, if I read this right, (and please confirm if I understand
correctly), an LTE system of 20mhz breaks itself into 100 180KHz chunks
(a PRB) and the base station then schedules which user gets to use which
PRB.

So instead of giving each use a time slot, OFDMA gives it one or more
PRB, a frequency slot  180KHz wide ?

I assume that this is how VoLTE gets priority, with VoLTE bandwidth
causing the base station to give the handset enough PRBs to handle the
VoLTE connection, at the expense of normal users who will see a reduced
number pf PRBs given to them for default data ?

Is that how it works ?

Would it be correct to assume there is some baseband signalling so the
base station tells each user which PRBs it should be listening to (and
sending on for the uplink) ?


(One piece of text which helped me understand was stating that LTE
doesn't transmit packets).





Re: Voice channels (FTTH, DOCSIS, VoLTE)

2016-11-21 Thread Jean-Francois Mezei
On 2016-11-21 15:18, joel jaeggli wrote:


> SRB and URB are the l2 presentation of the tunnels established for user
> and signaling traffic.

OK, so wth LTE, if carrier has 10mhz up and down, this represents a
single chunk of spectrum providing one pipe ? (in fibre terms: a single
light colour through one strand)

The "smoke and mirrors" is accomplished by having different tunnels
inside that one pipe, with some tunnels granted QoS or other
preferential treatment between the IMS/VoiP servers and the RAN ?

When a handset sends a VolTE packet to the "IMS" APN, is there any
preferential treatment given between the handset and the antenna ? Or
does preferential treatment begin at the RAN where the packet is
recognized as going to "IMS" APN and going on the fast track to it ?

or put another way. If everyone uploads a HD selfie movie at the same
time, are handset uploads slowled with normal TCP flow control (drop a
packet, no ack received, handset halves the TCP window size)?

In other words, some router near antenna, to prioriotize packets to the
IMS/VoLTE server, will flow control normal IP traffic to maintain
sufficient upload capacity for VolTE traffic ?

Or are the tunnels fixed in capacity such that unused capacity in one is
never used by the other ?



>From a policy point of view, if I propose a net neutrality policy, I
have to make sure it doesn't prevent normal VoLTE functioning, while
preventing abuse of the ability for an incumbent to prioritize/zero-rate
its own services.

For instance:


AT&T in USA zero rates voice but not video calls over VoLTE.
Rogers in Canada zero rates both voice and video calls over VoLTE.

So if VoLTE video travels to the same IMS as voice, and not through the
normal IP APN, that means AT&T has to count the video traffic separately
and add it. But if Video goes through the normal IP traffic APN, it gets
counted fairly, like Skype packets, but Rogers then captures that
netflow and later deducts it from the total usage.

The issue here is that VoLTE is the new kid on the block with video
capability and incumbents can use their power to displace competitors
such as Skype/Facetime and that may constitute undue preference, unless
the standards are such that they have no choice because that it how it
has to work. (But AT&T shows that it can still count video and treat
video calls fairly compared o skype video calls).




Re: Voice channels (FTTH, DOCSIS, VoLTE)

2016-11-21 Thread Jean-Francois Mezei
On 2016-11-21 02:53, Mikael Abrahamsson wrote:

> Typically it travels on another "bearer" compared to Internet traffic.
> 
> http://blog.3g4g.co.uk/2013/08/volte-bearers.html
> 
> Think of bearers as "tunnels" between the mobile core network and the 
> device. 

Many thanks for the pointer. The fact that VoLTE has its own dedicated
APN explains things.

I am however a bit confused on the "bearer" term.

Say a carrier has spectrum in 700Mhz  bands A and B  each 5mhz in each
direction, bonded together as a single 10mhz (each way) channel.

The docunment states:
"R.92 requires the use of a particular set of radio bearers"

Does this mean that a bearer is given specific spectrum within a block
(such as a dedicated colour on a fibre) or that it is just given
dedicated capacity on the single data channel formed by LTE compressing
all of the spectrum into one big channel ?

I though I understood the concept when the name "tunnel" had been
mentioned because I understand that a handset estabishes a "hopping"
tunnel with local IP which changes as you move from tower to tower, but
the tunnel itself maintains a permanent IP connection that remains
unchanged as you move from tower to tower. In such a concept, I could
understand each tunnel (one to the data APN, one to the IMS/VoLTE APN)
having bandwidth allocations.

But when the text brought up "radio bearer", I got confused again sicne
radio implies breaking the spectrum apart, which would reduce LTE
compression efficiency.






Voice channels (FTTH, DOCSIS, VoLTE)

2016-11-20 Thread Jean-Francois Mezei

I need to verify some claims made by incumbents in Canada that VoLTE
data travels on a totally separate channel between the phone and the
antenna.

Does anyone have links to relevant VoLTE documentation that would
provide how VoLTE is provisioned ? I was under the impression that it
was more of an "app" on the phone that used the same IP address given
for access to Internet. Does the phone get a separate IP and possibly
separate VLAN with dedicated bandwidth to ensure voice call quality?

Or are all the performance tricks done on land beyond the antenna once
the packets are identified as VoLTE, but the phone itself just treats
them as a normal app ?

I know that for FTTH, there is a separate "channel" where the "POTS"
emulation can be provided with its own dedicated IP and bandwidth.

Would DOCSIS be the same as FTTH, with the cableco voice service riding
isnide the same DOCSIS bandwidth but with pre-allocated bandwidth, or do
they allocate separate NTSC channels with a totally separate data pipe ?

(in which case, in systems with only 42mhz of uplink frequencies, the
voice would have its own NTSC channel on uplink?




Re: Here we go again.

2016-11-12 Thread Jean-Francois Mezei
On 2016-11-09 17:54, William Herrin wrote:

> I think this discussion is premature. We can hypothesize any number of
> evils from the President Elect but until someone introduces a bill we
> can only tilt at windmills.


The president elect chose Mr Eisenach to help fill jobs in FCC and other
telecom areas of govt.

Mr Eisenach is a regular "expert witness" hired by Telus in Canada at
CRTC hearing. In last week's CRTC hearing on net neutrality, he held his
very standard pro-incumbent stance against it. And the same arguments
will be used to convicne the USA to drop net neutrality to help foster
"dynamic competition" (euphemism for "unregulated duopoly").

It is a given that discussion on banning zero rating of content in USA
will not happen.

It is highly possible that Title II reclassification of ISPs will be
revoked.

Ted Cruz tried to block the IANA re-organisation.

So the direction of where things will be going is fairly clear.

AT&T announced zero rated video (aka T0-Mobile with throttling) just
after the election so they are quite confident this won't be blocked by
FCC.  (although merger with Time Warner may be blocked).





Re: Net Neutrality in Canada

2016-10-30 Thread Jean-Francois Mezei
On 2016-10-30 14:20, Rod Beck wrote:
> Hi Jean,
> 
> 
> What is the status of net neutrality in Canada?


The Telecom Act has had a clasue against undue
preference/discrimination, as well as a "cannot control content", but
both have loopholes. (27(2) , a carrier can argue a
preference/discrimination is not "undue", and for 36 (control of
content), exemptions can be granted by CRTC.

The 2009 ITMP framework was more about throttling and treating packets
differently.

In 2010, the CRTC decided to include wireless services into the ITMP
framework, treating them as ISPs.


 But since then, incumbents have begun to zero rate stuff and there were
2 challenges. In 2013 (decided in 2015), Bell Canada was challenged for
zero rating its own TV service on its own wireless service.  CRTC
decided Bell couldn't do that, but Bell went to Federal Court of Appeal,
arguing its MobileTV offering was covered under the Broadcasting Act and
not Telecom. Federal Court sided with CRTC, confirming that the content
may have been Broadcasting but it was delivered over telecom.

Despite this, Vidéotron launched Zero Rating for music in August 2015,
and instead of deciding on this the same way it did for Bell, the CRTC
decided to launch a wider public consultation on whether zero rating
should be allowed or not.

The hearing that will happen this week is a continuation of a process
which saw 2 rounds of submissions as well as 2 interrogatories and
included the record of the Vidéotron process from 2015. In a couple of
weeks we have final replies and CRTC will take 4-6 months to rule on matter.

Competition Bureau basically says that zero rating is OK unless the
contrent being zero rated is owned by the ISP's organisation. Consumer
groups state it isn't OK, and incumbents state it is OK and that there
should simply be individual challenges whenc onsumers feel one package
abuses 27(2) or 36.

As side note: Telus hires Eisenach lobbyist to write pro-incumbent
reports. He was also hired by the Trump campaign. Not sure if he will
appear this week.


FYI: Net Neutrality in Canada

2016-10-30 Thread Jean-Francois Mezei
This is a heads up, the CRTC (Canada's FCC) is holding a week long
hearing on net neutrality in Canada ("differential pricing" is the used).

Canada has had its "ITMP" (Internet Traffic Management Practices) policy
since 2009 which deals with unfair throttling, and now, we are arguing
on zero rating and sponsored content stuff).


It will be broadcasted at http://www.cpac.ca  (at bottom of home page
there should be a selection for the CRTC hearing). Note: you can choose
between english, french of floor (untranslated).

Days generally start at 09:00. Can end at any time.

Either @CRTCeng or @CRTChearings will be tweeting links to presentations
as each presentation begins.

hashtag: #CRTC #Diffpricing


Facebook did not wish to appear but was "invited". Last time the CRTC
did that, it was with Netflix and Google and sparks flew ("you don't
regulate us, we don't have to answer"). (It appears on Tuesday right
after me, so they should be roughly ~ 10:30 or 11:00.)

The agenda:
http://www.crtc.gc.ca/Telecom/eng/HEARINGS/2016/ag31_10.htm

The original Notice of Consultation:
> http://crtc.gc.ca/eng/archive/2016/2016-192.htm?_ga=1.136052530.24154879.1433393531

And the record of the consultation:
> https://services.crtc.gc.ca/pub/instances-proceedings/Default-defaut.aspx?EN=2016-192&Lang=eng&_ga=1.136052530.24154879.1433393531

Note: this started with a different proceeding in September 2015 2 parts
1 filings against Vidéotron who started zero rated music on its wireless
service for music services that Vidéotron approved/selected.
> https://services.crtc.gc.ca/pub/instances-proceedings/Default-Defaut.aspx?lang=eng&YA=2015&S=C&PA=t&PT=pt1&PST=a#201510735



Re: Spitballing IoT Security

2016-10-29 Thread Jean-Francois Mezei
On 2016-10-29 14:07, Eric S. Raymond wrote:

> You don't build or hire a botnet on Mirai's scale with pocket change.
> And the M.O. doesn't fit a criminal organization - no ransom demand,
> no attempt to steal data.

it is wrong to underestimate script kiddies and open source code. It is
wrong to underestimate a community that shares their own experiences
with different devices. One contributes default password for brand X
camera, one gives the defaults for brand Y router etc.

Imagine someone writes code for university project to scan the network
for improperly protected devices. That code, while designed as a
security audit, could be integrated into something far nastier.

At the end of the day, you may have plenty of open source information
available to assemble this into something like Mirai.


Yeah, there may be more sinister forces out there. The DYN attack may
have been a "demo" of capabilities that will be part of
threats/balckmail against other large players on the Internet.




> everybody else on the InfoSec side I've spoken with is thinking - the
> People's Liberation Army is the top suspect, with the Russian FSB
> operating through proxies in Bulgaria or Romania as a fairly distant
> second.

Or some guy in Arkansas starting a new blackmail/extortion business,
hoping to cash in on the software he put together.

And if we're gonna talk conspiracies, include Trump. he publishes a
"policy" on cyber attacks on a day, a couple days later a major cyber
attack happens. Coincidence ? :-)


I think the focus should be on preventing such attacks, and reducing
their impacts when they happen and improving traceability tools as they
happen. Speculating on who is reponsible doesn't do much to proect the
internet against such attacks.




Re: Spitballing IoT Security

2016-10-26 Thread Jean-Francois Mezei
On 2016-10-26 18:02, Ronald F. Guilmette wrote:

> http://p.globalsources.com/IMAGES/PDT/BIG/053/B1088622053.jpg
> 
> i.e. a multitude of wall plates in every room, each one bristling with a
> multitude of RJ11 sockets into which all manner of shiny new IoT things
> will be directly plugged, thence to be issued their own IPv6 addresses

You still need to have a SOHO router, which could simply block any
incoming calls unless a port has been opened for a specific IP address.
(or UPnP for computers).

> P.S.  As noted in my prior post, the proplem of regulating IoT devices to
> insure that they do not exceed their reasonably expected operational limits,
> vis-a-vis outbound bandwidth usage

A camera showing the baby in 4K resolution along witgh sounds of him
crying on dolby surround to the mother who is at work would likely
saturate upload just as much as the virus sending DNS requests. This
falls into the tonne of feathers weighting as much as a tonne of lead
category.






Re: Spitballing IoT Security

2016-10-26 Thread Jean-Francois Mezei
On 2016-10-26 16:58, Mark Andrews wrote:
>
> Actually things have changed a lot in a positive direction.
> 
> * Router manufactures are using device specific passwords.
> * Microsoft, Apple, Linux and *BSD issue regular fixes for their
>   products and users do intall them.
> * My smart TV has automatic updates available and turned on.
> * Other products do the same.

My smart TV not only hasn't gotten updates in years, but Sharp has
stopped selling TVs in Canada. (not sure if they still sell TVs elsewhere).

When manufacturers provide a 2 year support on a device that will last
10 years, it is a problem which is why they really need to get it right
when product is released and not rely on patches.


With regards to liability. Good luck suing a chinese outfit that no
longer exists.

And pray tell, who gets to pay the millions of dollars of lawyer fees it
will cost to sue that bankrupt company with no money ?





Re: Spitballing IoT Security

2016-10-26 Thread Jean-Francois Mezei
re: having gadgets certified (aka UL/CSA for electric stuff).

Devil is in the details. Who would certify it ? And who would set the
standards for certification?

How fast would those standards change? updated with each new attack?
Would standards update require agreement of multiple parties who rarely
agree?

Consider vendor X who starts to develop product based on standards
available in Oct 2016, but by the time he gets to market, standards have
changed and his device no longer conforms?

One of the beauties of the Internet is the freedom to innovate while
keeping to the core basic IP packet delivery. Start to regulate it or
add red tape and you start to hinder innovation.

Perhaps the RFC mechanism to define best practices for standalone "IoT"
devices might be a better mechanism.  Those who build IP stacks to be
used wholesale by gadget manufacturers could adhere to that RFC so that
end products en up using a proper IP stack that doesn't easily allow the
device to be "upgraded" to serve Dr Evil's botnet designed to take over
the world.


Re: Spitballing IoT Security

2016-10-26 Thread Jean-Francois Mezei
While I agree that fixing home routers is the best approach, something
bugs me.

If an IoT vendor doesn't even know that its devices have telnet or ssh
enabled by default (and hence, no management interface to change
passwords)  and only focuses on the web interface it has added , then
how come the kernel would be "UPnP" the telnet port to tell the router
to send inbound telnet to that device ?

And how do routers deal with multiple cameras each sending a "send port
23 requests to me" ?

I can understand a computer sending a UPnP request when you start a game
to tell router to forward inbound calls to a certain port to that
computer/app.  But for IoT devices that are on all the time, there
should be static setup, not UPnP.



Re: Spitballing IoT Security

2016-10-25 Thread Jean-Francois Mezei
On 2016-10-25 04:10, Ronald F. Guilmette wrote:

> If all of the *&^%$# damn stupid vacation pet feeders had originally shipped
> with outbound rate limits hard-coded in the kernel, maybe this could have
> been avoided.


I view this differently.

The problem is in allowing inbound connections and going as far as doing
UPnP to tell the CPE router to open a inbound door to let hackers loging
to that IoT  pet feeder to turn it into an agressive DNS destroyer.

Then again, you need to have the owner access the pet feeder from the
remote beach to feed the dog.

One way around this is for the pet feeder to initiate outbound
connection to a central server, and have the pet onwer connect to that
server to ask the server to send command to his pet feeder to feed the dog.

This way, there need not be any inbound connection to the pet feeder.




Re: Death of the Internet, Film at 11

2016-10-24 Thread Jean-Francois Mezei
Dumb question:

If some camera, vaccum cleaner, toothbrush or refrigirator is behind
NAT, can it do IP spoofing ?  Won't the "from" address be replaced by
the CPE router with the proper IP address assigned to that customer so
that on the Internet itself, that packet will travel with a real IP
routable back to the CPE ?

Could mobile phones become a source of such attacks ? Depending on
subscription, many are given actiual internet IPs and not NATted, so
they could theoretically send packets with spoofed IPs. (would likely
require rooted android phones, and how many of those are there ?)


Second dumb question:

If the number of infected devices in eastern USA is insufficient to have
caused that DDoS, can one infer that the attack used an actual IP
address instead of the anycast one in order to target the the easter USA
hosts irrespective of the location of the infected device ?

Could one operate such a host with the "real" IP address in a subnet
that has its own BGP announcement, and when there is an attack, one
would change the real IP to a different IP address in a different
subnet, and drop the route announcement for the first subnet (making
those attack packets unroutable at the origin). Is that a viable counter
measure ?





Re: Death of the Internet, Film at 11

2016-10-23 Thread Jean-Francois Mezei
Question:

For something like Mirai and others, there appears to be a timer that
starts the attack at a certain day/time (with unknown amount of time to
distribute the software to any/all infectable devices prior to attack).

Do these generally have a timer to also stop the attack and go dormant
awaiting instructions from its master ? or do they continue to send
those packets forever ?

If the attack is made using perfectly formed, legitimate DNS packlets
(or HTTP requests or whetever), can temporary mitigation measures
continue forever even if they block legitimate requests ?

Or is it general practioce for hackers to have short duration attacks to
reduce the time available to track them down ? (similar to old movies
where one had to hangup before the 2 minutes it took for police to trace
a phone call).




Re: Death of the Internet, Film at 11

2016-10-23 Thread Jean-Francois Mezei


A bit tidbits of information from:

> http://www.networkworld.com/article/3134035/chinese-firm-admits-its-hacked-products-were-behind-fridays-massive-ddos-attack.html

Chinese firm admits its hacked products were behind Friday's massive
DDOS attack

Hangzhou Xiongmai Technology, a vendor behind DVRs and
internet-connected cameras, said on Sunday that security vulnerabilities
involving weak default passwords in its products were partly to blame.
...
Because these devices have weak default passwords and are easy to
infect, Mirai has been found spreading to at least 500,000 devices,
according to internet backbone provider Level 3 Communications.
...
Xiongmai says it patched the flaws with its products in September 2015
and its devices now ask the customer to change the default password when
used for the first time. But products running older versions of the
firmware are still vulnerable.

To stop the Mirai malware, Xiongmai is advising that customers update
their product’s firmware and change the default username and passwords
to them. Customers can also disconnect the products from the internet.


##

Note: the company's web site does not (yet) show a press release.
Appears the information was sent to IDG via email.




Re: Death of the Internet, Film at 11

2016-10-23 Thread Jean-Francois Mezei
On 2016-10-23 15:46, jim deleskie wrote:
> Sure lets sue people because they put too many/bad packets/packets I don't
> like on the internet.  Do you think this will really solve the porblem?  Do
> you think we'll not just all end up with internet prices like US medical
> care prices?


If this were to get to a court of law, would there be proof that
products Axis IP Camera Inc or Panasonic or even Xerox Printers  were
(partly) responsible for the attack ?

Won't they deflect this to trying to find those who hacked their
products ? Won't they deflect this to onwers who did not secure their
networks from inbound telnet ?

And do those units really declare their port 23 to the NAT router via
UPnP ? that is really really stupid.


One problem with consumer goods is lack of documentation and support.
Could years back, I got a very early Smart RG DSL modem specially
modified to work on Bell Canada's non standard VDSL dslams.

No instruction manual, no documentation. I found a number of bugs in the
software, and sent a lengthy email to document them. As an early
adopter, I wanted to help the company fix those before wider deployment.
(and yes, the units have a command line, and from the command line you
can get into a linux shell).

The response I got:  Unless you sign a contract with one of our
distributors, you cannot report bugs.

Unfortunately, this appears to be widespread with consumer goods vendors
who sell sophisticated devices without documentation or support.


Re: FW: Death of the Internet, Film at 11

2016-10-22 Thread Jean-Francois Mezei
On 2016-10-22 19:03, Keith Medcalf wrote:

> This does not follow and is not a natural consequence of sealing the little 
> buggers up so that they cannot affect the Internet

Problem is that many of these gadgets want to be internet connected so
mother at work can check on her kids at home, start the cooking, raise
thermostat etc.

The problem is that as a novelty, people are quick to adopt, but don't
think about making their homes vulnerable to attack. (consider an
internet connected door lock)


Re: Death of the Internet, Film at 11

2016-10-22 Thread Jean-Francois Mezei
On 2016-10-22 18:35, Ray Van Dolson wrote:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__hub.dyn.com_dyn-2Dblog_dyn-2Dstatement-2Don-2D10-2D21-2D2016-2Dddos-2Dattack&d=DQIBAg&c=n6-cguzQvX_tUIrZOS_4Og&r=r4NBNYp4yEcJxC11Po5I-w&m=iGvkbfzRJPqKO1A6YGa-c1m0RBLNkRk03hCjvVGTH3k&s=bScBNFncB3kt_cG0L3iys0mfXBmwwUR7A8rIDmi94D4&e=
>  

Thanks for the link.

10s of millons of IP addresses. Is it realistic to have 10s of millions
of infected devices ? Or is that the dense smoke that points to IP
spoofing ?

re: newspaper reports: how did Flashpoint obtain enough details, while
attack was ongoing to be able to draw conclusions told to the media ? Or
was it educated speculation ?

Obviously, Dyn had packet contents to look at and range of IPs being
used etc. Would such a company typically release that info to a trusted
investigator "as it happens" ? (would Flashpoint be such an outfit ?)

Did the attack generate valid DNS queries (overwhelm the servers) or
flood the links with long "random" UDP packets (overwhel the links).


While I can understand that mitigation methods can be seen as
"proprietary", releasing info on the specifics of the attack would help
any/all neteowkrs and data centres better protect themselves.

Assuming hackers don't talk to each others in the 21st century is silly.
They already know how this was done, yet the victims typically remain
silent for fear of educating the hackers for more attacks.


Re: Death of the Internet, Film at 11

2016-10-22 Thread Jean-Francois Mezei
Generic question:

The media seems to have concluded it was an "internet of things" that
caused this DDoS.

I have not seen any evidence of this. Has this been published by an
authoritative source or is it just assumed?

Has the type of device involved been identified?

I am curious on how some hacker in basement with his TRS80 or Commodore
Pet would be able to reach "bilions" of these devices to reprogram them.
 Vast majority of homes are behind NAT, which means that an incoming
packet has very little chance of reaching the IoT gizmo.

I amn guessing/hoping such devices have been identified and some
homweoners contacted ans asked to volunteer their device for forensic
analysis of where the attack came from ?

Is it more plausible that those devices were "hacked" in the OEM
firmware and sold with the "virus" built-in ? That would explain the
widespread attack.

Also, in cases such as this one, while the target has managed to
mitigate the attack, how long would such an attack typically continue
and require blocking ?

Since the attack seemed focused on eastern USA DNS servers, would it be
fair to assume that the attacks came mostly from the same region (aka:
devices installed in eastern USA) ? (since anycast would point them to
that).

OPr did the attack use actual IP addresses instead of the unicast ones
to specifically target servers ?



BTW, normally, if you change the "web" password on a "device", it would
also change telnet/SSH/ftp passwords.


Re: Dyn DDoS this AM?

2016-10-21 Thread Jean-Francois Mezei
On 2016-10-21 18:45, david raistrick wrote:

> switch too..).   setting TTLs that make sense for a design that supports
> change is also easy.

Cuts both ways. Had Twitter had TTLs of say 7 days, vast majority
wouldn't notice an outage of a few hours because their local cache wa
still valid.

It does prevent one from reacting quickly to emergencies.


Re: Lawsuits for falsyfying DNS responses ?

2016-09-15 Thread Jean-Francois Mezei
On 2016-09-15 16:03, Owen DeLong wrote:

> Please explain to me how one modifies a request or response without
> managing to “control the content” or “influence the meaning or purpose”?
> 
> Blocking a request or simply failing to answer MIGHT be within the law,
> but returning a false record certainly seems to me that it would run afoul
> of the law cited.

Blocking would also be a form of control.  Because Section 36 has a
"unless authorized by CRTC" escape clause, one has to show to the CRTC
that granting permission would be bad.

Since court proceedings have already begun, it is likely the CRTC will
be involved in court, at which point, the more evidence they have, the
more chances they have of arguing against the QC loterry censorship.



Re: "Defensive" BGP hijacking?

2016-09-14 Thread Jean-Francois Mezei
I got to think about this (dangerous thing :-(

Ideally, law enforcement should have the smarts and tools to get
involved in DDoS and other similar situations and have the power to
compell upstream provider(s) to shut service to a suspect.

The current situation appears to be more of a wild-west situation where
everyone takes the law into their own hands. It sort of works but
everyone knows this lead lead to abuses.

If you start to tolerate falsifying BGP, it will likely lead to regular
abuses (including intelligence agencies who stad to gain by redirecting
traffic to their servers) as well as corporate spies etc. So mechanisms
to enforce 0 tolerance are perhaps necessary, even if this means that a
few legitimate BGP tricks to save customers from a failing ISP will no
longer work.



Falsifying BGP can be done by one person without any sanity checks.
There is no check for evidence or whether this action is warranted. On
the other hand, there is a sanity check if you have to convince an
upstream provider to cut access to one of their customers.











Re: Lawsuits for falsyfying DNS responses ?

2016-09-13 Thread Jean-Francois Mezei
On 2016-09-13 03:42, LHC wrote:
> I believe that the CRTC has rules against censorship - meaning that 
> Videotron, Bell etcetera have a choice between following the CRTC code or the 
> provincial law (following one = sanctions from the other), rendering internet 
> service provision to Québec impossible without being a dialup provider from 
> out-of-province.


Canada's Telecom Act (*) dates from 1993, which predates the Internet
being a primary transporter that drives the economy.

The clause being looked at by the CRTC is 36:

Content of Messages

36 Except where the Commission approves otherwise, a Canadian carrier
shall not control the content or influence the meaning or purpose of
telecommunications carried by it for the public.

There is not explicit clause about a carrier not modyfying content or
blocking access, so one has to frame an issue to fit existing clauses.

(For instance, network neutrality is driven mostly by 27(2)

(2) No Canadian carrier shall, in relation to the provision of a
telecommunications service or the charging of a rate for it, unjustly
discriminate or give an undue or unreasonable preference toward any
person, including itself, or subject any person to an undue or
unreasonable disadvantage.


The CRTC asked for supporting evidence to allow it to reach a conclusion
that the Québec plan would force ISPs to breach the federal
telecommunication law.

Because so far, only lawyers have been involved, they have not
understood the implications of forcing ISPs to give false DNS answers
which goes beyond just blocking packets.

(For instance, most ISPs will block packets destined to an external port
25 to reduce spam being emitted by infected customers, but they allow
any/all email to be sent via their own servers. There is an element of
controlling content but was never challenged.

Because Section 36 allows the CRTC to approcve some control of content,
it is important to show that the type of control being requested by the
QC government show never be allowed because it is far worse than just
blocking port 25.



(*) http://laws-lois.justice.gc.ca/eng/acts/T-3.4/FullText.html


Re: "Defensive" BGP hijacking?

2016-09-12 Thread Jean-Francois Mezei
On 2016-09-12 14:15, valdis.kletni...@vt.edu wrote:

> I don't see "hijacking" in your description of the iStop case - it appears
> to have been fully coordinated and with permission.


While I am not sure about fully coordinated and with permission, it is
an example where it was a desirable outcome to maintain service to
customers who would otherwise have have been left without service.

I pointed this as an example where "highjacking" can sometimes be
desirable. An automated system would likekely block such announcements
from ISP3 about ISP1's IP blocks pointing to ISP2's routers as it could
be seen as highly suspect.

Then again, with many mergers and acquisitions, this type or arrangement
may be common as acquiring ISP1 may start to make BGP announcements of
ISP2's IPs before those IPs have had time to be transfered.




Re: "Defensive" BGP hijacking?

2016-09-12 Thread Jean-Francois Mezei
On 2016-09-12 14:14, Hugo Slabbert wrote:

> Was this all done at iStop's request and with their full support?

When iStop's router stopped making BGP announcements to the world
(because its last transit link was cut), and ISP3 highjacked the IP
blocks and made BGP announcements pointing to ISP2, I don't think there
was much of iStop left to complain, and it was to the benefit of end
users, so this highjacking was not nefarious.

Either ISP2 was asleep at the switch and let this happen, or perhaps
they had a deal ith iStop that they would not do BGP until block of IPs
was transfered, so they got a friend at ISP3 to do the deed for them.

The transfer of IP to ISP2 happened shortly after that day, after which
ISP2 did the proper BGP announcements for IPs now assigned to it.




Re: "Defensive" BGP hijacking?

2016-09-12 Thread Jean-Francois Mezei
On 2016-09-11 16:54, Hugo Slabbert wrote:
> Hopefully this is operational enough, though obviously leaning more towards 
> the policy side of things:
> 
> What does nanog think about a DDoS scrubber hijacking a network "for 
> defensive purposes"?


Different spin but still "highjacking":

Many moons ago, iStop, a small ISP in Canada saw its services from Bell
Canada (access to last mile) cut.  However, its core network and transit
was still functional for a number of months.

ISP2 quickly offered to rescue the stranded customers. Once registred
with ISP2, a customer would see the DSL signal re-instated by Bell (now
paid by ISP2) but would continue to be handed IPs that belonged to iStop.

ISP2 made use of the continuing transit capacity from the iStop router
which therefore continued to make BGP announcements for the iStop IP
blocks (and the iStop router then just sent everythingt o ISP2's router
for distribution to end users). During this time, the iStop IP blocks
continued to belong to iStop from ARIn's point of view.

Eventually the transit to the iStop router stopped. That day, former
iStop customers now on ISP2 saw their access to internet essentially
killed. At that point, the iStop IP blocks still had not been transfered
to ISP2.

To save the day, ISP3 kicked in and started to make BGP annoucements for
iStop IPs and redirected the traffic to ISP2.

At that point, ISP3 hijacked iStop's IPs, but it was done to help the
situation, not to steal traffic or anything. (In fact, I think the GBP
announcements from ISP3 pointed to ISP2 routers).

Eventually, the iStop IP blocks was transfered to ISP2 which was then
legally able to do the BGP announcements for those IPs.

So there are some cases where BGP hijacking may be desirable. I guess
this is where judgement kicks in.




Lawsuits for falsyfying DNS responses ?

2016-09-12 Thread Jean-Francois Mezei
As many may know, the province of Québec has passed a law to protect the
interests of its lottery corporation.

To do so, it will provide ISPs with list of web sites to block (aka:
only allow its own gambing web site).

There is an opportunity to comment this week in which I will submit.

(I've gathered many arguments over the past little while already). But
have a specific question today:

Are there examples of an ISP getting sued because it redirected traffic
that should have gone to original site ?

For instance, user asks for www.google.com and ISP's DNS responds with
an IP that points to a bing server?

If the risk of a lawsuit is real, then it brings new dimension to
arguments already made agains that (stupiod) Québec law.

(And it also creates interesting issues for DNS servers from companies
such as Google which may have a anycast server located in Québec but are
not considered an ISP and won't receive those documenst from the gov
with list of websites to block.



Re: NIST NTP servers

2016-05-12 Thread Jean-Francois Mezei
On 2016-05-11 10:30, Mel Beckman wrote:

> Read deeper into the thread and you'll find where I sourced inexpensive 
> RF-based NTP servers using CDMA, GSM, and even WWV. 

For shortwave, you would need to calculate propagation delay between
transmitter and receiver. (does signal reach via line of sight, bounce
against ionosphere ?).

Since CDMA is dead outside the USA and drying in USA, I wouldn't rely on
that.  If GSM towers rely on a GPS receiver on the tower and those
towers are near enough to your location (< 30km), then chances are that
blocked GPS signals at your location would also jam the signals at the
GSM antenna.

And if you are setup to be totally autonomous in case of power failures,
you need to know whether the GSM antenna you are relying on is also on
permanent power backup or only has autonomy of a few hours.


Re: NIST NTP servers

2016-05-12 Thread Jean-Francois Mezei
On 2016-05-10 10:59, Stephane Bortzmeyer wrote:

> Yes, but they may switch it off for civilian use (by going encrypted,
> for instance) at any time, if it is better for *their* operations.


In the days of selected availability (GPS precision reduced on purpose),
the time signal was still very accurate from the point of view of using
it as a time source for computers.

When Clinton lifted SA, the military reserved the right to re-instate
it, and stated that it reserves the right to kill the civilian signal
outside the USA.

The EU considered launching their own constellation to counter the US
possibiliy of a shutdown.

Russia launched Glonass and eliminated the need for EU to launch their
own.  With Glonass now fairly common in GPS receivers, the USA can't
unilaterally shut it down anymore.

A satellite that is visible from Syria couldn't shutdown its signal
without affecting a WHOLE lot of other areas.  It is more likely that
the USA would just jam the frequencies from a plane over Syria or some
other means to geographically block those frequencies.

Today, if someone were to jam the GPS signal in an areas in USA, you'd
likely hear about large number of car accidents in the news before
noticing your systems canMt get time from the GPS-NTP and went to a
backup ip address (nist etc).


Standards for last mile performance

2016-04-30 Thread Jean-Francois Mezei

The CRTC hearing went well (thanks for all your help).

One of the unanswered questions was how to set performance standards for
the last mile to ensure people get advertised speeds (within reason).

I had asked the question about contention ratio and it appears there is
no proper way to set such a moving target as a regulatory standard.

During the hearing, someone suggested that advertised speeds be
achievable 80% of the time. (chairman then asked if "time" was 24 hours,
or just the time you needed to use the internet (aka: peak).

Out of curiosity, could such a thing be measured by the last mile operator ?


There is the easy answer of synch. For DSL, non delivery of advertised
speed is easy since that metric is on the modem statistics. However, for
fixed wireless, would a customer too far from tower see a lower synch
rate or would he just see poor performance due to lots of retransmits ?


So some generic questions:

What are the different ways used to determine if the last mile is
congested and needs to be upgraded ?

>From the network operator's point of view, would it not be looking at 5
minute throughput samples and trigger upgrades when it sees throughput
reaching  x% of last mile segment capacity for more than X minutes per
day ?

What other means do network admins use to monitor when it is time to
upgrade shared last mile segments such as coax or fixed wireless ?

from a policy point of view, is it possible to set the same standard for
different technologies or should each (dsl, coax, fixed wireless and
satellite) have they own standards and methods of measurements before of
intrinsic differences in how they work ?


In the case of Rogers (canadian cableco), the CRTC record shows that
they trigger node split process when capacity reaches 60%. This is
because it takes them so long to do all the paperwork, committees etc
that by the time the node split is done, that segment has grown to about
75% utilisation.

Would that be a sound basis to set a policy ?

For shared last mile, would different technologies have similar
thresholds that trigger the need for upgrades or would coax start to
degrade at 75% whereas fixed wireless or satellite  start to degrade at
lower/higher number ?

For FTTP, while likely not a big problem yet, would similar number apply
when the ~2gbps download and ~1gbps upload start to get filled by the 32
homes served ?


Re: DOCSIS 3.1 upstream

2016-04-20 Thread Jean-Francois Mezei
On 2016-04-20 13:09, Rob Seastrom wrote:

> Going to D3.1 in a meaningful way means migrating to either a mid-split at 85 
> MHz or a high split at 200 MHz 

Thanks. This is what I expected. But in the past, the canadian cablecos
had argued that removing the 42mhz upstream limitation was a huge
endeavour (they have to convicne CRTC to keep wholesale rates up, so
create artificial scarcity by claiming that replacing all those 42mhz
repeaters would cost a fortune, so they have to do node splits instead.

Arguing at CRTC is all about finding out what incumbent statements are
just spin and which are true.

Thanks for the links as well.é

> RFoG is its own kettle of fish.  Getting more than one channel on upstream 
> for RFoG is hard. 

But they can allocate a single very big channel, right ?  Or did you
mean a single traditional NTSC 6mhz channel ?





  1   2   3   >