BOTNET reference involving oscilloscope

[Howard C. Berkowitz]

Happy after-Thanksgiving to the USAians still digesting turkey. Indeed, I
can give thanks to the NANOG community for existing; I've gained a great
deal from it and want to keep giving back.

I'm writing a Wikipedia article on the overall topic of swarming, which, of
course, included DDoS. Unfortunately, I can't remember the title or author
of a presentation that either was at NANOG or a Cisco security event.
Precise, huh? :-<

Anyway, I'm hoping someone will remember it and can give a URL. It was from
a network security group at a European physics lab, had very good economic
analysis, and one of its more powerful example is that an oscilloscope
running Windows -- which no one thought of in applying security patches --
was the place the malware hid while the security admins were scrubbing every
obvious computer.

Anyone happen to know where I can find it?

RE: Why do we use facilities with EPO's?

[Howard C. Berkowitz]

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
Warren Kumari
Sent: Thursday, July 26, 2007 12:03 PM
To: [EMAIL PROTECTED]
Cc: Roy; nanog@merit.edu
Subject: Re: Why do we use facilities with EPO's?



On Jul 26, 2007, at 12:16 AM, [EMAIL PROTECTED] wrote:


Sometime I really need to write down all of the funny things that  
have happened over the years... Actually, if anyone has other, random  
funny (?!) stories, pass them along and I'll make a compilation


[Howard C. Berkowitz] 

While working at a distinguished university with a religious affiliation, I
learned, as did one of the priest-biologists, not to refer to a piece of
instrumentation as possessed. While one of the priest-theologians meant
well, we learned what happened when holy water is sprinkled into the high
voltage supply of a gas chromatograph. Beckman Instruments was so amused
they didn't charge for equipment abuse not under maintenance contract.

Re: SaidCom disconnected by Level 3 (former Telcove property)

[Howard C. Berkowitz]

Joe Abley wrote:
>
>
> On 16-Mar-2007, at 19:56, Wil Schultz wrote:
>
>> Almost ALL?
>
> Surely all those except those who are competing with you for the same
> customers should multi-home. :-)

To the NANOG T-shirt Committee: Please consider this as the slogan for the
next design.

RE: passports for NANOG-39, Toronto

[Howard C. Berkowitz]

particularly if you are in the DC area, call your congressman's district
(usually) office and ask them to send the Passport Office a "congressional
courtesy" request. In practice, this means that you don't stand in line, but
go upstairs to the diplomatic processing area, and, with proper documents
and photos, you'll probably have the passport in under an hour. 
I believe there is also a priority program for cities that have Passport
Office branches. Just one of the perks of incumbents.

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
> Robert E. Seastrom
> Sent: Wednesday, October 25, 2006 10:26 PM
> To: [EMAIL PROTECTED]
> Subject: passports for NANOG-39, Toronto
> 
> 
> 
> You may have heard that the US and Canada are going to start requiring
> passports for air travel between them beginning "soon".  That date is
> currently set as 8 Jan 2007, which is before February NANOG.  MERIT
> has noted this on the web site, but a cursory check of my list
> archives didn't turn up mention of it (sorry if I overlooked it; the
> last couple of weeks have been hectic), so I figured I'd include the
> pointer:
> 
>http://www.nanog.org/mtg-0702/passport.html
> 
> as well as a link to the State Department:
> 
>http://travel.state.gov/passport/passport_1738.html
> 
> Normal passport processing is "within six weeks", but that probably
> doesn't take the holiday season into account.  If you don't have a
> passport already and plan to travel from the US to NANOG 39 in
> Toronto, getting on that project sometime in the next month or so
> would allow plenty of spare time.  No reason to pay expedite fees if
> you don't have to.
> 
> ---Rob

Network graphics tools

[Howard C. Berkowitz]

Much of the enterprise market seems wedded to Visio as their network 
graphics tool, which locks them into Windows. Personally, I hate both 
little pictures of equipment and Cisco hockey-puck icons; I much 
prefer things like rectangles saying "7507 STL-1" or "M160 NYC-3".


Assuming you use *NIX platforms (including BSD under Mac OS X), what 
are your preferred tools for network drawings, both for internal and 
external use?  I'd hate to be driven to Windows only because I need 
Visio.

May-June meeting?

[Howard C. Berkowitz]

I've just gotten with a new firm that should let me play at NANOG 
again. While I probably can't meet the March proposal deadline, will 
there be lightning presentations again?


Two areas of interest for lightning or BOF:
Operational aspects of PBX hosting and general movement of ISPs into
telephony

More on emergency communications, especially software that complements
the 911 operations system, where there are little-known North American
standard software packages (DM-Services)

Middle Eastern Exchange Points

[Howard C. Berkowitz]

I know of a Cairo IXP, and possibly one in the UAE.  Is there one in 
Kuwait as yet?

Re: BGP terminology question

[Howard C. Berkowitz]

At 9:44 AM -1000 11/6/05, Randy Bush wrote:
 > A peer should never announce a route it has already announced unless 

 that route is withdrawn.


one of many counterexamples: change in igp will cause change in
med.  any attribute changes, and announcement is required.

e.g., an internal igp oscillation could cause what the op
describes.


For the OP, http://www.ietf.org/rfc/rfc3345.txt

www.usenetabuse.com?

[Howard C. Berkowitz]

I'm assisting in trying to deal with a group of flooders/trolls. One 
remailer directs complaints to www.usenetabuse.com.  Does anyone know 
if this is a legitimate anonymizer abuse desk, or phishing for 
details of exploits?

Re: New Orleans Comm Needs

[Howard C. Berkowitz]

At 7:05 AM -0400 9/3/05, Jerry Dixon wrote:

I don't know how many of you have a Ham license but see
below:

In talking with communication providers in the region they
just now got satellite phones and in process of assessing
and repairing communication lines for New Orleans.  They're
still somewhat challenged with comms including at shelters. 
See the request below from ARRL.


There is an HF net setup and efforts under way by NCS down
there.

-Jerry



For what it's worth for possible antenna alternatives, I was just 
reminded by a military friend that Army and National Guard field 
artillery units normally have a generous supply of good-sized 
balloons for meteorological sensors. While the deploying artillery 
units wouldn't normally take these to a disaster area, they could be 
asked to bring them along if these might be useful in RF 
communications.






http://www.arrl.org


Attention All Amateurs...
 Amateur Radio emergency communication volunteers needed!
(Sep 2, 2005) -- The ARRL now is seeking experienced Amateur
Radio emergency volunteers to help supplement communication
for American Red Cross feeding and sheltering operations in
Mississippi, Alabama and the Florida Panhandle--as many as
200 locations in all. Special consideration will be given to
operators who have successfully completed the ARRL Amateur
Radio Emergency Communications course training (Level I
minimum) to serve as team leaders. These volunteer operators
will help to provide communication and equipment for relief
efforts in the aftermath of Hurricane Katrina. If you're
interested and qualified, please send an e-mail message to
[EMAIL PROTECTED], providing name, call sign, contact
information and any equipment you're willing and able to
take along on a field deployment for an indefinite period.
Volunteers may face hardship conditions without the usual
amenities and will need to provide their own transportation
to the marshaling area.



 Original message 

Date: Fri, 2 Sep 2005 19:59:40 GMT
From: "Fergie (Paul Ferguson)" <[EMAIL PROTECTED]> 
Subject: FCC COORDINATING TECH AID FOR KATRINA DISASTER  
To: nanog@merit.edu



http://www.boingboing.net/2005/09/02/fcc_coordinating_tec.ht

ml


- ferg


--
"Fergie", a.k.a. Paul Ferguson
 Engineering Architecture for the Internet
 [EMAIL PROTECTED] or [EMAIL PROTECTED]
 ferg's tech blog: http://fergdawg.blogspot.com/

Re: A useful oversimplification for network surveillance?

[Howard C. Berkowitz]

At 11:15 AM -0500 8/25/05, sjk wrote:
We use both -- NetFlow gives us trending data which helps us 
identify issues and patterns, Snort allows us to perform a deeper 
analysis -- I don't think you could use one and not the other and 
have effective traffic inspection.


I think we are in agreement. Remember, I was dealing specifically 
with surveillance. Surveillance and deeper analysis are complementary.




 On Thu, 25 Aug 2005, Florian Weimer wrote:




I'd most certainly use an IDS (i.e. SNORT) for this instead of
netfow


Could you provide a use case at the ISP level where an IDS is indeed
superior to NetFlow data collection?

(Take into account that ISPs typically see the effects of new malware
well before the AV companies. 8-)



_
[EMAIL PROTECTED]
http://www.cupacoffee.net

No one can understand the truth until
he drinks of coffee's frothy goodness.
~Sheik Abd-al-Kadir



This .sig must be preserved. I go to refill my cup.

Has anyone ever quantified the relationship between available network 
clue and available caffeine?

Re: A useful oversimplification for network surveillance?

[Howard C. Berkowitz]

At 3:30 PM + 8/25/05, Fergie (Paul Ferguson) wrote:

Howard,

I'd most certainly use an IDS (i.e. SNORT) for this instead of
netflow


My concern is scalability, remembering I'm talking about the 
surveillance level. My preliminary sense is that SNORT is great in a 
sinkhole, but isn't as scalable as a reasonable NetFlow export.




- ferg

-- "Howard C. Berkowitz" <[EMAIL PROTECTED]> wrote:

  NetFlow is the key to analyzing traffic patterns outside the router,
  looking for DDoS signatures when known, and for traffic anomalies that
  may become DDoS.

A useful oversimplification for network surveillance?

[Howard C. Berkowitz]

I'm developing some guidance for ISP surveillance for infrastructure 
attacks, and my increasing impression is that for other than the 
expert level, there may be some useful simplifications of the 
applicability of tools. Remember that I am speaking of surveillance 
here, not the detailed analysis in a sinkhole.  Perhaps this could be 
the basis of some security architecture presentations/tutorials at 
NANOG.


Let me put up the following strawmen and invite people with flaming 
torches to go for them, with the caveat that these simplifications 
are for an introduction to the topic.


 NetFlow is the key to analyzing traffic patterns outside the router,
 looking for DDoS signatures when known, and for traffic anomalies that
 may become DDoS.

 SNMP is the key to analyzing the effect of exploits on network elements.
 For example, NetFlow might tell you there is a flood directed at TCP
 port 179, but your router may implement rate-limiting/policing such
 that the control processor doesn't see this flood and processor
 utilization stays within reasonable ranges.

 Syslog and SNMP traps focus on physical events by people (e.g.,
 reconfiguration), physical problems ranging from temperature alarms
 to router and interface shutdown, and exploits against security
 mechanisms.  Some of this asynchronous information has undergo
 root cause analysis: the interface you see go down may be perfectly
 fine; the problem is in the medium or distant interface.

Re: open source tools help (contract) in DC area?

[Howard C. Berkowitz]

At 8:27 PM +0200 7/25/05, Brad Knowles wrote:

At 1:12 PM -0400 2005-07-25, Howard C. Berkowitz wrote:


 I need to get some short-term contract help on setting up a lab dealing
 with SP security issues, in the Washington DC area.  Please contact me
 offline if interested. I am the technoid and will pass you on for the
 mercenary aspects.


	I'm not convinced that this is an appropriate on-topic 
posting for NANOG.  It seems to me that you would be much better off 
going through SANS or SAGE to find local groups in the area that 
could be helpful to you.


Actually, the interest is in open-source ISP tools.



	For example, I believe that if you contact the folks at 
dc.sage (see www.dc-sage.org), they are more likely to be able to 
help.  I know there are several security and network-knowledgeable 
system administrators in dc.sage.  I imagine that at least one or 
two of them should be consultants/contractors who can help you.



	But it does seem to me that a more targeted search for 
assistance would have been appropriate.


--
Brad Knowles, <[EMAIL PROTECTED]>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755

  SAGE member since 1995.  See <http://www.sage.org/> for more info.

open source tools help (contract) in DC area?

[Howard C. Berkowitz]

I need to get some short-term contract help on setting up a lab 
dealing with SP security issues, in the Washington DC area.  Please 
contact me offline if interested. I am the technoid and will pass you 
on for the mercenary aspects.


ccitraining.net is developing a complex set of network security lab 
exercises involving Cisco routers and switches, Slackware 10.0 LINUX 
servers and workstations, and Windows workstations, the latter to be 
infected with worms as part of running the lab.


We need a *NIX administrator to help us get the appropriate, 
primarily open-source tools installed, running, and documented. Since 
we do not intend to teach the full tool command set, we will need 
shell scripts and/or command files to be piped to a telnet/SSH client 
to let the students access useful tool functions without being fully 
trained in the device. For that reason, we expect the primary 
interface to the tools will be command line, so that the tool control 
can be scripted. Students will use GUI functions only to display 
output from tools, or to access graphic functions in the tools.


Since there are multiple people working on the project in a virtual 
team, at different locations, it is absolutely essential that 
documentation be generated at the start of working with a tool, and 
then to be polished with final parameterization and use 
documentation.  Documentation can be at the level of a couple of man 
pages, but it is essential that other team members can quickly find 
out how to parameterize and invoke the tools. Project managers also 
need to be able to track the status of tool implementation -- we do 
not consider an undocumented tool as installed.


Identified tools include:

   syslogd
   RRD (successor to MRTG)
 MIB objects to be accessed
   Flowscan/Flowtools (successors to cflowd)
   Ethereal

In addition, we will need a number of scripting tools to make 
incremental changes to router, switch and host configurations, as 
well as loading complete executables and images.  We will also need 
Windows control to infect hosts with specific viruses and possibly 
bots, and to restore infected hosts to a stable environment.


Understanding, from the Windows and protocol standpoint, of worms, 
other DDoS, and BOTNETs will be very helpful.  Knowledge of packet 
crafting tools for *NIX, which let us build arbitrary protocol 
packets to be used in attacking hosts and routers, will also be a big 
help.

Clueful security contact at Adelphia?

[Howard C. Berkowitz]

Could someone at Adelphia, who understands there is a difference 
between newsgroup trolling and newsgroup denial of service, contact 
me offline?


Thanks!
--
Howard C. Berkowitz
5012 25th Street South
Arlington VA 22206

(703)998-5819 voice
(703)998-5058  fax (alas, sometimes poorly operated by "helpful" cat)

[OT] Re: what will all you who work for private isp's be doing in a few years?

[Howard C. Berkowitz]

At 1:48 PM -0700 5/12/05, David Barak wrote:
--- Matthew Crocker <[EMAIL PROTECTED]> wrote:

 On May 12, 2005, at 4:23 PM, Jeff Rosowski wrote:
 >
 >
 >> | So imagine a residential area all pulling
 digital video over 
 >> wireless.
 >> | Sound familiar? Ironically close to TV! (yet so
 different)
 >>
 >> You mean like VoIP over dsl ?
 >>
 >
 > I'm looking to setup DSL over VoIP over DSL next.
 
 >

 I'm going for v.90 over VoIP over DSL.  Hopefully
 I'll be able to get 
 a 28.8k connection over my DSL line ;)
One of the vendors from a previous NANOG (IIRC, it was
Pluris, but don't quote me) had a shirt extolling the
benefits of IP over MPLS over ATM over X.25 over
Frame-Relay over MPLS over PPP over Ethernet over HDLC
over SONET. 

everything old is new again :)
What happened to the LLC/SNAP?

Re: Sinkhole Architecture

[Howard C. Berkowitz]

At 1:34 PM + 4/29/05, Christopher L. Morrow wrote:
On Fri, 29 Apr 2005, Howard C. Berkowitz wrote:
 I've seen some Cisco security presentations that include sinkholes
 composed of an ingress and egress router, interconnected with a
 switch. The switch provides access for tools such as packet
 analyzers, IDS, routing analyzers, etc. The multiple routers also
 provide more horsepower for inspection, filtering, and
 overhead-imposing measurements such as NetFlow.
the multiple routers could just be a way to get a MAC to the ingress
router for delivery over the ethernet... a sun/linux/bsd/*unix box might
provide the same function. (please logging, analysis, ids, flow
collection)
The architecture described doesn't have the two routers treating the 
Ethernet as a destination:

 SinkholeIn--->Switch-->SinkholeOut
  |
  |
   analyzers

 I am unclear about the BGP relationship between the two routers,
 which are meant to be treated as one subsystem.  The ingress router
 (with respect to the outside) clearly has to have its BGP isolated
 from the rest of the AS, so it can't be part of the iBGP mesh.
why can't it be part of the ibgp mesh? I'm not sure I see why that would
be BAD, aside from it bouncing under load and affecting all ibgp
neighbors... so, aside from route-churn and neighbor setup/teardown churn
what other reasons?
The most basic is whether I am diverting a maliciously inserted route 
to it from the edge router.

Sinkhole Architecture

[Howard C. Berkowitz]

I've seen some Cisco security presentations that include sinkholes 
composed of an ingress and egress router, interconnected with a 
switch. The switch provides access for tools such as packet 
analyzers, IDS, routing analyzers, etc. The multiple routers also 
provide more horsepower for inspection, filtering, and 
overhead-imposing measurements such as NetFlow.

I am unclear about the BGP relationship between the two routers, 
which are meant to be treated as one subsystem.  The ingress router 
(with respect to the outside) clearly has to have its BGP isolated 
from the rest of the AS, so it can't be part of the iBGP mesh.

My assumption is that the ingress router has to be either a 
confederation AS, or router reflector client, talking to the egress 
router.  The latter is part of the main iBGP mesh, although it could 
be a client in a next hierarchical reflection cluster. Do any of 
these iBGP arrangements impact having the sinkhole ingress with an 
anycast address?

Is this a correct architectural assumption?  Can anyone point me to, 
or provide a representative configuration?

I also wanted to confirm the failure modes under which static ARP 
between the routers is desirable.

Howard

Re: Quantifying risk of waiting vs. upgrading for router vulnerabilities

[Howard C. Berkowitz]

At 1:05 AM -0700 1/31/05, Pete Kruckenberg wrote:
After another long week of dealing with "upgrade now or die"
vulnerabilities, I'm wondering...
Is there data or analysis that would help me quantify the risks of
waiting (while I plan and evaluate and test) vs. doing immediate
software upgrades?
With many router vulnerabilities, exploits are in the wild within 24
hours. But how often are they used, and how often do they cause actual
network outages? There have been several major router vulnerabilities
during the last 2 years which have provided a reasonable data sample to
analyze. Can that data be used to create a more-accurate risk-analysis
model?
The risk of outage is very high (or certain) if I jump into upgrading
routers, and the quicker I do an upgrade, the more likely I am to have
a serious, extended outage. However, this is the only choice I have
absent information other than "every second gives the miscreants more
time to bring the network down."
If I delay doing the upgrade, using that delay to research and test
candidate versions, carefully deploy the upgrade, etc, I reduce the
risk of outage due to bad upgrades, at the expense of increasing the
risk of exploitation.
I'd love to find the "sweet spot" (if only generally, vaguely or by
rule-of-thumb), the theoretical maximum upgrade delay that will most
reduce the risks of upgrade outages while not dramatically increasing
the risks of exploitation outages.
Ideas? Pointers?
Pete.
Pete,
You touch on a broad area where I think there is data relevant to 
network operators, but they aren't aware of it:  clinical medicine, 
more narrowly public health, and specifically epidemiology.  What you 
describe is very much like the situation where there is a disease 
outbreak, and, perhaps only an experimental drug with which to treat 
it. How does one look at the risk versus reward tradeoff?

There are many medical approaches to considering the value of a drug 
or treatment -- this falls into the discipline, as well, of "evidence 
based medicine."  There are assorted metrics for such things as "cost 
per year of life extension", and, more recently, "cost per year of 
quality life extension."  These models include the cost of the 
treatment and both the probability of protection/improvement and of 
adverse effects. Adverse effects can range from a drug having no 
benefit but doing no harm, but precluding the use of a drug known to 
have some, but probably lesser efficacy -- or perhaps much more 
toxicity.  The "clinician" has to assess the probability that the 
software or medical "bug fix" will kill both the bug and the patient.

It may be worthwhile to study the rather fascinating and 
time-sensitive problem faced every year, in coming up with the 
appropriate mixture of influenza substrains for that year's vaccine. 
The process is rather fascinating.  Influenza strains initially 
classify by which of three H and two N factors are present in a given 
virus. There are substrains below, say, H3N2.

In general, the first of the new year's strains start in animals in 
Western China. They may mutate on their way into human form.  There 
is a practical limit on how many strains can be put into the same 
batch of vaccine, and there is a lead time for vaccine production. 
Vaccine specialists, even ignoring things like this season's 
production disaster, have to make an informed guess what to tell the 
manufacturers to prepare, which may or may not match the viral 
strains clinically presenting in flu season.

There really are a number of applications of epidemiology to network 
operational security. In this community, we note the first 
appearances of malware and have informal alerting among NOCs and 
incident response teams, but I am unaware of anyone using the formal 
epidemiological/biostatistical methods of contact/first occurrence 
tracing.  Applying some fairly simple methods to occurrence vs. time 
vs. location, for example, can reveal if there is one source of 
infection that infects one victim at a time, if there is contagion 
(different from infection) from victim to victim, etc.  Indeed, some 
of the current work in early warning of biological warfare attack may 
have useful parallels to recognizing random infection versus an 
intelligently controlled BOTNET DDoS.

Howard

Re: NANOG Changes (and proposal)

[Howard C. Berkowitz]

Hi everyone - apologies for a rather long message, but I wanted to 
bring you up-to-date on some steps the Program Committee and Merit 
have taken to evolve NANOG since our community meeting in Las Vegas. 
*Many thanks* to those of you who attended and gave us feedback - we 
learned a lot and look forward to working with all of you to 
maintain the high standards we have come to expect from NANOG.


Second, the NANOG Program Committee has elected a new chair - thank 
you Steve Feldman!  Steve will now handle speaker communications 
that deal with content, and will make any last-minute decisions 
about what to include on the agenda.

Third, we are creating a new email list, NANOG-futures, to discuss 
NANOG's evolution.  We hope you'll participate - watch for a message 
later today or tomorrow about subscribing and a proposed time-line 
for moving us forward.

In the past, I've suggested (and volunteered for) NANOG to have a 
more extensive publication program, not simply an archive of 
presentation. There are some extremely valuable pages on the NANOG 
website, but I believe there is value to having a slightly more 
formalized publication process.  RIPE and RIPE-NCC have done so for 
some time, with very useful outputs.

It has been suggested that the IETF RFC process can serve, but there 
are problems with that. IETF's process is optimized more for 
developers than operators. It also can be slow, not from controversy 
but simply from administrative process and workload. I'm sure I'm not 
the only author to see a year or two elapse between working group 
consensus and final RFC publication.

Betty, would you see this discussed on NANOG-futures? Is it 
worthwhile to reopen exploratory decisions on the main list?

Router-switch-router "peering module"

[Howard C. Berkowitz]

I'm hunting for some presentations or papers on what I've seen called 
a "peering module", using a router, a L2 switch, and a router in 
series, rather than a single router. Unfortunately, I can't remember 
where I saw the detailed description, and I haven't been able to find 
it in the NANOG archives.

IIRC, the rationale was to spread filtering and rate limiting over a 
set of processors, also to keep the configuations more manageable.

Does anyone have any pointers to detail?  I've seen the topology, but 
not a detailed discussion, in a couple of Cisco presentations.

Re: ISP Policies

[Howard C. Berkowitz]

At 11:04 AM +0530 9/9/04, Tulip Rasputin wrote:
Hi Chris,
Or, you just don't want to send traffic through Bill Manning's ASN because
you dislike his hawiian T-Shirt Policy? There are probably a few hundred
reasosn why you'd avoid an ASN... In general though I'd think that like
Michel said: "It's a pain and its doing something that bgp should do for
you without lots of messing about"
That's why i explicitly asked for some "social/political/etc." 
reasons where an ISP may not want his traffic to traverse some 
particular AS number(s). Something which is beyond BGP to determine 
as of now ! :-)

I believe with the responses that i received both on the list and 
offline, that it is indeed quite normal for ISPs to filter routes 
based on the AS Paths for 'other' reasons. Reasons, quite beyond BGP 
as a protocol to handle! And this can happen, when an ISP doesnt 
want its traffic to traverse some AS(es).

Thanks,
Tulip
IIRC, at some long ago time, there was a Canadian policy, derived 
from a policy on telco transit, that two Canadian providers could not 
use transit through the USA to get to one another. This is long gone.

I can't say if it is still the case, but, at one point, the PRC would 
drop routes that used Taiwan as transit.

Re: who's next?

[Howard C. Berkowitz]

At 12:12 PM -0700 9/8/04, Fred Baker wrote:
At 04:29 PM 09/08/04 +, Paul Vixie wrote:
i guess this is progress.  the press keeps bleating about stopping 
spam from being received -- perhaps if they start paying attention 
to how it gets sent and how many supposedly-legitimate businesses 
profit from the sending, there could be some flattening of the spam 
growth curve.
I think both approaches have value.
Consider this by comparison to the "war against drugs". One line of 
reasoning says "if there is no supply, there will be no market". 
Another line of reasoning says "if there is no demand, there will be 
no market". A third line of reasoning notes that with purveyance of 
such come a multitude of other social ills, and focuses on the 
"businessmen" in the trade: "if there is no way for supply and 
demand to meet, the market will fail."
The solution lies in a combination of the two. If enough spammers 
take enough drugs, they will be unable to spam. Properly propagated 
rumors may variously suggest:

  1. Becoming a spammer will put you into drug rehab at best.
  2. Spammers now become a target for no-knock raids by the Drug
 Enforcement Administration.


Where this gets interesting is with so-called "legitimate spam". At 
least under US law, if you and I have a relationship as buyer and 
seller, the seller has a right to advertise legitimate services and 
products to the buyer. I travel in a vertical direction when I get 
spam from my employer; I have sat down with the designated spammer 
and have been told in detail that as a user of that equipment I am a 
buyer and they have a right to advertise to me, and take pretty 
serious steps to target and not annoy their audience. There is a 
part of me that wants to site in an 18" gun using their building as 
a target; there is another part of me that notes the photography in 
magazines and on billboards and the little jingles that go by on TV 
and the radio, and notices that legitimate advertising is in fact 
treated as (ulp!) legitimate.
And Jerry Springer is a legitimate means of advertising. I will 
confess that after an especially long, exhausting day at an IETF 
plenary, I have searched for something as mindless as Mr. Springer to 
clear my brain for sleep.  To the best of my recollection, I have not 
reached that level of exhaustion at NANOG.

RE: ARIN Comment

[Howard C. Berkowitz]

Perhaps there is a message here that ARIN might pick up on some of 
the work done in the IETF PIER WG.  This did produce RFC 2071 on why 
one wants to renumber (coauthored with Paul Ferguson) and RFC 2072 
(my Router Renumbering Guide).  At the time, no one in the WG really 
wanted to go after the application renumbering issues.

I can barely spell HTML -- I'm just trying to get a basic webpage. 
Clearly, I am not the right person to spearhead an effort, but either 
ARIN, or perhaps the RIRs acting as a team, might do well to put out 
additional guidance on renumbering.  After such guidance has been 
available for some time, the RIRs might reexamine their policies and 
see if what evolves as Best Current Practice should enter the 
policies.

Is there active work on RFC 2050 bis?  I haven't heard anything in a while.
AFAIK, there's been no activity in ARIN CLEW for some time.
--
Howard C. Berkowitz
Chief Technology Officer
  and ARIN representative when I last looked
Gett Communications
5012 25th Street South
Arlington VA 22206
(703)998-5819 voice
(703)998-5058  fax (alas, sometimes poorly operated by "helpful" cat)

Re: The use of .0/.255 addresses.

[Howard C. Berkowitz]

At 10:03 PM -0400 6/26/04, Richard A Steenbergen wrote:
This is what happens when your educational system continues to teach
classful routing as anything other than a HISTORICAL FOOTNOTE
*coughCiscocough*. This is also how you end up with 76k /24s in the global
routing table.
Do you part to help control the ignorant population: whenever you hear
someone say "class [ABC]" in reference to anything other than a historical
allocation, smack them. Hard.
May I take this opportunity to remind people of my Atlanta 1998 
(IIRC) NANOG tutorial on ISP addressing, "Good Providers have No 
Class"?

Re: Can a customer take IP's with them?

[Howard C. Berkowitz]

At 3:29 PM -0400 6/25/04, Eric Gauthier wrote:
 > Only one customer?  There are a couple "consulting" firms in
 particular around here that use arbitrary space on internal
 networks.  Sometimes a currently-dark IP block is configured, so
 "it works for us".  It gets annoying after a while.
The worst one I've seen so far is Ticketmaster... last month.  If you want to
sell tickets through them and connect via the network, they require you to
have a private, backend connection to them and then require you to route
29.2.0.0/15, 29.4.0.0/15, and 29.6.0.0/16 via that connection.
Several third-party health payors, as well as a few HMOs and the 
like, do exactly this sort of thing with medical service providers. 
It makes hospital addressing, at times, rather interesting.

Some of them used the rationalization that if the space wasn't in the 
Internet routing table, it was more secure. To make it worse, a 
couple further expected you to address some of your hosts with their 
bogus address space, and then run transport-mode IPSec to them.

If you have never had a good sized hospital decide you are their new 
ISP (or network manager), it's good to find someone that will write 
prescriptions for legal drugs. On your first site visit, when you 
start discovering some of their addressing oddities, you will want to 
go to the pharmacy and get the scripts filled, to help you get 
through the day.

While newer applications, if anything, go overboard for security, 
some earlier medical applications, especially laboratory 
instrumentation, just send all their data to 255.255.255.255.  I 
asked one of the programmers why they did that, and he said they 
didn't know if somebody might plug in a device that needed the data, 
so they didn't want to be bothered putting in support for it.

You will find there are now an assortment of security and privacy 
laws that the hospital has to support, HIPAA being the best known, 
but also 21CFR11 for clinical trials, DEA electronic prescribing of 
controlled substances, and COPPA for pediatric data. Unfortunately, 
no one has ever decided to harmonize the security requirements for 
the different mandates.  If it helps put things in perspective, the 
legislation enabling recent extensive modifications and additions to 
HIPAA was titled the HIPAA Administrative Simplification Act.  George 
Orwell would have loved it.

Re: Can a customer take IP's with them?

[Howard C. Berkowitz]

At 7:29 PM -0400 6/23/04, Robert Blayzor wrote:
Howard C. Berkowitz wrote:
This would absolutely have to be challenged on cross-examination. 
Were I the attorney, especially if the plaintiff had mentioned 
telephone number portability, I would ask the plaintiff to explain 
what additional work had to be done to the POTS network to 
implement portability. Should the plaintiff start mumbling, I'd 
impugn his credibility, and then ask a bunch of hard questions 
about SS7 (including the TCAP mechanism for portable number 
translation), how IP routing works, how IP routing has no 
authoritative mechanism for global translation, etc.  I'd 
interrogate the customer about DNS and why they weren't able to 
solve their portability requirement with it. I'd look for detailed 
familiarity with RFC 2071 and 2072.

I wouldn't expect the customer to be able to answer many of these. 
As the defendant, I would expect to bring in my own expert witness 
who is very good at explaining these differences, and how the 
telephone and IP routing environments are different.
Apples and Oranges.
My point exactly, that enough explanation will show there is no 
operational or protocol equivalent to number portability.  The 
defendant has to be prepared to shoot down that argument.

There is something called DNS which handles how hosts are "known" 
by.  The whole reason behind DNS is so a user owns a name but 
doesn't matter what "number" they have.
Well, yes.
In the telco world you do not have this option since many businesses 
advertise their telephone number.  (ie: yellow page ads, business 
cards, advertisements, etc.)  When it comes to "the net" IP 
addresses are irrelevant as people are known by name, names which 
are transparently resolved to IP addresses.

The technology exists so that people don't have to bring IP space 
with them.  The routing tables are big enough as it is and the last 
thing we need is a bunch of judges comparing number portability to 
IP space portability.
Again, I don't see how we are in disagreement. What I was describing 
was an approach to getting the judge and/or jury to see they are NOT 
the same thing.

Re: Can a customer take IP's with them?

[Howard C. Berkowitz]

does your contract with your customers state that the space
is non-portable, that they can't take it with them and that
they WILL have to renumber ??  If so then they are asking
the court to change the contract you and they entered, which
i doubt would happen.
the legal risk here is if the court gets it in their head
that IP's are like phone numbers and that they (numbers)
should be portable..
Judge:  "So what exactly is an IP address"
Plantif: "Judge, an IP address is a unique number that is
assigned to each computer on my clients network.  It unique
like a phone number, and like each person in the company has
a unique phone number, they must have a unique IP address"
Judge: "So basicly its like a phone number for computers."
Plantif: "Yes"
This would absolutely have to be challenged on cross-examination. 
Were I the attorney, especially if the plaintiff had mentioned 
telephone number portability, I would ask the plaintiff to explain 
what additional work had to be done to the POTS network to implement 
portability. Should the plaintiff start mumbling, I'd impugn his 
credibility, and then ask a bunch of hard questions about SS7 
(including the TCAP mechanism for portable number translation), how 
IP routing works, how IP routing has no authoritative mechanism for 
global translation, etc.  I'd interrogate the customer about DNS and 
why they weren't able to solve their portability requirement with it. 
I'd look for detailed familiarity with RFC 2071 and 2072.

I wouldn't expect the customer to be able to answer many of these. As 
the defendant, I would expect to bring in my own expert witness who 
is very good at explaining these differences, and how the telephone 
and IP routing environments are different.

also how much space are we talking about here ?? a /19
or a /27 ? or ???   If its smaller than a /20, Hell
let them announce it, most places won't take the route
and their connectivity will be dorked.
You might also advise ARIN that this customer has IP space
that is not being used and that it should be reclaimed
as they didn't renumber from your space.
john brown
On Wed, Jun 23, 2004 at 01:16:48AM -0400, Alex Rubenstein wrote:

 (also sent to nanog)

 Should a customer be allowed to force a carrier to allow them to announce
 non-portable IP space as they see fit to any other carriers of their
 choosing when they are no longer buying service from the original carrier
 [that the space is assigned to]?
 According to ARIN regulations, the space does not "belong" to us but we
 have the right to assign or revoke the space to our customers as we see
 fit. In addition ARIN regulations specifically prohibit us from transferring
 or selling the IP space to another customer (even if we want to).
 NAC has a customer who is leaving NAC. As part of normal procedure (and
 also because the space provided to us by ARIN is non portable), the
 customer has been informed that the IP space used by the customer will not
 be available to be used by the customer subsequent to them leaving us.
 It should be mentioned that the following facts exist, and cannot be
 disputed:
 a) customer has obtained space directly from ARIN over a year ago, but has
 chosen not to renumber from space allocated from us. This was solely their
 choice, and we did not restrict this in any way.
 b) customer is exercising the right not to renew the business agreement,
 and is leaving NAC voluntarily.
 Thus, they are attempting to file for and obtain a temporary restraining
 order (TRO), and ask for the following:
 -- start --
 "NAC shall permit CUSTOMER to continue utilization through any
 carrier or carriers of CUSTOMER's choice of any IP addresses that were
 utilized by, through or on behalf of CUSTOMER under the current agreement
 during the term thereof (the "Prior CUSTOMER Addresses") and shall not
 interfere in any way with the use of the Prior CUSTOMER Addresses,
 > including, but not limited to:
(i) by reassignment of IP address space to any customer;
 aggregation and/or BGP announcement modifications
(ii) by directly or indirectly causing the occurrence of
 superseding or conflicting BGP Global Routing Table entries; filters
 and/or access lists, and/or
(iii) by directly or indirectly causing reduced prioritization of
 access to and/or from the Prior CUSTOMER Addresses.
 NAC shall provide CUSTOMER with a LOA within 7 days of CUSTOMERS's written
 request for sale,
 NAC shall permit announcement of the Prior CUSTOMER Address to ANY
 carrier, IP transit, or IP peering network."
 -- end --
 In other words, customer is asking a court to rule whether or not IP space
 should be portable, when an industry-supported organization (ARIN) has
 made policy that the space is in fact not portable. It can be further
 argued that the court could impose a TRO that would potentially negatively
 affect the operation of my network.
 NAC does not want to be forced to rely on a customer's ability to properly
 make complex routin

RE: CCO goes down the tubes

[Howard C. Berkowitz]

At 6:58 AM -0800 3/29/04, Michel Py wrote:
 > Maybe I'm the only one left who sees a need to be
 able to check on things from a vt100 at a remote site.
You are not. A telnet version without all the fluffy bullshit would be
more than welcome.


I suppose it's trivial in the grand scheme of things, but on a fairly 
small screen, I can'tget full access to the search without scrolling 
to the right. We wouldn't want to reduce the priority of advertising 
information display to the user who probably has already bought 
equipment and has a question about it, would we?

Perhaps a nastier effect is that the more eye candy, the harder it is 
to use disability access features. One of the incredibly positive 
social effects of the Internet is that it is inclusionary, not 
exclusionary.

The regrettable tendency of many enterprises to equate the Internet 
with the latest and greatest in Web technology leads to both economic 
and sensory exclusion.  Personally, I resent having to buy new 
hardware to run the new operating system that runs the new browser 
that runs the latest plugin, in order to see straightforward 
reference material [1]. In addition, the more visually intensive an 
interface metaphor, the more difficult it is to adapt it to magnified 
images, text-to-speech, or other things needed for people with visual 
disabilities. The more mouse/trackball/pointing device intensive, the 
more difficult it is to adapt to people with motor disabilities -- 
including the all-too-common repetitive stress injuries to hands.

Re: Publish or (gulp) Perish

[Howard C. Berkowitz]

At 10:29 PM +0100 3/26/04, Randy Bush wrote:
 > What ever happened to the blue, paper-back-book-sizes
 periodical, "Proceedings of the Bell Laboratories" or
 summatlikethat?
Bell System Technical Journal.

 >
 (H...I wonder which library _those_ are buried in.)
well, a copy of "the bell technical journal" with the first
paper describing unix is on my shelf.
randy
AFAIK, it still is still published under the name "AT&T Technical Journal."

Re: Publish or (gulp) Perish

[Howard C. Berkowitz]

At 10:45 AM + 3/25/04, [EMAIL PROTECTED] wrote:
 >> Powerpoints have a hard time matching the depth of a refereed journal
 submission, because with the powerpoint, soundbites tend to take
 precedence over content.

Vijay hit it on the head - have we all been foolish by trying to put our
collective expression of service provider best practices and network
design
into an archive of Powerpoint? To quote the Magic Eight Ball, "All
indications point to yes"
It's true that a lot of slide presentations don't have any other
information to back them up. When a researcher presents something
you can almost always go to their published (or pre-published) works
for more information. But this is less true of operator presentations.
So, should NANOG sponsor a document series, like the RFCs or the RIPE
documents, that would form a body of knowledge for IP network operations
best practice? If we did do this, it wouldn't spring into being
overnight, but the program committee could give precedence to
presenters who submit a paper backing up their slides.
Or is this all too academic and too formal for this self-organized
criticality that we call NANOG?
--Michael Dillon
Maybe, maybe not. While RIPE and NANOG aren't exactly the same, RIPE 
(as distinct from RIPE-NCC) does seem to publish things.

I've offered several times to help organize, edit, etc.

Indeed, there may be a possible increment that I've tried personally. 
Right now, I just have www.netcases.net set up as an anonymous FTP 
server (I'm pretty HTML illiterate but am supposed to get help). 
I've put my NANOG PowerPoints in a directory there, along with other 
directories for other presentations and publication.

What I have done is, at least, make corrections to the original slide 
presentation, and I've been intending to update some and perhaps 
cross-reference. Backing them up with some papers could be a start. 
Maybe this approach can be a testbed to see what a minimalist 
supplemental paper might look like.

Re: Enterprise Multihoming

[Howard C. Berkowitz]

At 4:06 PM + 3/12/04, Stephen J. Wilcox wrote:
I think its too easy, thats the problem.
Hoping that I don't sound too much like Bill Clinton, that depends on 
what you mean by "it." If "it" is multihoming, with your own ASN, to 
two providers, your raise some valid points.

Is there an intermediate alternative before you go all out?  Yes, I 
think so, assuming your current provider has multiple POPs.  Let me 
examine some of your points if we consider RFC 1998-style 
multi-POPping (I just invented that highly technical term) using PA 
address space.

For <$1000 (excluding bandwidth/ccts)
you can buy a box, connect to your two providers, get an ASN and IPs 
and you're
away.
Alternatively, another POP link, and preferably another router. If 
you are more concerned with loop failures than router failures, not a 
completely unreasonable assumption, you could get away with one 
router that has multiple interfaces, and spend some of the savings on 
backup power -- possibly a backup power supply in addition to the 
UPS, such as a Cisco RPS on their smaller routers.  While you'll 
probably take a performance hit, or if you can reduce to critical 
traffic on an outage, you might get away with a second smaller router.
I dont agree that connecting to two+ upstreams makes you better. In my
experience end networks have a couple of orders of magnitude more 
downtime than
a PoP in any reasonably large ISP. Ie the percentage theoretical 
improvement is
small.
Like everything else, It Depends. My experience is that access links 
fail more often than provider routing systems, especially with a 
clueful provider.  Since you can't guarantee that your physical 
connectivity to two different ISPs doesn't involve a shared risk 
group in the lines, there are still some things you may not be 
protected against.

One option, depending on the plant in your area, is that if you are 
considering a second router, consider putting it in a nearby 
building, reachable by WLAN (if you are minimizing costs), where that 
building minimally has different ducts to the telco end office, and 
ideally goes to a different end office. Not always possible, but to 
be considered.  Longer-range wireless (radio or optical) links get 
more expensive.

In addition you seriously increase the complexity of your system, chances are
you're using the cheapest kit you could find (or at least cheaper and smaller
than what I would use).. its not great at BGP and may fall over when you get a
minor DoS attack, you probably generate flaps quite a bit from adhoc 
changes and
if you're announcing a /24 then thats going to get you dampened quickly..
That's a motivation for PA address space, where the provider 
aggregate is less likely to be small and easily damped.

 so you
actually create a new weakest link. Also most of the corporates I've 
dealt with
take defaults rather than full tables.. so if the provider does have an issue
you still forward the traffic, theres no failover of outbound routing.
Again looking at intermediate solutions, there are always partial 
routes such as customer routes of the provier.

Even if you spend (waste) the money on some decent gear, you're on 
your own and
when a problem occurs the ISPs are going to be less helpful to you (not by
choice, I mean they dont have control of your network any more.. 
there knowledge
of whats causing problems is limited to the bit that they provide to you), so
chances are your problems may be more serious and take longer to diagnose and
fix.
Again, an operational advantage of multiPOPping and working with one 
carrier, although you aren't going to be protected against insanity 
of their BGP/

IMHO avoid multihoming. You will know when you are big enough and 
you *need* to
do it, if you're not sure or you only want to do it cause you heard everyone
else is and its real cool then I suggest you dont.
MHO would be to look at "multihoming" as a spectrum of solutions 
rather than a binary choice of single-provider-single-link versus 
multiple-provider.  In given situations, you might also want to look 
at DSL or cable for diversity, tunneling to an ISP since the 
broadband provider is unlikely to be willing to speak BGP. Even 
dialup/ISDN, sometimes for critical workstations, has its place.

Shameless plug:  I do go through these options in my book, Building 
Service Provider Networks (Wiley).  Even there, though, I only run 
through the alternatives. You will still have to make your own 
cost-benefit decisions based on business policy, budget, clue level 
and cost of alternatives.

Re: WLAN shielding

[Howard C. Berkowitz]

At 9:51 PM -0500 11/26/03, Sean Donelan wrote:
On Wed, 26 Nov 2003, David Lesher wrote:
 Speaking on Deep Background, the Press Secretary whispered:
 > My company is investigating the use of wireless in a couple of our
 > conference rooms.  Aside from limiting the scope of reception with various
 > directional antennae, does anyone have any suggestions or pointers for
 > other ways to limit the propagation of signals (i.e. special shielding
 > paint, panels or other wall coatings)?
 As I told Andy, you need a "RayProof" or similar brand shielded
 conference room. This is Faraday Cage, with a tight-fighting door,
 etc.
Uhm, dumb question.  If it is that important, why are you using
wireless at all?  Why not install a cheap switch/hub in the middle of the
conference table and let people plug a patch cord from the hub to their
laptops?
Stupid pen-test tricks, instead of using an expensive WiFi scanner and
cracking WEP; often you can collect better intelligence with a radio
turned to the frequency used by wireless lapel mics used by executives
during briefings.
Or by lecturers forgetting them as they went to the bathroom. I only 
did that once.

Re: WLAN shielding

[Howard C. Berkowitz]

At 9:06 PM -0500 11/26/03, David Lesher wrote:
Speaking on Deep Background, the Press Secretary whispered:


 My company is investigating the use of wireless in a couple of our
 conference rooms.  Aside from limiting the scope of reception with various
 directional antennae, does anyone have any suggestions or pointers for
 other ways to limit the propagation of signals (i.e. special shielding
 paint, panels or other wall coatings)?
As I told Andy, you need a "RayProof" or similar brand shielded
conference room. This is Faraday Cage, with a tight-fighting door,
etc.
I don't know what they cost, but I've installed one or 2. Outside
of labor, I suppose they might be in the $50-500K range or so,
for small (12'x6') ones.
Note it's a PITA to keep tight; as the door needs very
tight-fitting gaskets.
You'll need to bring phone/Ethernet in over fiber,
but that's not hard.
If you do put one in, and your local laws don't prevent smoking, make 
it an absolutely no-smoking area. Ventilation tends not to be 
wonderful.

I was once attending a Federal Telecommunications Standards Committee 
meeting, where we were displaced from our regular conference room and 
given a SCIF vault/conference room.  It was stuffy enough as we met 
for a couple of hours, but as we adjourned, the NSA representative 
lit a cigar.

That's when we found out that the vault door was jammed.

No simple cipherlock. Full combination lock.  Trust me. Do not ever 
get in a mostly-sealed room with a dead cigar and some smoke 
remnants.  When we got out, maybe two hours later, our faces matched 
the government green [1] walls. If this hadn't been in the 
then-Defense Communications Agency headquarters with resident 
locksmiths, I don't know how long we'd have been there!

Seriously, give ventilation a lot of thought. You'll need ducts with 
grounded screening and lots of 90-degree bends.

Also, consider having a kick-out panel for emergency escape.  Even 
without high-security locks, I've seen the gasketed doors get stuck 
just in shielded labs.  Think of fire protection -- you really don't 
want a fire suppression gas release in a vault.

[1] I believe the proper descriptor for that shade of green is "gang".

Equant contact?

[Howard C. Berkowitz]

Might I impose on someone who knows the corporate organization of 
Equant to contact me offline?

TIA,
Howard

Re: data request on Sitefinder

[Howard C. Berkowitz]

At 7:26 AM -0700 10/21/03, Owen DeLong wrote:
To inform? Not yet, although I have the feeling that this will be
changed due to historic record. However, changes that have an effect
are always analyzed and a course of action chosen. I believe this is
the job of ICANN. At some point, ICANN's power will need to be
tested and set in stone. Only the community can create or strip that
power. Yet if an organization is going to exist to serve the
community and maintain order, then it needs the power to do it.

I will point out that it will be much easier for the community to strip
that power than to vest it in another entity.  To strip that power only
requires one of two things:
1.  Enough of the community heading in a different direction
and disregarding said entity (ICANN).
2.  An organization such as Verisign openly defying ICANN
and ICANN failing to make a sufficiently strong response
to enforce and protect the consensus will of the community.
I think item 1 is unlikely unless fueled by item 2.  Verisign would do well
to notice that if they do implement the sitefinder wildcards again, and,
ICANN does not successfully put a stop to it, the single most likely outcome
is for the community to view ICANN as irrelevant and impotent.  Once this
happens, the inevitable result is a fragmentation of the DNS, disparate
roots, and, loss of the convention of a single recognized authority at
the root of the tree.  This convention is fundamental to the stability
of the current internet.  Losing it would definitely have negative impact
on the end user experience.
In every forum to which I have convenient access, Verisign has repeatedly
attempted to restrict the discussion to the technical issues around the
wildcards.  The reality is that the technical issues are the tip of the
iceburg and, while costly and significant, they are not the real danger.
The issues that must be addressed are the issues of internet governance,
control of the root (does Verisign serve ICANN or vice-versa), and
finally, whether the .com/.net zones belong to the public trust or to
Verisign.  Focusing on the technical is to fiddle while Rome burns.
Related issues include whether the IETF process, even if flawed, is the
consensus means of proposing and discussing changes in the
infrastructure. Whether or not the operational forums like NANOG have a
role in this process, or even in presenting consensus opinions, also is a
basic question for Internet governance.
The IETF process is the consensus means of proposing and discussing changes
in the DESIGN of the infrastructure, not the construction or maintenance.
Valid observation.

That _IS_ the role of the network operators and the operators forums.
If one thinks of using the IETF model in the operational forums, 
which I don't consider an unreasonable idea, the operational forums 
will need specialized mailing lists/working groups, a document 
handling procedure, and a means of signing off on the "best current 
practices" track (analogous to standards track). This isn't rocket 
science, since most of the conceptual work has been done in the IETF 
-- mind you, if we ever do this, could we NOT perpetuate some of the 
more obscure rules in RFC formatting?

I'd certainly be willing to work with developing such a process as 
long as I have a roof over my head (In this economy, I'm more in the 
"will network for food" category). I think others will, and I would 
hope that there's even a little time this week for hallway discussion 
of the idea.

For
this to work, however, the operators have to be generally of good will and
cooperative for the greater good.  This model is somewhat antithetical to
capitalism because for it to operate efficiently, it requires the long term
good of the community to take precedence over the short-term gains of the
individual or single organization.  Capitalism is well optimized for the
short-term gains of the individual or single organization.  This is one
of the growing pains that comes from the internet being originated as a
government-sponsored community research project.
Although there have been precedents for competitive profit-making 
organizations to cooperate for their mutual benefit.  One of the 
earliest examples is the cooperation of insurers to form Underwriters 
Laboratories.

  The design was done
assuming a collection of organizations whose primary motivation was to
cooperate.  As we shifted to a privatized internet, that fundamental design
assumption was broken and we have seen some interesting changes as a result.
Again, there are precedents for conflict-avoiding organizations, or 
organizations to mitigate industry-created threats. In the first case 
are both government organizations like the air traffic control 
system, and private monitoring centers for utility and telephone 
networks. We have intermediate cases like the VISA network, owned by 
its competing member banks. In the latter, we have

Re: data request on Sitefinder

[Howard C. Berkowitz]

At 9:46 PM -0500 10/20/03, Jack Bates wrote:
todd glassey wrote:
Richard -
Do they (Verisign)  have any legal reason to??? - is there anything between
them and ANY of their clients that requires them to inform them before any
changes to protocol facilities are made - I think not.
To inform? Not yet, although I have the feeling that this will be 
changed due to historic record. However, changes that have an effect 
are always analyzed and a course of action chosen. I believe this is 
the job of ICANN. At some point, ICANN's power will need to be 
tested and set in stone. Only the community can create or strip that 
power. Yet if an organization is going to exist to serve the 
community and maintain order, then it needs the power to do it.
Throughout this affair, I've been puzzled by what seems to be an 
assumption that once a contract exists, it cannot be changed or 
cancelled. Yet such changes and cancellations happen daily in 
business.  They may require litigation, lobbying of the Congress or 
executive when government is involved, market/consumer pressures, 
etc., but change is not impossible.

Jack makes excellent points here, which I might restate that this is 
a defining moment for ICANN to establish its viability and relevance 
as an organization.  If ICANN is to be meaningful in the future, it 
_must_ make a strong stand here.

Related issues include whether the IETF process, even if flawed, is 
the consensus means of proposing and discussing changes in the 
infrastructure. Whether or not the operational forums like NANOG have 
a role in this process, or even in presenting consensus opinions, 
also is a basic question for Internet governance.

Purely from my experience in journalism, media relations and 
lobbying, I have to respect the effectiveness of the Verisign 
corporate folk who largely have been setting the terms of debate, and 
managing the perception -- or misperception -- of this matter in the 
business and general press.

Apropos of that, lots of people equate "privatization" of the 
Internet to its "commercialization."  Privatization isn't nearly that 
binary. If privatization, in general, is getting the US government 
out of Internet governance, we still have the options of:

   -- transferring such control as exists (and there may be no control
  mechanism) to a quasi-governmental body such as ICANN.
   -- transferring control, especially with regard to stewardship,
  to a not-for-profit corporation (e.g., ARIN)
   -- accepting that an organization such as IETF will manage a consensus
  process
   -- subcontracting, but closely monitoring, to a general for-profit
  enterprise.
   -- transferring control to a regulated technical monopoly, probably
  with a financial model of return-on-investment rather than maximizing
  shareholder value.
   -- transferring control, at least for a defined period, to a for-profit
  enterprise with a fiduciary responsibility to maximize shareholder value
   -- transferring control to competing for-profit organizations
Howard,
who is puzzled by what seems to be lots of tunnel vision (and I don't 
mean GRE).

I think Vixie has alluded to this a few times, and I know there is 
much that goes on in the hallways concerning the overall problem of 
who controls what. Verisign is just helping to push the process 
along. I doubt it will end as they want it to.

-Jack

Re[3]: data request on Sitefinder

[Howard C. Berkowitz]

On Mon, 20 Oct 2003 17:15:23 -0400 "Howard C. Berkowitz" 
<[EMAIL PROTECTED]> wrote:
 At 5:04 PM -0400 10/20/03, Richard Welty wrote:
 >may i suggest another operational issue then?

 >how does verisign plan to identify and notify all affected parties
 >when changes
 >are proposed?

 >for example, in the current case, how do they plan to identify every
 >party running
 >postfix and inform them that they need to upgrade their MTA?

 >this seems non-trivial to me.

 Purely from an operational standpoint, it would be a mark of
 efficiency to have a central repository of who is running what.  That
 would mean that notifications would only be sent to those that need
 them, and also would provide objective information to determine how
 many organizations would be affected by a change.  In other words,
 something that actually would be useful.
i maintain that building this list is phenomenonally difficult. the set of
people running mail servers is substantially larger than the set of
people who read nanog, run backbones, run regional ISPs, etc., etc.
I don't really disagree with you, even ignoring that many providers 
would consider much of this information proprietary, much as they 
might for private peering arrangements. This is something of a 
thought experiment on what would have to be available for a Verisign 
or the like to make unilateral changes without presenting the idea 
for comment, well in advance.

The process of asking for comment through IETF and the operational 
forums has the proven benefit of getting major players to look at the 
issue and decide to comment.  Now, as you point out, there are many 
people who run mail servers and the like, who don't follow any 
relevant mailing lists.

I would suggest, however, that the number of people that do read 
these lists run mail servers with more end users than the small 
system administrators that do not.

The absence of a list such as I've described, the difficulty of 
creating of which you point out, makes it more unlikely to me that an 
organization can really assess the effects of unilateral design 
changes, especially when that assessment is shrouded in commercial 
secrecy.

i don't disagree that it would be useful, but how are you going to
build it without actively probing mail servers across the internet?
and it can't possibly ever be complete, with PIX firewalls obscuring
SMTP banners and sysadmins depending on security-by-obscurity
who change their banners to elminate MTA identification.
richard
--
Richard Welty [EMAIL PROTECTED]
Averill Park Networking 518-573-7592
Java, PHP, PostgreSQL, Unix, Linux, IP Network Engineering, Security

Re: data request on Sitefinder

[Howard C. Berkowitz]

At 5:09 PM -0400 10/20/03, [EMAIL PROTECTED] wrote:
On Mon, 20 Oct 2003 16:31:45 EDT, "Steven M. Bellovin" 
<[EMAIL PROTECTED]>  said:
 A number of people havce responded that they don't want to be forced to
 pay for a change that will benefit Verisign.  That's a policy issue I'm
 trying to avoid here.  I'm looking for pure technical answers -- how
 much lead time do you need to make such changes safely?
OK, since you asked

At least from where I am, the answer will depend *heavily* on whether Verisign
deploys something that an end-user program can *reliably* detect if it's been
fed a wildcard it didn't expect.  Note that making a second lookup for '*.foo'
and comparing the two answers is specifically *NOT* acceptable due 
to the added
lookup latency (and to some extent, the attendant race conditions and failure
modes as well).

Also note that it has to be done in a manner that can be tested by an
application - there will be a *REAL* need for things like Sendmail to be
able to test for wildcards *without the assistance* of a patched local DNS.
And yes, this means the minimum lead time to deploy is 'amount of 
time to write
a "Wildcard Reply Bit" I-D, advance through IETF to some reasonable point on
standards track, and then upgrade DNS, end host resolvers, and applications'.
You make an assumption here -- one with which I agree completely -- 
but that certainly wasn't followed during the Sitefinder debacle. The 
assumption is that the IETF provides a tested mechanism for 
disseminating information and making comments.

Verisign claims that they had tested their ideas with a 
Verisign-selected group of organizations, and made their commercial 
decisions based on the proprietary data it generated from those 
organizations.

Re[2]: data request on Sitefinder

[Howard C. Berkowitz]

At 5:04 PM -0400 10/20/03, Richard Welty wrote:
On Mon, 20 Oct 2003 16:31:45 -0400 "Steven M. Bellovin" 
<[EMAIL PROTECTED]> wrote:

 A number of people havce responded that they don't want to be forced to
 pay for a change that will benefit Verisign.  That's a policy issue I'm
 trying to avoid here.  I'm looking for pure technical answers -- how
 much lead time do you need to make such changes safely?
may i suggest another operational issue then?

how does verisign plan to identify and notify all affected parties 
when changes
are proposed?

for example, in the current case, how do they plan to identify every 
party running
postfix and inform them that they need to upgrade their MTA?

this seems non-trivial to me.
Purely from an operational standpoint, it would be a mark of 
efficiency to have a central repository of who is running what.  That 
would mean that notifications would only be sent to those that need 
them, and also would provide objective information to determine how 
many organizations would be affected by a change.  In other words, 
something that actually would be useful.

Unfortunately, we have seen Verisign constantly take the position 
that information they learn through operations is their intellectual 
property, to be used as they see fit, and generally to be kept 
proprietary.

So if we try to separate operational from policy, we see white-winged 
ships sail by, carrying data that might be useful, but then have them 
crash on the rocks of stewardship of the data.

Re: IAB concerns against permanent deployment of edge-based filtering

[Howard C. Berkowitz]

At 10:57 AM -0700 10/20/03, Owen DeLong wrote:
OK... I've been lurking for a while.

I think the definition IAB intended to express concern about was:

Backbones (transit providers) deploying [permanent] filtration on their
connections with other ISPs.
I would like to propose the following terminology definitions FOR 
THIS EMAIL message
and ask that my following comments be viewed with these definitions in mind:
What you're doing here reminds me of what we did in the BGP 
Convergence Technology draft, 
http://www.ietf.org/draft-ietf-bmwg-conterm-05.txt (now  with the 
IESG).  We made what we felt was a useful but informal distinction 
between customer edge routers at the POP, and interprovider edge 
routers.  Randy Bush, as the advisor, pointed out these definitions 
are not rigorous, and indeed might be considered a research problem. 
We modified our language to reflect his concerns, and that we didn't 
expect the definitions to be normative.

Nevertheless, there seems a practical need, which you put as a policy 
consideration here, to distinguish between routers that connect an 
ISP to transit and nontransit peers.  The nontransit peers often will 
not be taking full routes.

"Edge Network"	A network which does not provide transit between 
multiple BGP speaking
		neighboring ASs.

"End Network"	Or "End User Network" a network which has may or may 
not speak BGP, but,
		provides services to a single organization and has a 
single administrative
		control at it's border(s).  (e.g. Sun Microsystems, 
Tellme, HP, etc.,
		not MSN, C&W, Verio, etc.)

"Transit Network"
		A network which does not meet the definitions of Edge 
Network or
		End Network.

I think given those terms, there is generally agreement that the 
following are good
operational practice:

	1.	Edge Networks and End Networks should not emit 
packets containing source
		address specifications outside of their assigned (or 
allocated) ranges.
		They should employ filters at their peering-points to 
prevent this.

	2.	Transit Networks should _NOT_ permanently (or 
quasi-permanently) block
		traffic to other transit networks other than to the 
minimal extent
		necessary to meet operational considerations around 
attacks.  The general
		deployment of such filters would in itself be a form 
of denial of service.

	3.	End networks should accept and emit traffic related 
to their desired
		service profile (what internet features they want to 
take advantage of)
		and block others.

	4.	Any network connecting to an Edge or End network may 
(and in some cases
		should) cooperate in filtering traffic at 
conneciton-points to said
		Edge or End networks in a manner consistent with the 
desires of the
		Edge or End network in question.  To do so contrary 
to the wishes of
		the Edge or End network in question is a form of 
denial of service.

	5.	It is generally good practice for any network 
providing services to
		Edge or End networks to have a published AUP and to 
disconnect customers
		which violate the AUP.  This is not contrary to the 
wishes of the client,
		or they should not have signed the AUP.

Having said that,
	I don't think IAB is trying to tell people how to run their networks.
	I do think IAB has a point that if I'm connected to an ISP 
which is a customer
of UUNET, and I want to exchange traffic of some unpopular service 
with another host
that is a connected via an ISP that is a customer of C&W, it is a 
bad thing if C&W
and UUNET block that traffic at their peering point(s).  If my ISP blocks it
or the ISP that connects the other host blocks it, then, presumably, I (or the
person at the other end of the connection) have some ability to 
address it with
the service provider we are paying.  Having UUNET or C&W block it at 
some arbitrary
point in between is:

1.  Hard to isolate.
2.  Hard to troubleshoot.
3.  Unexpected damage
4.  Not a good idea in most cases.
Assuming that this is what IAB was attempting to address, I agree it 
should be addressed.
The fact that I need to make this assumption should, IMHO, also be 
addressed by IAB and
they should clarify what their concern is.

Owen

--On Monday, October 20, 2003 05:00:58 AM -0700 [EMAIL PROTECTED] wrote:



 prudent/paranoid folk over the years have persuaded me that
 it makes the best sense to only run those applications/services
 that I need to and shut off everything else - until/unless there
 is a demonstrated need for it.
very true for a host, even somewhat true for a site.  very untrue
for a backbone.
randy

there appears to be a disconnect in the wording of the IAB document:
it starts:

IAB concerns against permanent deployment of edge-based filtering
The IAB notes that there ISPs/ASes undertaking permanent deployment of
edge-based protocol number/port number packet filtering on traffic
received from eBGP peers.

it can be viewed from the perspective of a transit provider
looking toward its edges, the client

RE: data request on Sitefinder

[Howard C. Berkowitz]

At 5:22 PM +0200 10/20/03, Jeroen Massar wrote:


Ahem, so Verisign wants to change the complete working of the
internet with the currently installed base because they want
to gather all the typo's??? Are they going to pay us the money
for upgrading/verification/checking/testing etc?
Fix the Webbrowsers, which in most cases already support the
functionality their 'application' gives. If Verisign wants
the webbrowsing folk to use their 'sitefinder technology'
then they should take a share in Microsoft, AOL, Opera and
a number of other companies and pursuade them to include it.
Given that this functionality does exist in web browsers, there's the 
flavor of monopolistic competition that may be vulnerable to 
antitrust action.

Don't change something that doesn't need fixing.
(Ignoring the spam thing :)
*if* Verisign gets it through that the installed base has
to bend over because they introduce such a thing it would
be a very bad thing for the internet as a whole and it would
really mean that the internet is yet another commercial
thing controlled by one single entity.
Look at the interview with Verisign's CEO at 
http://news.com.com/2008-7347-5092590.html?tag=nefd_gutspro, and I 
think you'll see that your "what it would really mean" is exactly 
Verisign's position.

Re: AOL E.mail issues..

[Howard C. Berkowitz]

At 7:30 AM -1000 10/18/03, Robert Mathews wrote:
Is anyone experiencing E.mail weirdness, specifically bounces from AOL
today at all?  I have spoken to a number of others from various parts
of CONUS, who seem to have a similar issue..
I've been seeing it from them, but also from non-AOL sources. They 
appear to be virus bounces with a French-language message in the 
headers.Not speaking French, I don't remember the exact phrase and 
unfortunately just purged my email trash, but the phrase had DENOYE 
in it.

All of these bounces, when viewed, contained the continuing Microsoft 
"update mail".

Re: Fascinating interview with Verisign CEO

[Howard C. Berkowitz]

At 10:59 AM -0500 10/17/03, Eric A. Hall wrote:
on 10/17/2003 3:17 AM Hank Nussbacher wrote:

 http://news.com.com/2008-7347-5092590.html
First reaction is that this guy *really* needs some schooling in the value
of having public-interest bodies facilitate and regulate interstate
commerce in a federated system. Second reaction is that "commercializing
the infrastructure" is a fairly dumb way to frame the debate, since we're
not talking about the entire infrastructure but instead are talking about
a couple of zones. Third reaction is that his opinion of what the Internet
"needs" is not only wrong, but even if it were correct it would not give
him the authority to usurp control over those zones. What next, all
domains below the root have to pay a tax to the new emporer?
A subtlety often lost in this discussion is that while we might want 
to get government out of the process, privatization does not 
necessarily mean commercialization.  On one hand, privatization can 
go to a not-for-profit.  Another alternative is to commercialize, but 
to treat the commercial enterprise as a regulated utility.  Verisign 
is operating as a totally free entity.

A Cautionary Tale of Tomatoes

[Howard C. Berkowitz]

I do believe the Verisign discussion and the opinions of operators is 
very much on-topic. At the same time, while I participated in the fun 
of tomato discussions, people are right in saying it won't play well 
to media.

Perhaps still on topic, if we consider protest and HOW it's received, 
I had a strange public experience involving tomatoes (well, tomato 
juice).

It was many years ago, when I was an OSI protocol evangelist.  Mind 
you, OSI is still the answer, but we still haven't figured out the 
question.  I had to give a tutorial in San Francisco, and had a nasty 
case of laryngitis.

The only thing that kept my throat working was constantly sipping hot 
soup. So, I asked the hotel to make soup available for me, at the 
podium, from 9AM on.  They were resistant, and said they didn't do 
soup until lunchtime. Exasperated, I croaked out that they could heat 
up a can of tomato juice, give a dash of tobasco and pepper, and send 
it to me in a pitcher.

To get the full effect, you need to understand that the room was set 
up in theater style, with a central aisle. You also have to 
understand that while this was San Francisco, apparently the waiters 
were all unemployed actors, ala Los Angeles.

About 9:15, the double doors swung open, and a waiter rolled a white 
covered cart, featuring a pitcher of steaming red liquid and a single 
glass, to me.  In a fairly good Transylvanian access, the waiter 
croaked out "your nourishment, Sir," and theatrically slunk out.

My students looked at me very, very strangely for the rest of the 
class. So, I can testify tomato products can backfire.

RE: Pitfalls of _accepting_ /24s

[Howard C. Berkowitz]

A proposal was made some years ago, which I thought was by Tony Li, 
but, IIRC, he says it wasn't original with him.  It does require 
cooperation from competitors, but can reduce the number of 
announcements. Under some circumstances, it may cause blackholing, 
but so can /24 filtering.

The idea is to establish bilateral blocks of provider space. Let us 
say Provider A and Provider B recognize that they have a significant 
number of common multihomed customers.  Arbitrarily, one of the 
providers (assume A) starts off with a block -- let's say a /19 or 
/20 to which both providers will assign their multihomed customers. A 
and B peer and send more-specifics to each other.

To the outside world, however, A advertises its largest aggregate 
plus the multihomed block.  B advertises this block of Provider A 
space as well as its own aggregates.

If A and B do not peer, the likelihood of blackholes become much 
higher since they may not see the more-specifics in the multihomed 
block.

Has anyone reexamined this proposal lately?

Re: Tomatoes for Verisign at NANOG 29

[Howard C. Berkowitz]

At 12:45 PM -0700 10/16/03, JC Dill wrote:
At 11:56 AM 10/16/2003, Chris Strandt wrote:

Maybe a "vote" at the end of the presentation would be better.

After Verisign has to say what they want, it would be interesting 
to see what the participants think of starting Site Finder again.

Its not as press worthy... but it lets Verisign have their say, and 
gives the community a voice right there on the spot on the issue.
Have a vote with the tomatoes.  All those who oppose the wildcards 
would put their tomatoes or tomato-substitutes in one pile, all 
those who approve of the wildcards would put their token-of-choice 
in another pile.
I'm not sure I really want to think about the 
alternate-token-of-choice. Let me stipulate that I was born and 
brought up in Northern New Jersey, started as a chemist exposed to 
the usual organic chemistry aromas, and STILL consider some things as 
smelling bad. :-)

We need a *visual* way to make the point to Verisign, to ICANN, to 
the press, to the Internet using public who do not understand the 
underlying technical issue.  The press is not going to put this 
story on the wire and carry it on the front page of newspapers 
around the world if the story is "NANOG meeting attendees voted 417 
to 3 to indicate disapproval of Verisign's wildcard scheme in the 
.com and .net root server".  A photo montage showing people buying 
tomatoes, holding tomatoes, placing tomatoes in a pile, a room full 
of people wearing red shirts, etc. THAT is what the press needs to 
get the message out to the people to shame ICANN into doing the 
right thing.  Look at this as a photo-op for NANOG participants to 
collectively make their point known in a visually graphic way.

From experience back in my more general political days, either have a 
see-through bag or basket for the tomatoes. While a nice pyramid 
would be nice, without very careful handling, we will have tomatoes 
bouncing in all directions.

Re: Tomatoes for Verisign at NANOG 29

[Howard C. Berkowitz]

If you are attending NANOG 29, please attend this session and wear a
red shirt.
Ahem. Many of us are Star Trek experts, and it will take a LOT more than
this to get people to wear a red shirt.
Huh?  I'm somewhat familiar with Star Trek, and, I realize the red shirts
are usually the first to die
Right the first time!

Porjecting the movie shows some level of group leadership support.
Stacking tomatoes shows just how much of the community is opposed
to wildcards.  I think in this case, a physical show of tomatoes is
more effective.
You're right--it's just that I enjoy the movies as one of the worst 
implementations ever done. Seemed appropriate.

The neatly stacked voting pyramid is an excellent idea, especially if 
there were any media coverage.

Sort of reminiscent of the end of Conan the Destroyer, as the torches go out.

Re: Tomatoes for Verisign at NANOG 29

[Howard C. Berkowitz]

Dan and Owen, I nominate you two for the tomato acquisition and 
distribution committee.

To recap:  At NANOG 29 in Chicago, on Monday October 20th at 9:15 am 
a session on "VeriSign's Wildcard Record: Effects and Responses" 
will be held, with Mark Kosters and Matt Larson from VeriSign and 
Suzanne Woolf from ISC:



If you are attending NANOG 29, please attend this session and wear a 
red shirt.
Ahem. Many of us are Star Trek experts, and it will take a LOT more 
than this to get people to wear a red shirt.

If possible, please buy/bring a tomato.  Take your tomato to the 
front of the room and place them in a pile before Verisign, to make 
it absolutely clear how you feel about Verisign's wildcards and 
sitefinder.  Please do not throw your tomatoes.

Given that we deal, in any case, with virtualizations, I rather like 
the idea, instead, of projecting "Attack of the Killer Tomatoes" on 
several walls. One can also explain that the careful designers at 
Verisign are spiritual kin to the producer and director of that movie.

Re: Finding clue at comcast.net

[Howard C. Berkowitz]

At 10:40 PM -0400 10/9/03, Brandon Ross wrote:
On Thu, 9 Oct 2003, Matt wrote:

  > I wouldn't recommend that actually.  The local folks do not have any
  > control over the IP infrastructure, they only handle the HFC plant.
 Do you think that may have anything to do with the complaints cited here?
Nope, most of the complaints here seem to be about technical support.

As far as networking problems, I think most folks on NANOG would agree
that to run a stable network, the network needs to be designed and
operated by a single organization.
As a customer quite frustrated with support, I have to support 
Brandon about a centralized authority for routing and other common 
services such as mail.  I can appreciate the issue of customer 
support for a residential service starting with level 1's whose only 
level of clue over Joe Sixpack is a flipchart. What frustrates me is 
the inability to escalate, and the lack of communication between 
customer support and the real operations folk.

Nobody's perfect.  When I was a Verio customer, I sometimes was able 
to get things escalated, but there was a time or two where I wound up 
appealing to Randy Bush.  If I do call upon a colleague like that, I 
like to think that I've thought through the issue and have either a 
diagnosis or a solution -- perhaps a better procedure for Level 2 and 
up.

It's a tough world (and I'm not singling out Comcast). If I were 
paying for an OC-192, you'd better believe I'd get clueful support. 
With what I pay for residential broadband, there's only so much 
support budget, and I recognize many of the incoming calls ARE from 
lack of end user clue. But, it still strikes me that proper 
escalation of a user with a technical explanation is in the long-term 
interest of the service provider.

Semi :-), I sometimes wonder if Level 1 should automatically escalate 
a customer that says certain magic words. Hey, if we are going to 
talk about magic, I want a spell that lets me turn anyone who doesn't 
know what a traceroute is into a frog. Let them ribbit rather than 
ping.

Re: Is there anything that actually gets users to fix their computers?

[Howard C. Berkowitz]

At 3:26 PM -1000 10/9/03, Michael Painter wrote:
http://www.wired.com/news/digiwood/0,1412,60613,00.html

"When students first register on the network, they are required to 
read about peer-to-peer networks and certify that they will not
share copyright files. Icarus then scans their computer, detects any 
worms, viruses or programs that act as a server, such as Kazaa.
Students are then given instructions on how to disable offending programs."

Kinda' does some of what you want done? 

Icarus.

Just sort of scares me that some students might use a hair dryer on his wings.

Re: Finding clue at comcast.net

[Howard C. Berkowitz]

- Original Message -
From: "Howard C. Berkowitz" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Thursday, October 09, 2003 11:20 AM
Subject: RE: Finding clue at comcast.net

 At 9:29 AM -0500 10/9/03, Austad, Jay wrote:
 >Comcast's phone support department is the *worst*, WORST, I've ever dealt
 >with.  I think they are outsourced, they have to go by a script, and many
of
 >them probably hardly know what a computer even is.  Once I called because
of
 >a problem on their network, and I told the person on the phone that there
 >was a problem on their network, and I pinned it down to a couple of
routers
 >where the problem may be, and she responded, very sternly, "Sir, WE DON'T
 >HAVE ANY ROUTERS"
 Same thing here. Last night, I was told that no escalation personnel
 were available.
* Depending on how big a company is or how the outsourcing company staffs at
night this can be true. No escalation personnel may be physically present,
but this doesn't mean there isn't someone they can call. An outsourcing
companies call center agents have to first decide (policy?) that the issue
warrants escalation, and then they probably have to call THEIR manager (of
the outsourcing company). This manager then gets to decide if it REALLY
warrants escalation to their client (the cable company). They don't want to
call them after-hours unneccesarily. And cable companies are used to having
24-hours to resolve most outages, and if it doesn't affect a LOT of their
customers it isn't considered an outage worth escalating. A real world
example is: 6 calls in one cable node with the problem persisting for 15 to
30 minutes (calls keep coming in) would be a case for an on-call technician
to be called. Anything less just gets Service Calls placed in CableMaster on
AS/400. These things can wait for a scheduled (all-day) appointment unless
the customer insists on a time-frame.
*sigh* Y'know, I could live with it if I could even have a mailbox to 
which I could send detailed trouble reports, even if no one looked at 
them on the next day.  While their routing seems to be fairly stable 
these days, there would be times I'd traceroute from several sites I 
could reach and take views from multiple looking glasses, giving me a 
pretty fair idea where, and even what, the problem is.

The customer disservice people that really drive me nuts are the 
first-levels that believe they are NEVER wrong.

If you say there's an IP routing problem, they may say "how do you 
know there's a problem with our ippp(rhymes with pip)?"

"We don't support SMTP or POP3. You have to use Outlook.."

"You must remove your firewall and router so we can troubleshoot."

"It's irrelevant that you can ping the access router. It must be your 
modem. Go to the local office and exchange it."

"I'm sure the problem will be resolved, so there's no reason to give 
you a trouble ticket"

"b'gop? bajop? We don't support bee-gee-pee in our network."

"access router? We just have Windows servers."


* Exactly what I described above. But I wouldn't accept "hopefully somebody
would respond". That is NOT acceptable. Someone should respond within 1
business day at most. Again your not going to find many on-call or
higher-level support reading email after-hours and responding to things.
Even I couldn't do it ALL of the time. And I was the only one doing that in
a local cable company (not a national company) with 2 cities.
I'd be happy, again, if they'd let me give them a trouble ticket. 
Oh, they have told me at times that I could do that at their website, 
which is an interesting problem when you don't have connectivity.

I'd gladly pay extra for dial backup at low speed, but they don't offer that.


 At the time, I was paying for their "Pro" service, intermediate
 between regular residential and full business. My contact said that
 while that was supposed to get better customer support, an early plan
 to route it to business Comcast failed, and there really was NO
 > separate Pro support organization. I dropped the Pro service after I
 learned that residential service no longer insisted you remove any
 local routers and firewalls before deigning to troubleshoot. They
 still ask you to do that, but repeated NO responses can get them to
 proceed.
* Pro services, where I was working, gets escalated like the above
description I wrote. If you are not completely down you're probably not
going to see something done about it until the next business day (assuming
after-hours).
They treated completely-down situations like that.


 A few NANOGs back (Atlanta), I did a presentation on customer
 satisfaction, which, frankly, was in many respects a case study of
 how I'd reform customer support at my then ISP/D

Re: Sitefinder and DDoS

[Howard C. Berkowitz]

 > Let's also assume someone sets up a popular webpage with malware
 HTML causing it, perhaps with a time delay, to issue rapid GETs to
 deliberately nonexistent domains.
You don't even have to imagine that.

Imagine a long-term port 80 Denial of Service (DoS) attack against a
given website (using the website url rather than IP, which is not
uncommon).
Imagine the attacked domain administrator removes their DNS records
from the registry to alleviate the attack.
The attack is now directed at the Verisign Sitefinder service.

Adam
OUCH. Yet worse.

Re: Sitefinder and DDoS

[Howard C. Berkowitz]

At 10:41 PM +0300 10/9/03, Petri Helenius wrote:
Howard C. Berkowitz wrote:

I am NOT suggesting this simply as an argument against Sitefinder, 
and I'd like to see engineering analysis of how this vulnerability 
could be prevented.
With $100M annual revenue at stake, I would be willing to provide 
distributed solutions
to this problem if you send me a reasonable fraction of that money.

Pete
As long as I get a finder's fee! :-)

Sitefinder and DDoS

[Howard C. Berkowitz]

Let's assume for a moment that Verisign's wildcards and Sitefinder go 
back into operation.

Let's also assume someone sets up a popular webpage with malware HTML 
causing it, perhaps with a time delay, to issue rapid GETs to 
deliberately nonexistent domains.

What would be the effect on overall Internet traffic patterns if 
there were one Sitefinder site?  (flashback to ARPANET node 
announcing it had zero cost to any route)

How many Sitefinder nodes would we need to avoid massive single-point 
congestion?

AFAIK, the issues of distribution of Sitefinder, and even a formal 
content distribution network, were not discussed. I asked some 
general questions that touched on this at the ICANN ISSC committee 
meeting, but I think they were interpreted as directed toward the 
reliability of the Sitefinder service in operation, rather than 
potential vulnerabilities it might create.

I am NOT suggesting this simply as an argument against Sitefinder, 
and I'd like to see engineering analysis of how this vulnerability 
could be prevented.

Sitefinder and DDoS

[Howard C. Berkowitz]

Let's assume for a moment that Verisign's wildcards and Sitefinder go 
back into operation.

Let's also assume someone sets up a popular webpage with malware HTML 
causing it, perhaps with a time delay, to issue rapid GETs to 
deliberately nonexistent domains.

What would be the effect on overall Internet traffic patterns if 
there were one Sitefinder site?  (flashback to ARPANET node 
announcing it had zero cost to any route)

How many Sitefinder nodes would we need to avoid massive single-point 
congestion?

AFAIK, the issues of distribution of Sitefinder, and even a formal 
content distribution network, were not discussed. I asked some 
general questions that touched on this at the ICANN ISSC committee 
meeting, but I think they were interpreted as directed toward the 
reliability of the Sitefinder service in operation, rather than 
potential vulnerabilities it might create.

I am NOT suggesting this simply as an argument against Sitefinder, 
and I'd like to see engineering analysis of how this vulnerability 
could be prevented.

Customer support tutorial

[Howard C. Berkowitz]

I was about to answer a request for my presentation about customer 
support, but I had a cat-on-keyboard exploit and I don't know who 
asked.

In any event, the Atlanta presentation is at 
http://www.nanog.org/mtg-0102/cust.html

I did it as a two-part with an intermediate BGP tutorial. In one of 
the two, there is an RFC 2270 slide. Please, please ignore it -- my 
brain went out to lunch on that one!

Actually, this raises the interesting point -- is there an interest 
in updating and having running commentary on older presentations, 
keeping some content up to date?  Is this something the NANOG site 
could support?

RE: Finding clue at comcast.net

[Howard C. Berkowitz]

At 9:29 AM -0500 10/9/03, Austad, Jay wrote:
Comcast's phone support department is the *worst*, WORST, I've ever dealt
with.  I think they are outsourced, they have to go by a script, and many of
them probably hardly know what a computer even is.  Once I called because of
a problem on their network, and I told the person on the phone that there
was a problem on their network, and I pinned it down to a couple of routers
where the problem may be, and she responded, very sternly, "Sir, WE DON'T
HAVE ANY ROUTERS"
Same thing here. Last night, I was told that no escalation personnel 
were available.

On the couple of occasions where I got escalation, I once had an 
informal conversation with a 3rd level. Their phone center is in 
Halifax, NS -- didn't find out if it is outsourced or not. While the 
person with whom I spoke was reasonably clueful, he told me that 
customer support had no interactive communication with network 
operations -- at best, they could send an email about a routing, 
SMTP, etc. problem and hope somebody would respond.

At the time, I was paying for their "Pro" service, intermediate 
between regular residential and full business. My contact said that 
while that was supposed to get better customer support, an early plan 
to route it to business Comcast failed, and there really was NO 
separate Pro support organization. I dropped the Pro service after I 
learned that residential service no longer insisted you remove any 
local routers and firewalls before deigning to troubleshoot. They 
still ask you to do that, but repeated NO responses can get them to 
proceed.

A few NANOGs back (Atlanta), I did a presentation on customer 
satisfaction, which, frankly, was in many respects a case study of 
how I'd reform customer support at my then ISP/DSL, cais.net. If 
NANOG ever did formal documents, I'd like to see a guideline on how 
to run customer support.

In any case, if you manage to get the call escalated a couple of times
(after lying about rebooting your computer 47 times),
You forgot reinstalling Windows. On a Mac.

you'll get someone
good.  Also, there are some good people who read this list.  But calling
their phone support to get anything useful is like trying to squeeze blood
from a rock.
-jay

 -Original Message-
 From: Howard C. Berkowitz [mailto:[EMAIL PROTECTED]
 Sent: Wednesday, October 08, 2003 7:36 PM
 To: [EMAIL PROTECTED]
 Subject: Finding clue at comcast.net


 I'm rapidly beginning to believe this is equivalent to finding the
 pot of gold at the end of the rainbow. When my broadband alternative
 is Verizon and it's looking better, this is scary.
 Sometime today, their SMTP server started bouncing messages with more
 than 3 addressees.  When I called customer support, I was told "we
 only handle troubleshooting, not mail service."  The operator
 "guessed" they might be doing software updates on the mail service,
 had no information, and said there was no person to which it could be
 escalated.
 She insisted that I call my local cable office to find out when the
 "server repair" would be completed, because "they schedule all
 repairs."
 Is this a bad dream?

RE: Verisign on Process

[Howard C. Berkowitz]

At 5:23 PM -0700 10/8/03, David Schwartz wrote:
 > Is it possibly time to suggest that perhaps ICANN should
 call for formal separation of regiSTRAR functions from
 regiSTRY functions, and stipulate that stewards of record
 for regiSTRY functions not participate in regiSTRAR roles?
	Already done -- read the contact between ICANN and VeriSign.

 Certainly it's been shown to be very difficult to resist
 the temptation to extend editorial control over what entries
 get placed into the DNS records as a regiSTRY if you also
 happen to be able to increase the profits from your regiSTRAR
 role.
	Sure, but you could also do it to increase the profits from 
your registry
role, which seems to have been the intent of SiteFinder.

 If the functions are stipulated to be kept separate,
	They are.

 then we have a much better opportunity to engage a system of
 checks and balances, to self-limit potential future abuses
 like this.
	How would that stop a registry from, for example, adding a 
wildcard record
that goes to pages sold to the highest bidder? Registry functions are
supposed to be wholly ministerial. VeriSign doesn't get that and ICANN is
going to have to force them to -- probably more than once.

	DS

Finding clue at comcast.net

[Howard C. Berkowitz]

I'm rapidly beginning to believe this is equivalent to finding the 
pot of gold at the end of the rainbow. When my broadband alternative 
is Verizon and it's looking better, this is scary.

Sometime today, their SMTP server started bouncing messages with more 
than 3 addressees.  When I called customer support, I was told "we 
only handle troubleshooting, not mail service."  The operator 
"guessed" they might be doing software updates on the mail service, 
had no information, and said there was no person to which it could be 
escalated.

She insisted that I call my local cable office to find out when the 
"server repair" would be completed, because "they schedule all 
repairs."

Is this a bad dream?

Re: News coverage, Verisign etc.

[Howard C. Berkowitz]

At 3:54 PM -0400 10/8/03, Steven M. Bellovin wrote:
 >In these days of corporate malfeasance scandal coverage, you'd think that
Verisign's tactics would have whetted the appetite of some bright
investigative reporter for one of the major publications.
For all that I'm critical of wildcards in TLDs -- I spoke at the
meeting yesterday, and my slides are on my Web page -- I don't think
there are any issues of malfeasance.  No one has been looting
Verisign's coffers, they're not cooking the books, etc.  I see three
issues:  is this technically wise, did Verisign have the right to do
this under their current contract with ICANN, and should they have such
a right.  I don't see anything resembling dishonesty.
Steve, I think that's a fair summary. They are being an aggressive 
business, and perhaps an aggressive business isn't the right steward 
for a TLD. In my "10,000 foot view," I tried to distinguish what the 
ideal should have been -- and maybe should be reflected in future 
contracts -- versus what did happen.

There's an old quote that applies to some extent, "Never attribute to 
malice what can be adequately explained by stupidity."  I'm not 
saying the contract drafters were stupid, but they were under time 
pressure, couldn't foresee future operational contingencies, etc.

Nevertheless, we may have a legal situation not completely unlike the 
recent issues with do-not-call. When a judge ruled additional 
legislation was needed, it was passed and signed in what was close to 
an all-time record. Now, Verisign has a contract, but, if they 
continue to be disruptive, there are options. It is my hope that 
Verisign will moderate.

More news coverage

[Howard C. Berkowitz]

Associated Press at 
http://www.boston.com/business/technology/articles/2003/10/08/experts_describe_site_finder_problems/

I am talking with the reporter, Ted Bridis. Associated Press 
reporters are under about as tough a deadline as TV, so it's a good 
fast first pass.

Re: News coverage, Verisign etc.

[Howard C. Berkowitz]

At 11:56 AM -0700 10/8/03, Eliot Lear wrote:
Howard C. Berkowitz wrote:

I have gotten a reasoned response from the technology editor of the 
Washington Post, and we are discussing things.  While I wouldn't 
have done it that way, he had a rational explanation of why the 
story was written the way it was, and definitely indicating there 
will be continuing coverage of the issue.  He believes there's 
always room for improving coverage.

Care to share?

Eliot
I was thinking about that, and now have a very red face. Eudora, for 
some reason (out of storage without a message?) seems to have lost 
about an hour of outbox messages.  I'm hoping to get a copy sent back 
to me.

In any event, in working with media, there's a time where some level 
of confidentiality is useful, when you are building the relationship 
and giving background.  Let me summarize that the Post initially saw 
this more as a business than technology issue, and gave Verisign its 
chance to tell its side of the story.  I believe the relevant editor 
now believes the issue is much more complex.

I'd want his permission to share the specific response.

Howard

Re: Verisign on Process

[Howard C. Berkowitz]

At 2:51 PM -0400 10/8/03, Dean Anderson wrote:
On Wed, 8 Oct 2003, Howard C. Berkowitz wrote:

 >VeriSign's vice president for its registry service. Citing concerns
 >of proprietary information and competitive advantage, he added that
 >he didn't think he could guarantee any advance notice of similar
 >changes in the future.
 Gomes' position truly bothers me if a registry, given that it meets
 the formal definition of a technical monopoly, is planning around
 competitive advantage.
This is incorrect. Verisign is not a monopoly. There are many registrars
of .net and .com domain names which compete with Verisign.
		--Dean
It is not a monopoly in its regiSTRAR function.

It is a monopoly as regiSTRY of .net and .com. It couldn't have 
inserted the wildcards if it wasn't. Having control of the TLD 
servers makes you a monopoly for that TLD.

News coverage, Verisign etc.

[Howard C. Berkowitz]

I have gotten a reasoned response from the technology editor of the 
Washington Post, and we are discussing things.  While I wouldn't have 
done it that way, he had a rational explanation of why the story was 
written the way it was, and definitely indicating there will be 
continuing coverage of the issue.  He believes there's always room 
for improving coverage.

Verisign on Process

[Howard C. Berkowitz]

Both here and in private mail, people have been talking about 
Verisign's view of the process. Unfortunately, I was only able to 
attend the afternoon part of yesterday's ICANN ISSC committee meeting.

But Declan McCullough was there, and picked up an interesting quote 
from Verisign:


By Declan McCullagh
Staff Writer, CNET News.com
http://news.com.com/2100-1038-5088128.html
Legal and policy questions were not on the agenda, and VeriSign 
representatives repeatedly objected when the discussion veered in 
that direction.

"Are we going to focus on security and stability, or usability?" 
asked VeriSign's Ben Turner, saying the committee's mandate was too 
narrow to include broader questions about Site Finder.

Stephen Crocker, one of the Internet's original architects and the 
ICANN committee's chairman, asked VeriSign why the wild card was 
introduced without giving network operators any warning. "I know for 
a fact that VeriSign has no problem finding its way to those 
(technical discussion) forums," Crocker said, referring to the 
company's ongoing participation in them.

"I don't want to go beyond the agenda," replied Chuck Gomes, 
VeriSign's vice president for its registry service. Citing concerns 
of proprietary information and competitive advantage, he added that 
he didn't think he could guarantee any advance notice of similar 
changes in the future.


Gomes' position truly bothers me if a registry, given that it meets 
the formal definition of a technical monopoly, is planning around 
competitive advantage.

Other speakers pointed out that the functionality of Sitefinder could 
be implemented at the edge, not breaking the end-to-end assumption 
and still allowing innovation. Internet Explorer, for example, has 
such functionality.

MS and VS. Reminds me of some recent wars where observers were sad 
that only one side could lose. :-)

Re: Sitefinder fan - this guy needs a clue.

[Howard C. Berkowitz]

At 9:29 AM -0700 10/8/03, Owen DeLong wrote:
On the CNET article that ZDNET seems to have copied lock stock and barrel,
he did actually identify himself as a Verisign VP in the by line.  As such,
I think this is ZD's fault, not McLaughlin's.  I don't have any love lost
for Verisign, but, I think we need to be very careful to stick to criticisms
that can stick to them.  Otherwise, they will divert attention to our false
claims and use them to prove that we are the net.kooks they are trying to
make us out to be.
Owen

In the news/media business, focused complaints to an editor (or 
ombudsman when one exists) about unfair/inaccurate attribution tend 
to get noticed.  In such a communication, identify only enough about 
the story that it's clear an attribution was omitted or only one side 
is being presented.

10,000 foot view of DNS/Sitefinder/Verisign

[Howard C. Berkowitz]

After attending the afternoon ICANN Security & Stability Committee 
meeting, I realized that the issues involved fall into several 
related but independent dimensions.  Shy person that I am *Cough*, I 
have opinions in all, but I think it's worthwhile simply to be able 
to explain the Big Picture to media and other folk that aren't 
immersed in our field.

In these notes, I'm trying to maintain neutrality about the issues. I 
do have strong opinions about most, but I'll post those separately, 
often dealing with one issue at a time.  For those of you new to the 
media, it's often best to put things into small, related chunks.

1. Governance issues

Did Verisign have the right, regardless of technical merit, to do 
what it did without prior warning?  I'm simply saying "did they do 
anything contractually or otherwise legally forbidden", not "was it 
strongly counter to the assumptions of the Internet" or "were they 
mean and nasty."

The news/political interest here is whether any other group should or 
could have affected this, or if we need new governance mechanisms.

Has this revealed any conflict of interest issues?  To what extent 
should a registry be able to act unilaterally?  These points are 
meant to be examined here in the context of law, regulation and 
governance, as opposed to the less formal points in #2.

2. Process (slightly different than governance) issues.
--
Moving away from  the letter of their contracts, what should they 
have done (if anything)  about open comment and forming consensus? 
This is vaguely making me wonder if they had evidence of 
WMDsoops, wrong controversy.

Assume they had no requirement for prior discussion.  What, if any, 
requirements did they have for testing and validating their approach, 
given that a top-level registry is in a unique connectivity position 
with special privileges.

3. Internet architectural impact (slightly different than effects on
   innovation and/or effect on existing software).
--
I think it's reasonable to state that Sitefinder, and changes of 
"internal" behavior, violates   at least the traditional end-to-end 
and robustness principles. This  should be considered in the spirit 
of the core vs end state discussion in RFC 3439, and the 
architectural work going into midboxes.

A general question here is to what extent is it important that the 
Internet be consistent with its relatively informal architectural 
assumptions?  Even among the newer technical folk, when teaching, I 
rarely hear anyone aware of the architecture work---they think "7 
layers" is the ultimate answer [1].

   [1] I spent over five years of my life in OSI research, development and
   promotion. We may have had the answer, but, unfortunately, we never
   could articulate the question.  That is a lesson here
4. Is the Internet the Web? Are all Internet users people?
--
I don't think it's unfair to say Sitefinder is web-centric. The 
current responses may be useful for people who can interact with it. 
Apparently, there are patches that will help with mail response and 
even anti-spamming tools.

But what of other protocols, especially those intended to run without 
human intervention?  What about failover schemes that employ DNS 
non-resolution as an indication that it's time to pick an alternate 
destination?

Is the apparent trend to move from "everything over IP" to 
"everything over HTTP" a good one? _could_ it be a good one in 
well-defined subsets of the Internet?

5. Effects on innovation

Innovation and stifling innovation has come up quite a bit. If one 
looks at the End-to-End Assumption, the historic perspective is that 
the "killer apps" appear at the edges and depend on a consistent 
center (e.g., web and VoIP, the latter with a QoS-consistent center 
[2]). Development in the core tends to be more evolutionary and 
subject to discussion (e.g., CIDR).  Other development in the core 
tends to be with the implementations (e.g., faster routers and lines).

   [2] Remember that the access links to an ISP usually aren't the QoS
   problem.  Once you get to the POP, voice and other delay-critical
   services can go onto VPNs or other QoS-engineered alternatives to
   the public Internet.
Verisign says Sitefinder is innovative, and let's assume that it is. 
But, if so, it's an innovation in the core, which is not the 
"time-proven way". When I speak of time-proven, I certainly don't 
mean that there isn't innovation -- this message did NOT reach you 
over a 56 Kbps line between IMPs.

Internet Explorer, for example, has a means of dealing with domain 
typos, but it is contrary to the way Sitefinder does things.  IE also 
does it at the edge.  How do we deal with potential commercial wars 
between the edge and core as far as competition for inn

Re: Verisign's public opinion play

[Howard C. Berkowitz]

At 10:59 AM -0400 10/7/03, William Allen Simpson wrote:
"Howard C. Berkowitz" wrote:



 > I hope to get to at least part of the ICANN meeting


I think I'll have myself organized enough to get there for the 
afternoon part of the meeting. Wish they had said if there was a 
working lunch.

In the interest of good sleep for the members of NANOG, if you buy 
Sears' otherwise very nice 6-gallon 2" hose shop-vac, be aware that a 
cat jumping on top of it will turn it on.  Said cat and his/her 
associates immediately conclude it is a dangerous carnivore, and rush 
at high speed across my occupied bed.

Thinking about Verisign, while dealing, in the middle of the night, 
with both a tongue bath from a cat seeking reassurance, as well as a 
backache from an overly aggressive attack on my lawn, does not make 
for being well rested.

Has anyone been able to get the RealVideo stream from the meeting? I 
can't seem to connect.

Re: VeriSign list (was: sitefinder ...)

[Howard C. Berkowitz]

At 9:27 AM -0400 10/7/03, William Allen Simpson wrote:
Mark Kosters wrote:
 In the interest in gaining more community review and comment, a discussion
 list has been setup to discuss factually-based technical issues
 and solutions surrounding the operational impact of wildcards in
 top-level domains on Internet applications.
We already have such mailing lists.

 1) for technical "specification of message formats, message handling,
and data formats used for DNS client-server and server-server
communication": http://ops.ietf.org/lists/namedroppers/
A lively moderated discussion of wildcards is already underway.

 2) for "[i]ssues surrounding the operation of DNS, recommendations
concerning the configuration of DNS servers, and other issues with
the use of the protocol": http://ops.ietf.org/lists/ops-area/

 VeriSign technical people will participate in discussions that are within
 the scope for this mailing list.
VeriSign technical people should have participated elsewhere, and this
mess might have been less likely to occur.
I will not participate in a VeriSign sponsored list, as that might
give fodder for another "press release" claiming network operators and
designers had reviewed and approved the VeriSign changes.  I recommend
that others only join neutral unaffiliated discussion lists.
Judging by my troubles in signing up for it -- and, sort of as a 
matter of crochety pride, I am NOT going to go through a browser to 
sign up for a mailing list (at least run by people who ought to know 
better about email but seem to assume the World is the Web), the list 
may become moot.

I thought I was on namedroppers, but I haven't seen a message in a 
long time and will make a point of resubscribing.

Re: Verisign's public opinion play

[Howard C. Berkowitz]

At 8:13 AM -0400 10/7/03, Kee Hinckley wrote:
At 10:27 AM +0100 10/7/03, [EMAIL PROTECTED] wrote:
 >I think this list may be a very good choice of where to construct
such a response.
Are you being paid by Verisign?
A "constructed" response is the worst thing we could
do. Everyone should write their own responses in their
own words based on their own experiences or their own
skills and knowledge. That's the only way to demonstrate
that Verisign was wrong, wrong, wrong.
I have to disagree.  Verisign is playing this game with considerable 
political savvy.  Disparate responses of varying quality do not get 
the media's attention, and they play right into Verisign's hands, 
because they have characterized this as a dispute between a 
respectable, secure and reliable company against a bunch of 
scattered techies.  I don't think that sending those letters and 
writing those articles does any harm per se, however I think the 
focus should be in providing technical *and marketing* ammunition to 
ICANN and focusing our defense there.  A single organization pushing 
a message over and over is more likely to get press attention.  Note 
also that we are at a considerable disadvantage in that our 
discussions of what approach to take our taking place in public 
forums ("Hi, Verisign").  Nothing like advance warning.
True. But even if it gives some warning to Verisign, the very 
openness of the process contrasts with the way they did things -- 
and, if made clear, could be newsworthy.  Individuals speaking to the 
press, Congress, Homeland Security, etc., need not give early warning.

The other thing I think would help is to paint the picture in terms 
that the general public can understand.  Verisign can do this 
because the "benefit" is something that web users understand.  Type 
something wrong--get a search page.  Most of the drawbacks are much 
more technical.  I have an idea in this space, I'll post it later 
today. The other thing we should focus on is process.  Verisign is 
claiming that we fight innovation and commercialization of the 
internet (pretty wacko, given the business we are in).  The fact of 
the matter is that there are established procedures for innovating 
the core technology of the internet, and they didn't follow them. 
We need to push the fact that they didn't just break the internet, 
they broke the rules, and this "innocent company being held back by 
techies" ploy is a bunch of garbage.
Exactly. The most fundamental problem of education here is that the 
Internet is more than the Web, even for nontechnical users. I'm 
certainly not privy to Verisign's business plans, but Sitefinder 
seems to have an inherent assumption that Web browsing is all anyone 
wants to do.

Even staying with the Web, there may well be ways the Verisign-style 
TLD wildcard could carry supplemental information that doesn't break 
existing software but could carry optional redirection information 
that could work with such things as content distribution or failover, 
yet not break anything. I freely admit that I am not a DNS designer 
but have an operational-level understanding; if I do make design 
claims, it's in routing and closely related management. I do have a 
few thoughts.

Perhaps a lifetime supply of "Be conservative in what you send, be 
liberal in what you accept" T-shirts might be commissioned by 
Verisign.

I'm not sure if it helps in this argument to rehash all the other 
problems with Verisign (like, how they managed to take the 
guaranteed cash cow of domain registration and manage it so 
insecurely and with such poor customer service that we all ran 
quickly to other registrars).  Certainly it would be good to counter 
their public image, but it probably should be done separately from 
this issue.
--


:-) there WAS a "cash cow" in an old Nortel commercial; maybe we 
should see if, like myself, she's an ex-Nortel employee that could 
join the effort. Ah, corporate speak...I wasn't laid off, or even 
downsized or rightsized. My boss (a great person) sadly call me to 
tell me, in Approved Nortelspeak, that I had been "optimized".  Great 
flashback to George Orwell.

Re: Verisign's public opinion play

[Howard C. Berkowitz]

At 10:27 AM +0100 10/7/03, [EMAIL PROTECTED] wrote:
 >I think this list may be a very good choice of where to construct
such a response.
Are you being paid by Verisign?
A disclaimer seems appropriate -- right now, I'm being only 
occasionally paid for consulting by clients not having anything to do 
with Verisign.

A "constructed" response is the worst thing we could
do. Everyone should write their own responses in their
own words based on their own experiences or their own
skills and knowledge. That's the only way to demonstrate
that Verisign was wrong, wrong, wrong.
A "constructed" response is pure politics and only
demonstrates that a certain opinion is shared by
a bunch of people with no basis in fact. We don't
need to trumpet our opinions; we need to document and
publish the facts of the matter.
And this list is definitely not the place to
discuss writing a letter of protest. If political
activity is your bag, then try http://www.meetup.com
Writing a "constructed" response, I would agree, should either come 
through an already existing group (e.g., IETF/IAB), or from an ad hoc 
organization, which, in this context, MUST have a truly open process.

That being said, writing something, even as an individual, which 
strikes the interest of news media is not necessarily a skill 
everyone here has. I do have some background in this, and would be 
happy to work with a team, and with individuals to the extent my time 
permits.  It's important to get this out of a perspective of 
"regulator versus poor persecuted Verisign". versus technically 
perfect analyses, a product of an open process, that are 
incomprehensible or at least uninteresting to a nonspecialist 
reporter, Congressional staffer, etc.

When you start to write anything, may I suggest one of the things you 
have to keep at the front of your mind is that many of your audience, 
outside the engineering community, equate the Web and the Internet. 
This isn't stupidity, it's lack of knowledge.  They have to realize 
that an ISP can be the point of access to non-public yet critical 
services, such as being the entry point for VPNs for anything from 
credit authorization to secure medical reporting.

I hope to get to at least part of the ICANN meeting -- unfortunately, 
I had an uncomfortable night -- both a cat circus on the bed and 
using some unfamiliar muscles yesterday and resulting in a painful 
back -- so I'm now running on 3-4 hours of sleep.  Ah, for the days 
when the Internet was young and I could get by with that much 
sleep...;-)

Trying to subscribe to Sitefinder list

[Howard C. Berkowitz]

Well, I've been trying. I got a double opt-in that gave me a deadline 
to respond of 5AM Wednesday. I replied.

No confirmation.

Tried to post (crossposted to NANOG).

Got error message telling me I was not yet on the list.  Of course, 
with the apparent assumption the Internet is the Web, the first 
directions were to use a browser. Another option was to respond with 
a token in the message, a common enough procedure for mailing lists.

I didn't read the fine print well enough.  The first time, I 
discovered that the token had "confirm no" in it.  Removed "no".

Reread instructions.  Just removing wasn't enough. Had to edit it to 
"confirm yes".

Is there something wrong with the user friendliness of this picture, 
assuming that people actually use something other than a web browser, 
shocking as that might be for a m-a-i-l-i-n-g  l-i-s-t?

g.

Architectural issues involving Sitefinder & related functions

[Howard C. Berkowitz]

(since I haven't gotten back my enrollment confirmation, it seemed 
appropriate to crosspost this to NANOG.  While I will address 
Sitefinder, there are broader architectural and operational issues).

Let me assume, for the sake of this discussion, that Sitefinder is an 
ideal tool for the Web user, helping with the problem of 
not-quite-correct URLs.  Given that, I'll stipulate in this 
discussion that the implementation of Sitefinder, along with the .com 
and .net wildcards that lead to it for unresolved domains, is a true 
benefit for the Web user.

The Internet, however, is more than the World-Wide Web. It seems only 
logical to be able to discuss Sitefinder in two contexts:

  1. Where it becomes the default, as with the recent Verisign
 wildcards
  2. Where it is reached in some other manner.

My architectural concern is defining a way in which context #1 serves 
the _non-Web_ services of the Internet.  If DNS were purely an 
information service for Web users, the architectural conflict would 
go away, and only commercial and policy issues remain.

I would hope that within the scope of the Sitefinder discussion list, 
or alternatively in another forum, is an approach to reconciling the 
IP-level DNS such that it continues to serve non-Web applications.

Is there disagreement that Sitefinder provides no functionality to 
SMTP trying to deliver to an unresolved domain? To a user who 
mistypes the name of an FTP site and does not intend to use a Web 
browser?

What about failover schemes for non-HTTP cooperative research across 
the Internet, where the inability to resolve a host name (assume that 
cached records have a zero lifetime) triggers selection of an 
alternate server?

Seriously, technical people at Verisign may have thought about this 
and actually have suggestions. They may be very good ones, but, 
judging on the reactions to the Sitefinder deployment, it might be 
well to discuss them in open technical forums before a change is made.

I'm really not trying to make it a matter of personalities, but there 
have been public statements by Verisign executives that such a 
process inhibits innovation.  If Verisign policy is that as operator 
of .com and .net, it has the right to make unilateral changes, I 
think that needs to be clear to all concerned. I recognize that a 
number of independent parties suggest that the ICANN contract does 
not explicitly prohibit such unilateral action.

Ironically, I worked with the original founders of Network Solutions, 
and almost was a principal back when it was a couple of rooms in 
McLean. Gary Desler, the founder  and a fine engineer, always used to 
say "there is no technical solution to a management problem".  In the 
current context, I simply want to know the rules for the playing 
field.

Re: Verisign's public opinion play

[Howard C. Berkowitz]

At 11:15 PM -0400 10/6/03, Brian Bruns wrote:
Wish someone who was good with the clue-axe would take a swing at these
dolts.
We all know they are crying babies because their new method of profit was
shut down.
Now, the interesting question will be, how can we prevent them from adding
sitefinder again?
/* begin Karnak the Magnificent soothsaying

Next, they will put an "improvement" into reverse DNS.  Whenever 
there's no corresponding domain, it will take you to rednifetis.com.

Baghdad Bob, fresh from "there is no tank behind me", will be the new 
spokesman.

/* end sooth

You know, I almost looked to see if rednifetis.com is assigned, and 
decided I don't want to know.

David McGuire article on Verisign 10/4/2003

[Howard C. Berkowitz]

Let me begin with appropriate disclaimers and identifiers. While in 
college in 1966-1967, I was a part-time science writer for The 
Washington Post, so have some familiarity with the news process. At 
the present time, I am an independent consultant in networking and 
medical computing, with experience including Internet operational 
design. With respect to the latter, I have four published books, 
including one on ISP design:  _Building Service Provider Networks_ 
(Wiley).  I am a participant in the Internet Engineering Task Force 
and North American Network Operators' Group. I have no financial 
interest in Verisign or its competitors.

My concern is first with journalistic balance with respect to 
sources, and second with technical inaccuracy.  The article quotes a 
Verisign executive, as well as an executive of a firm with a 
commercial offering similar to Verisign's Sitefinder process.In 
contrast,  the Post cited "the close-knit group of engineers and 
scientists who are familiar with the technology underpinning the 
Internet" without naming a single name of an acknowledged expert on 
the Domain Name System, the Internet function that translates 
human-oriented names to computer-oriented Internet addresses. It 
would be simple to find recognized professionals with no financial 
interest in the type of redirection from Verisign and Paxfire.

Balanced reporting should cover both sides of the story.  There are a 
great may individuals and firms that were adversely affected by 
Verisign's action, and considerable sentiment in the worldwide 
Internet engineering community that the Verisign action was 
technically unsound, and in a manner that can be demonstrated 
objectively, interfered with the normal operations of the Internet.

While I wouldn't quite call the article a Verisign press release, I'm 
appalled either that Mr. McGuire failed to obtain opinion from 
independent, financially disinterested individuals, or, 
alternatively, that the editorial staff removed such material.

Let me summarize some of the major operational concerns, and not get 
into the governance issues between Verisign and ICANN.  Strong 
arguments can be made that adding the wildcard (i.e., that which 
causes any undefined domain to be redirected to Sitefinder) to .com 
and .net breaks the operational and even protocol aspects of DNS. A 
great many network security tools, especially spam filters, depend on 
checking if domains are undefined. There is a specific DNS protocol 
message for undefined domain, which the wildcard defeats.

Beyond security, the wildcards have an indirect effect of potentially 
slowing electronic mail or causing it to be dropped. One thing that 
Verisign seemed not to consider is that the Internet is more than the 
Web, and mail agent redirection to Sitefinder provides absolutely no 
value to the mail-using Netizen.

Here's the problem.  Let's say I misaddress a piece of mail to 
foo.com, which I shall assume is a nonexistent domain.  When an ISP 
first tries to deliver it without the DNS wildcards, when it 
discovers there is no such domain, it will treat that as an error, 
usually returning the mail to sender with an appropriate error 
message.

With wildcards, however, an unmodified SMTP agent will get back an 
address (Sitefinder) and try to set up a SMTP session with it. At 
best, it will discover that Sitefinder does not support mail exchange 
and treat the message as undeliverable, again returning it.

It's more likely, however, that the SMTP software will decide that 
since it can find foo.com (with sitefinder's address), a temporary 
error is interfering with delivery. It will requeue the message for 
retry.  Typically, mail agents try to redeliver for several days, and 
may or may not return intermediate warning messages.

We now have the effects:

  --ANY mail to an incorrectly spelled name gets added to the outgoing
mail queue for retry, increaasing queue length.  Doing so:
-- slows down mail delivery due to the need for repeatedly
   processing mail that will never be delivered
-- consumes queue storage resources and increases ISP costs,
   which may be passed on to the end user
  --Inconveniencing the user, who, if they received a prompt error
notification, might discover they spelled an address incorrectly
and simply need to correct the message and resend it.  With the
wildcards, days may elapse before the sender even knows there
is a problem.
--
Howard C. Berkowitz
5012 25th Street South
Arlington VA 22206
(703)998-5819 voice
(703)998-5058  fax (alas, sometimes poorly operated by "helpful" cat)

ICANN requirement for "information refreshing"?

[Howard C. Berkowitz]

I just received an email from Verisign customer service requesting I 
"refresh my information:" on an active domain that doesn't expire 
until 2004.  My concern is that the request specifically stated ICANN 
required them to do this.

On searching the ICANN-Verisign contract at the ICANN site, I could 
find no requirement for refreshing. I'm concerned this may be a 
covert marketing activity, since the web page for "refreshing" very 
easily could have led me into buying services from Verisign. This 
seems to be of operational interest to service providers hosting 
domains, if Verisign/Netsol can confuse people into shifting their 
service to them.

Am I completely off base here? Is there some ICANN requirement I've missed?

RE: [off topic]Re: NANOG costs

[Howard C. Berkowitz]

>Howard C. Berkowitz said,
>>Chinese poutine. The mind reels.
>
>Wrong Province, but not so far off really.
>
>While your eating your imported poutine you can visit the Feng Shui Research
>Center around the corner from the Hotel on West Beaver Creek.
>
>-R


It's creeping across the nation.  Perhaps Agriculture Canada is 
looking at the wrong threat, with dedicated chickens standing sentry 
against West Nile Virus from the south.

Re: [off topic]Re: NANOG costs

[Howard C. Berkowitz]

>On Wed, 10 Apr 2002, Douglas A. Dever wrote:
>
>>
>>  Previously, Jim Mercer ([EMAIL PROTECTED]) wrote:
>>  >
>>  >
>>  > minor quibble.
>>  >
>>  > on http://www.nanog.org/mtg-0206/index.html, it says:
>>  > "Richmond Hill, Ontario, CN"
>
>iso-3166 says cn is china, that might further than I'm willing to travel
>for nanog.
>

Chinese poutine. The mind reels.

RE: Best provider to use ?

[Howard C. Berkowitz]

>Of all the places to try to get away with this type of thing 
>It's like {insert your favorite comparision here}


A modernized version of Chief Joseph and the Trail of Tiers?

>
>
>At 05:48 PM 4/6/2002 -0500, Daniel Golding wrote:
>
>>Even better...the anonymous trolls!
>>
>>- Dan
>>
>>>  -Original Message-
>>>  From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On
>>>  Behalf Of [EMAIL PROTECTED]
>>>  Sent: Saturday, April 06, 2002 12:30 PM
>>>  To: [EMAIL PROTECTED]
>>>  Cc: [EMAIL PROTECTED]
>>>  Subject: Re: Best provider to use ?
>>>
>>>
>>>
>>>
>>>  On Sat, 6 Apr 2002, [EMAIL PROTECTED] wrote:
>>>
>>>  > Out of the Tier 1s who is the best to use ?
>>>  >
>>>  > Thanks.
>>>
>>>  Please don't feed the trolls...
>>>
>>>  --
>>>  Yours,
>>>  J.A. Terranson
>>>  [EMAIL PROTECTED]
>>>
>>>  If Governments really want us to behave like civilized human
>>>  beings, they should give serious consideration towards
>>>  setting a better example: Ruling by force, rather than
>>>  consensus; the unrestrained application of unjust laws (which
>>>  the victim-populations were never allowed input on in the
>>>  first place); the State policy of justice only for the rich and
>>>  elected; the intentional abuse and occassionally destruction
>>>  of entire populations merely to distract an already apathetic
>>>  and numb electorate... This type of demogoguery must surely
>>>  wipe out the fascist United States as surely as it wiped out
>>>  the fascist Union of Soviet Socialist Republics.
>>>
>>>  The views expressed here are mine, and NOT those of my
>>>  employers, associates, or others.  Besides, if it *were* the
>>>  opinion of all of those people, I doubt there would be a
>>>  problem to bitch about in the first place...
>>>  
>>>
>>>