Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-04-02 Thread Joe Greco
 On 03/30/2012 07:41, Joe Greco wrote:
  On 3/29/2012 7:01 AM, Joe Greco wrote:
  On 3/28/2012 1:59 PM, Mark Felder wrote:
  FreeBSD 8-STABLE, 8.3, and 9.0 are untested
 
  As much as I'm sensitive to your production requirements, realistically
  it's not likely that you'll get a helpful result without testing a newer
  version. 8.2 came out over a year ago, many many things have changed
  since then.
 
  Doug
 
  So you're saying that he should have been using 8.3-RELEASE, then.
 
  That isn't what I said at all, sorry if I wasn't clear. The OP mentioned
  9.0-RELEASE, and in the context of his message (which I snipped) he
  mentioned 8-stable. That's what I was referring to.
  
  And since both the poster and I made it clear that this doesn't seem
  to be a case of it fails reliably on a machine of your choosing,
  just installing random other versions and hoping that it's going to
  cause a fail ... well, let's just say that doesn't make a whole lot
  of sense.  Or at least it's a recipe for a hell of a lot of busywork,
  busywork not guaranteed to return any sort of useful result.
 
 And since you can't reliably reproduce the problem, how do you expect us
 to? I understand that these sorts of bugs are difficult/annoying, etc.
 Been there, done that.

Nobody expected you to.  We're trying to figure out any commonalities
that might exist; these may serve to help shed light on where the
problem lies.

The interesting thing is that I took it and looked at it and came to a
conclusion that might have been wrong, though I think the trail of
reasoning I used was itself reasonable, given my exceedingly small (one
example of problem) sample size.

Mark's able to actually *reproduce* the problem on separate installs
and with circumstances that are at least somewhat different than what
my theory involved, though it is not quite possible to rule out some
sort of corruption.

Since I have to *assume* that many sites run some sort of FreeBSD on
their VMware gear, given that VMware actually lists it as a supported
version and VMware generally does things for profit, I am still kind
of of the opinion that this is some sort of corruption bug, one that I
triggered inadvertently, but one that Mark's environment reproduces
rather more frequently.  That just seems so unlikely, but more unlikely
things have come to pass, so I'm holding onto it as my working theory ;-)

I still plan to try to recover my broken VM from backups at some point
if time permits.

But in short, to answer your question:  I don't *care* if you can
reproduce the problem.  As a user, you can't win.  If you don't report
a problem, you get criticized.  If you report a problem but can't figure
out how to reproduce it, you get criticized.  If you can reproduce it
but you don't submit a workaround, you get criticized.  If you submit a
workaround but you don't submit a patch, you get criticized.  If you
submit a patch but it's not in the preferred format, you get criticized.

Hm.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-04-02 Thread Joe Greco
 On 4/2/2012 11:43 AM, Joe Greco wrote:
  As a user, you can't win.  If you don't report
  a problem, you get criticized.  If you report a problem but can't figure
  out how to reproduce it, you get criticized.  If you can reproduce it
  but you don't submit a workaround, you get criticized.  If you submit a
  workaround but you don't submit a patch, you get criticized.  If you
  submit a patch but it's not in the preferred format, you get criticized.
 
 I'm still not sure what you're taking as criticism. Nothing I've said
 was intended that way, nor should it be read that way. If you feel that
 you've been criticized by others in the manner you describe, you should
 probably take it up with them on an individual basis.

It certainly seemed to me that

 As much as I'm sensitive to your production requirements, realistically
 it's not likely that you'll get a helpful result without testing a newer
 version. 8.2 came out over a year ago, many many things have changed
 since then.

was an unwarranted criticism for reasons that I've already explained.

Or perhaps this:

 And since you can't reliably reproduce the problem, how do you expect us
 to? I understand that these sorts of bugs are difficult/annoying, etc.
 Been there, done that.

Which would appear to be suggesting that either (or possibly both):

1) The reporter has a duty to be able to reliably reproduce the problem
   prior to reporting, and/or

2) That there was some unreasonable expectation on the reporter's part
   that you were expected to reproduce it.

I consider 1) to be ridiculous, as long as the reporter is reasonably
willing to work to resolve the issue, that should certainly be good
enough, and he's certainly been interactive enough to _my_ comments,
and 2) seems to be nowhere in sight in the reporter's comments, but
is nonetheless present in your response.

Please respect Reply-to.  Thanks.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-30 Thread Joe Greco
 On 3/29/2012 7:01 AM, Joe Greco wrote:
  On 3/28/2012 1:59 PM, Mark Felder wrote:
  FreeBSD 8-STABLE, 8.3, and 9.0 are untested
 
  As much as I'm sensitive to your production requirements, realistically
  it's not likely that you'll get a helpful result without testing a newer
  version. 8.2 came out over a year ago, many many things have changed
  since then.
 
  Doug
  
  So you're saying that he should have been using 8.3-RELEASE, then.
 
 That isn't what I said at all, sorry if I wasn't clear. The OP mentioned
 9.0-RELEASE, and in the context of his message (which I snipped) he
 mentioned 8-stable. That's what I was referring to.

And since both the poster and I made it clear that this doesn't seem
to be a case of it fails reliably on a machine of your choosing,
just installing random other versions and hoping that it's going to
cause a fail ... well, let's just say that doesn't make a whole lot
of sense.  Or at least it's a recipe for a hell of a lot of busywork,
busywork not guaranteed to return any sort of useful result.

What you suggest is a fine solution for My ASUS Sempron box fails 
when I do X! -- in such a case, Try a different version of FreeBSD 
makes lots of sense.  The problem is, in a virtualization environment,
theoretically the virtual hosts are all the same sort of hardware 
(modulo any specific configuration changes of course), so when someone
presents a problem that afflicts only a percentage of their VM's, it
is important to keep in mind that you are not interacting with physical
hardware, and that reinstalling an OS on a problem VM...?  

Well, let's just say I like real hardware better for many reasons.

In the meantime, it's unrealistic to tell people to use supported
releases, to wait fifteen months between releases, and then to criticize
people complaining about problems with a supported release for using
old code.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-30 Thread Joe Greco
 On Thu, 29 Mar 2012 19:27:31 -0500, Joe Greco jgr...@ns.sol.net wrote:
 
  It also doesn't explain the experience here, where one VM basically
  crapped out but only after a migration - and then stayed crapped out.
  It would be interesting to hear about your datastore, how busy it is,
  what technology, whether you're using thin, etc.  I just have this real
  strong feeling that it's some sort of corruption with the vmfs3 and thin
  provisioned disk format, but it'd be interesting to know if that's
  totally off-track.
 
 We've ruled out SAN, but we haven't ruled out VMFS. Even FreeBSD Guests on  
 standalone ESXi servers with no SAN exhibit this crash.

 For the record, we only use thick provisioning and if it was corruption  
 I'm not sure what layer the corruption could be at. The crashy servers  
 show no abnormalities when I run either `freebsd-update IPS` or  
 `pkg_libchk` to confirm checksums of all installed programs. Now the other  
 data on there... it's not exactly verified, but our backups via rsnapshot  
 seem to prove there is no issue there or we'd have lots of new files each  
 run.

Crud, there goes part of my theory :-)

Have you migrated these hosts, or were they installed in-place and
never moved?

fwiw the apparent integrity of things on the VM is consistent with
our experience too.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
 Hi,
 
 * have you filed a PR?
 * is the crash easily reproducable?
 * are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create
 a ramdisk image using nanobsd?) and do some stress testing inside
 that?
 
 It sounds like you've established it's a storage issue, or at least
 interrupt handling for storage issue. So I'd definitely try the
 ramdisk-only boot and thrash it using lighttpd/httperf or something.
 If that survives fine, I'd look at trying to establish whether there's
 something wrong in the disk driver(s) freebsd is using. I'm not that
 cluey on ESXi, but there may be some PIC/APIC/ACPI change between 7.x
 and 8.0 which has caused this to surface.

We've seen this.  Or something that seems really like it.

We run dozens of FreeBSD VM's, many of which are 8.mumble.  We have a
scripted build environment dating back many years, so generally servers
come out in a fairly reproducible form.

After several months of smooth running, we had need to shuffle some
things around, and migrated some servers to a different datastore.
Suddenly, one particular VM, our corp Jabber server, started randomly 
disconnecting people every morning.  Some inspection showed that the
machine was running, but disk I/O in the VM was freezing up.  
Subsequent inspection suggested that it was happening during the 
periodic daily, though we never managed to get it to happen by manually 
forcing periodic daily, so that's only a theory.  Given that several 
times it appeared that one of the find commands was running, I was 
guessing that something in the thin provisioned disk image for the 
system had gone bad, but reading the entire disk with dd didn't cause 
a hang, running the periodic daily by hand didn't cause a hang, etc.

Migrating the VM to a different host and datastore did not fix the
issue.  Migrating the VM from an Opteron to a Xeon host with all the
latest ESXi 4 patches also didn't make any difference.  Migrating the
disk image from thin to full seemed to fix it, but I only gave it a
day or two, then decided there were other good reasons to reload the
VM, so I nuked the VM, which, of course, fixed it.

In the meantime, a dozen other similar VM's alongside it run just
fine.  My conclusion was that it was something specific that had gone
awry in the virtual machine, probably in the disk image, but I could
not identify it without significant digging that I had no particular
reason or inclination to do; since it appeared to be a VMware problem,
the reload it and be done with it seemed the quickest path to 
resolution.

That having been said, if anyone has any brilliant ideas about what 
would constitute useful further steps to isolate this, I can look at
recovering the faulty VM from backup and seeing if it still exhibits
the problem.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
 On 3/28/2012 1:59 PM, Mark Felder wrote:
  FreeBSD 8-STABLE, 8.3, and 9.0 are untested
 
 As much as I'm sensitive to your production requirements, realistically
 it's not likely that you'll get a helpful result without testing a newer
 version. 8.2 came out over a year ago, many many things have changed
 since then.
 
 Doug

So you're saying that he should have been using 8.3-RELEASE, then.

If you'll kindly go over to http://www.freebsd.org and look under
Latest Releases, please note that 8.2 is a production release.
If you don't want it to be a production release, then find a way
to make it so, but please don't snipe at people who are using the
code that the FreeBSD project has indicated is a current production
offering.

There are many good reasons not to run arbitrary snapshots on your
production gear.  It's unrealistic to expect people to run non-
RELEASE non-production code on their production gear.  We can have
that discussion if you don't understand that, drop me a note off-
list and I'll be happy to explain it.

Otherwise, you've told him to run a newer version, of which NONE
IS AVAILABLE, unless you're thinking 9.0, but FreeBSD has a rather
catastrophic history of point zero releases, and most clueful
admins won't run those in production without carefully measuring
the risks and benefits.  So you've basically told him to run a
newer version without any such version being realistically 
available.

WTF? 

You want people not to use releases that came out over a year 
ago?  The generally sensible solution to that is to release 
RELEASEs more than once every fourteen or fifteen months.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
 On Thursday 29 March 2012 15:42:42 Joe Greco wrote:
   Hi,
 
 Do both 32- and 64-bit versions of FreeBSD crash?

We've only seen it happen on one virtual machine.  That was a 32-bit
version.  And it's not so much a crash as it is a disk I/O hang.

The fact that it was happening regularly to that one VM, while a
bunch of other similar VM's were running alongside it without any
incident, along with the problem moving with the VM as it is moved
from host to host and from Opteron to Xeon, strongly points at 
something being wrong with the VM itself.  Our systems are built
mostly by script; I rebuilt the VM a few months ago and the
problem vanished.  The rebuilt system should have been virtually
identical to the original.  I never actually compared them though.

My working theory was that something bad had happened to the VM
during a migration from one datastore to another.  We have a really
slow-writing iSCSI server that it had been migrated onto for a little
bit, which was where the problem first appeared, I believe.  At
first I thought it was the nightly cron jobs just exceeding the iSCSI
server's capacity to cope, so we migrated the VM onto a host with
local datastores, and it remained broken thereafter.

So my conclusion was that it seemed likely that somehow VMware's 
thin provisioned disk image had gotten fouled up, and under some
unknown use case, it could be teased into locking up further I/O
on the VM.  I wasn't able to prove it.  I tried a read-dd of the
entire disk - passed, flying.  I tried several things to duplicate
the nightly periodic tasks where it seemed so prone to locking up.
They all ran fine.  But if I left the machine run, it'd do it
again eventually.

I explained it at the time to one of my VMware friends:

 But here's where it gets weird.  Three times, now, one VM - our Jabber
 server - has gone wonky in the wee early AM hours.  Disk I/O on the VM
 just locks up.  You can type at the console until it does I/O, so you
 can put in root at the login: prompt but never get a pw prompt.  My
 systems all run top from /etc/ttys and I can see that a whole bunch
 of processes are stopped in getblk.  It's like the iSCSI disk has gone
 away, except it hasn't, since the other VM's are all happily churning
 away, on the same datastore, on the same VMware host.

http://www.sol.net/tmp/freebsd/freebsd-esxi-lockup.gif

 Now it's *possible* that the problem actually happens after the 3AM cron
 run (note slight CPU/memory drop) but the Jabber implosion actually
 happens around 0530, see drop in memory%.  But the root problem at the
 VM level seems to be that disk I/O has frozen.  I can't tell for sure when
 that happens.  All three instances are similar to this.
 
 I can't explain this or figure out how to debug it.  Since it's locked up
 right now, thought I'd ping you for ideas before resetting it.

Now that was actually before we migrated it back to local datastore,
but when we did, the problem remained, suggesting that whatever has
happened to the VM, it is contained within the VM's vmdk or other
files.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
 On Thursday 29 March 2012 17:49:30 Joe Greco wrote:
   On Thursday 29 March 2012 15:42:42 Joe Greco wrote:
 Hi,
   
   Do both 32- and 64-bit versions of FreeBSD crash?
  
  We've only seen it happen on one virtual machine.  That was a 32-bit
  version.  And it's not so much a crash as it is a disk I/O hang.
 
 It almost sounds like the lost interrupt issue I've seen with USB EHCI 
 devices, though disk I/O should have a retry timeout?

That doesn't seem to fit.  Why would a perfectly functional VM suddenly
develop this problem when given a slow underlying datastore (fits so far)
but then the problem *remains* when returned to a fast local datastore,
even on a different host and architecture?  And why wouldn't the other
VM's running alongside develop the same problem?

 What does wmstat -i output?

No idea, we reloaded the VM months ago.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

2012-03-29 Thread Joe Greco
  And then there is this one with similar symptoms and a workaround:
 
  http://forums.freebsd.org/showthread.php?t=3D27899
 
 I'm now investigating those loader.conf options. I have my crashy machine
 set to use them on next boot so we'll see if it crashes now that I'm using
 LSI SAS emulated controller. If it still crashes, we'll see what happens 
 after that with those loader.conf options enabled.

Um, if I may, that's something completely different.

VMDirectPath, or PCIe passthru, is making a hardware device on a VMware
host available directly to a guest.  It'll take your LSI controller, in
the example cited, and make it unavailable to VMware ESXi, and present
it instead inside the guest environment.  You do this when you have an
app whose performance would suffer greatly when made to operate through
the indirection that a VM naturally lives in; for example, it is quite
common for FreeNAS users to pass a disk controller through to a VM guest
in order to allow a virtualized FreeNAS instance to directly manage the
physical disks.

In that case, there are some issues with ESXi and interrupt delivery to
the guest VM; virtualization doesn't actually get rid of the possibility
of ESXi problems, since the hypervisor is still ultimately involved.  It
is certainly possible that there's some common issue involving interrupt
delivery somehow, but I wouldn't get my hopes up.

It also doesn't explain the experience here, where one VM basically
crapped out but only after a migration - and then stayed crapped out.
It would be interesting to hear about your datastore, how busy it is,
what technology, whether you're using thin, etc.  I just have this real
strong feeling that it's some sort of corruption with the vmfs3 and thin
provisioned disk format, but it'd be interesting to know if that's 
totally off-track.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org


Re: Rack-Mount Server cases

2004-08-27 Thread Joe Greco
 On Friday 09 July 2004 07:19 am, Eric Crist wrote:
 
  I'm just asking opinions here, but:
 
  What do you prefer for a 2U rack mount server case?  I want to keep the
  cost down, but I want something that looks nice and is functional.  I've
  got 5 servers I'm looking at replacing existing cases on to make them
  match, as well as to free up some rack space, as some cases I currently
  own are 4U, and some are 2U.
 
 I would recommend you avoid the Antec 2U unit ( probably all Antec 
 rack-mounts if the 2U is any indication).
 
 Jay

Antec appeared to have no practical experience with mid-density servers
when they made their 2U abomination.  I had the product which appears to
have morphed slightly into the 2U26ATX300XPR.

Power supplies for a chassis should be mounted in the back.  Antec
apparently decided that there was a benefit to using a traditional ATX
style PS, but then *still* modified it to have a cord extension to the
power socket on the back.

The power supply cable won't reach to certain motherboards - in my case,
the ASUS P2B-DS.  I despise having to use extenders in a 2U case.

The brilliant internal drive arrays are enough to make me scream.

They no longer make the nice case that they once made, the 3480B, which
was a 4U short case with three front accessible bays.  We still use the
ones we have, generally retrofitting them with a 5-drive-to-3-bay SCA
converter.  They take a normal ATX MB and PS with no fuss, and the only
major complaint I ever had was that I couldn't stick a large ATX MB like
the P2B-DS in and also a hardware RAID controller in the bottom two bays,
because they'd overlap by about a quarter of an inch.  This is a function
of the RAID controller being too long, not really an Antec issue.  :-)

We no longer buy Antec products because they have made themselves
irrelevant.

If you are looking for *cheap*, I doubt you can beat the price on a
Skyhawk 2U like the IPC-2025L - usually around $100.  They are not great
units.  They are made of cheaper, thinner steel, and the screws thread
out if you're not careful (we're a power screwdriver shop).  They do use 
a regular ATX power supply, but front-mounted (ecch).  Once you discard 
the riser card that comes with them, make sure the speaker isn't shorted
out against the case, and commit to buying a real riser, however, there 
are not too many other issues.  They're basically usable after several
lessons in frustration.

If you are looking for *nice*, AIC/T-Win has some nice stuff in a 
multitude of configurations, some I like, some I don't, but all of 
which seem to have been targetted at various specific applications, 
so I can at least appreciate their thoroughness.  
http://www.aicipc.com (mfr website)

We currently use a number of their RMC2Q-XP cases out at Equinix Ashburn.
Very nice, 6 drives on trays, good airflow, keeps reasonably cool even 
though there is a pair of AMD MP 2400's on a S2469 and 6 x 15K RPM 
Fujitsu drives.  We've had one major incident in the last year which
*might* have been related to the chassis; we swapped a SCSI backplane. 
I am reasonably certain it was the drive though (but when you're flying,
you're replacing all the possible bad components).

The 2Q-XP comes with 300W, 460W, or various redundant supplies, can be
configured for normal riser card/PCI or low profile PCI, etc., etc.
Very thoughtful.

That is, of course, more pricey than you'll want for most applications,
but the design appears representative of the rest of their products (of
which we have a number).

If you're looking to buy any of these, drop me a line and I can point
you right.  You can get truly screwed buying them from some vendors.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]