date:20060609

| If you apply the needed
| 27,000 patches to qmail, you can actually get it to refuse garbage
| HELO/EHLO arguments like the '192.168.0.4' that is came in with (or
| client hosts with no rDNS, etc.);  Or you could update to a MTA which
| is supported by its author(s) still.

LMAO,

Well said.  Although I do use Qmail on a few servers, you hit the nail on the 
head!  I love my
Sendmail ;-)

Re: size of bayes db

2006-06-09 Thread Kai Schaetzl

Stefan Jakobs wrote on Fri, 9 Jun 2006 11:06:47 +0200:

> It is a dbm db! The server process ~ 80 000 Mails per Day and the bayes_seen 
> db is 5 month old.

If you count both dbs together 1 GB might be what you end up with this volume 
and no expiry. What's your "salearn --dump magic" output? That will show you 
some statistics about your db. As an example, this is a dump of a 42 MB dbm 
database. I let it expire with a threshold of 1.5 Mio. tokens or so.

0.000  0  47588  0  non-token data: nspam
0.000  0  87524  0  non-token data: nham
0.000  01231268  0  non-token data: ntokens


 With such a large db you may be better off in terms of performance by using a 
sqlized one. But expect it to take even more space. With the volume of mail you 
get I'd expire everything older than a month.

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: is there a way to block email coming from

2006-06-09 Thread Kai Schaetzl

Daryl C. W. O'Shea wrote on Thu, 08 Jun 2006 17:50:33 -0400:

> I agree that outright blocking based on dynamic IP range lists often 
> doesn't suite a particular organizations needs.  I was just pointing out 
> that some people do rely on these lists, often blindly, and that anyone 
> who is aware that they are on such a list and does nothing to avoid the 
> problems that it causes is also foolish.

All of that agreed :-)

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Ignore first upstream client in report

2006-06-09 Thread Per olof Ljungmark


Hi all,

I'm moving our content filtering from the frontend smtp-servers to an
inside relay. After a test, I can see that while it seems to work ok,
all spam is marked "Spam from local .

I have "trusted_networks" and "internal_networks" set in local.cf with 
CIDR annotation.


How can I tell SA (or amavisd) to ignore our incoming servers in the 
report and display the last external ip instead? Please note that there 
are no problems AFAIKS in the actual filtering, only in reporting:

SPAM FROM LOCAL [ip-of-our-mx] <[EMAIL PROTECTED]>

All are running postfix 2.2 or 2.3 and amavisd-new-2.4.1 / SA 3.1.3.

ext->MX smtp->smtp/amavis/SA/smtp->IMAP mail server
port25  1002425

Perhaps I'm missing the obvious here?

Thanks,
Per olof

dumb sa-update question

2006-06-09 Thread Ben Lentz


Greetings List,
When I run sa-update and download the new set of include-d rules, it 
seems the system doesn't define a few make-style variables that /do/ get 
defined when installing from CPAN. I guess the most obvious one is the 
report_contact @@CONTACT_ADDRESS@@ in 10_misc.cf. To fix this, should I 
just sed -i -e 's/@@CONTACT_ADDRESS@@/me/g' after each run of sa-update? 
How are others fixing this?


I do have report_contact defined inside my regular ruleset in 
/usr/share/spamassassin/10_misc.cf, but of course is being 
overwritten/redefined when 
/usr/share/spamassassin/updates_spamassassin_org.cf and subsequently 
/usr/share/spamassassin/updates_spamassassin_org/10_misc.cf is loaded. 
I'd just bulk move to /var/lib/spamassassin/$version/ but I've had this 
break in the past and then I'd still be stuck sed-ing the 10_misc file. 
So, as a safety, I'm using sa-update --updatedir /usr/share/spamassassin 
and redefining everything. That way, if some of the files end up 
missing, I can still run with the stock rules from the main release.

Editing Question....

2006-06-09 Thread Tom Ray

Is there a way to edit the message that appears in your mail when it's 
marked as spam.


This is the text I want to edit:

"Spam detection software, running on the system "mx02.detroitonline.com", has
identified this incoming email as possible spam.  The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email.  If you have any questions, see
[EMAIL PROTECTED] for details."

Actually, the better way to phrase this is that I want each domain owner 
able to edit their own version of that. I do virtual email hosting but I 
run each domain under it's own user name so the Admin controls their 
spam rules. I want them to be able to edit that with their own 
information. Can we do that?


--

Tom Ray
Detroit Online

http://www.detroitonline.com
Toll Free: 888-235-6817 x501
Local: 313-887-0805 x501

RE: Editing Question....

2006-06-09 Thread Martin Hepworth

Tom

Depends on what's call SA. SA will only mark the spam, any processing beyond
that is up to you..

--
Martin Hepworth 
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300

> -Original Message-
> From: Tom Ray [mailto:[EMAIL PROTECTED]
> Sent: 09 June 2006 17:17
> To: users@spamassassin.apache.org
> Subject: Editing Question
> 
> Is there a way to edit the message that appears in your mail when it's
> marked as spam.
> 
> This is the text I want to edit:
> 
> "Spam detection software, running on the system "mx02.detroitonline.com",
> has
> identified this incoming email as possible spam.  The original message
> has been attached to this so you can view it (if it isn't spam) or label
> similar future email.  If you have any questions, see
> [EMAIL PROTECTED] for details."
> 
> Actually, the better way to phrase this is that I want each domain owner
> able to edit their own version of that. I do virtual email hosting but I
> run each domain under it's own user name so the Admin controls their
> spam rules. I want them to be able to edit that with their own
> information. Can we do that?
> 
> --
> 
> Tom Ray
> Detroit Online
> 
> http://www.detroitonline.com
> Toll Free: 888-235-6817 x501
> Local: 313-887-0805 x501



**

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.   

**

Re: dumb sa-update question

2006-06-09 Thread Theo Van Dinter

On Fri, Jun 09, 2006 at 12:13:49PM -0400, Ben Lentz wrote:
> When I run sa-update and download the new set of include-d rules, it 
> seems the system doesn't define a few make-style variables that /do/ get 
> defined when installing from CPAN. I guess the most obvious one is the 

Yes.  This is a known issue:
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4862

> report_contact @@CONTACT_ADDRESS@@ in 10_misc.cf. To fix this, should I 
> just sed -i -e 's/@@CONTACT_ADDRESS@@/me/g' after each run of sa-update? 
> How are others fixing this?

The easiest method is to put the report_contact setting in your local.cf, then
you don't have to care what sa-update does. :)

-- 
Randomly Generated Tagline:
"Well, President Bush visited New York today.  Hillary Clinton, Senator
 Clinton of course, was on the Air Force One with him.  One embarrasing
 moment: Bush asked Hillary what it was like going from First Lady to
 one of a hundred.  She said: "I don't like to talk about my husband's
 sex life."" - Bill Maher on Politically Incorrect, 2001.07.10


pgp2INBrnaNeP.pgp
Description: PGP signature

Re: dumb sa-update question

2006-06-09 Thread Ben Lentz




- Original Message -
*From:* Theo Van Dinter <[EMAIL PROTECTED]>
*Sent:* 06/09/2006 12:27:22 PM
*To:* users@spamassassin.apache.org
*Subject:* dumb sa-update question




On Fri, Jun 09, 2006 at 12:13:49PM -0400, Ben Lentz wrote:
  
When I run sa-update and download the new set of include-d rules, it 
seems the system doesn't define a few make-style variables that /do/ get 
defined when installing from CPAN. I guess the most obvious one is the 



Yes.  This is a known issue:
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4862

  
report_contact @@CONTACT_ADDRESS@@ in 10_misc.cf. To fix this, should I 
just sed -i -e 's/@@CONTACT_ADDRESS@@/me/g' after each run of sa-update? 
How are others fixing this?



The easiest method is to put the report_contact setting in your local.cf, then
you don't have to care what sa-update does. :)

  

Wow, a reply from the man himself!

"I don't know, but we should probably do something." ;)

Defining in local.cf works for me, I only wish /I'd/ thought of that. 
Thanks!

Re: Mail somehow bypassing spamassassin entirely showing up in my Inbox

2006-06-09 Thread Arias Hung

On Thu, 08 Jun 2006, Daryl C. W. O'Shea delivered in simple text monotype:

On 6/7/2006 8:09 PM, Arias Hung wrote:

For the "normal timeouts", it sounds like you might be consistently having a problem with bayes expiry. Although, such a problem
isn't normally consistent AND long (time wise) when using spamd. You could try running an "sa-learn --force-expire" to see if it
helps.

<---snip--->

I tried force-expire, although it didn't seem to do much. BTW I'm using bayes
with the mysql plugins as if that makes any difference.
Everything on the mysql end of things is correct, tables created as they
should, and logging as i expect.

As for the copy_config timeouts... what kind of system load are you seeing. 10, 50, 500, or higher? The current 20 seconds alarm
is twice the original alarm timeout, but if you've got a high enough load it could still be a problem. You could increase this
value to something practically unusable, like 300, but I'd be really surprised (and would like to about) if the timeout isn't being
caused by insane load or excessive swapping.

Actually, with the load at just around 5-6 I'm notcing spam starting to seep
through. When my snapshot utility kicks in twice
a day to take snapshots of my filesystem that seems to unleash a the largest torrents of unchecked spam.

So... how much memory do you have in this machine, how much is free, and how much (hopefully none or little) swap is being used.
If swap is being used, how much of the spamd processes are being swapped out (check will the system is idle after it's been busy
for a bit).

1 gig. Here's a sample of my free/spam:

# free -m
total used free sharedbuffers cached
Mem: 1010982 28 0 94149
-/+ buffers/cache:738272
Swap: 1953516 1436

My spamc looks as follows: spamc -s 25 -t 450

Although lately I've begun experimenting with the -x flag with spamc and have
discovered the following in my procmail logs:

procmail: Program failure (74) of "/usr/bin/spamc"
procmail: Rescue of unfiltered data succeeded
From [EMAIL PROTECTED] Fri Jun 9 08:13:45 2006
Subject: XXXlXX
Folder: Inbox/new/1149866025.31852_3.radio

The program failure (74) and Rescue unfiltered data lines always preceeds
missed spam going into my Inbox.

This is with the alarm timeout value increased to 40 in spamd as you suggested.

Does this ring any bells?

Thanks for you continued help on this.

pgpEJlCCLdgOU.pgp
Description: PGP signature

Re: dumb sa-update question

2006-06-09 Thread Theo Van Dinter

On Fri, Jun 09, 2006 at 12:37:04PM -0400, Ben Lentz wrote:
> >The easiest method is to put the report_contact setting in your local.cf, 
> >then
> >you don't have to care what sa-update does. :)

> Wow, a reply from the man himself!

Hehe.  :)

> "I don't know, but we should probably do something." ;)
> Defining in local.cf works for me, I only wish /I'd/ thought of that. 

FWIW: I think what we'll do for a short-term kluge is to take
report_contact out of the update channel, so at least there'll be the
default text ("the administrator of that system").  I'm adding in a FAQ on
the wiki page (http://wiki.apache.org/spamassassin/RuleUpdates) as well.

-- 
Randomly Generated Tagline:
"A CAT scan should take less time than a PET scan.  For a CAT scan,
 they're only looking for one thing, whereas a PET scan could result in
 a lot of things."   - Carl Princi, 2002/07/19

pgpMUZuyoWR5A.pgp
Description: PGP signature

Why does this rule not hit?

2006-06-09 Thread Rosenbaum, Larry M.

Why does this rule:

body ORNL_STOCKT_N2 /H o l l y w o o d  I n t e r m e d i a t e/i
scoreORNL_STOCKT_N2 6

not hit on a message containing the following text?
(Lint shows no errors, spamd was restarted.  The string appears in the
text several times.  Every paragraph is actually a single line.)  Spam
text follows:

H o l l y w o o d  I n t e r m e d i a t e provides a proprietary
technology of Digital Intermediate services to feature filmmakers for
post-production for film mastering and restoration. This technology
gives the filmmakers total creative control over the look of their
productions. Whether shooting on film or acquiring in HD or SD video, H
o l l y w o o d  I n t e r m e d i a t e puts a powerful cluster of
digital tools at the director's disposal to achieve stunning results on
the big screen. Matchframe Digital Intermediate, a division of H o l l y
w o o d  I n t e r m e d i a t e, Inc., packages a full array of
post-production services with negative handling expertise and
cost-effective 2K digital intermediate and 35mm film out systems. The
Digital Intermediate process eliminates current post-production
redundancies by creating a single high-resolution master file from which
all versions can be made, including all theatrical and High Definition
formats. By creating a single master file with resolution hig

Red H0t News:

H o l l y w o o d  I n t e r m e d i a t e Expands the Creative Palette
for Independent Filmmakers GLENDALE, CA--(MARKET WIRE)--May 31, 2006 --
H o l l y w o o d  I n t e r m e d i a t e, Inc. A provider of digital
intermediate film mastering services, announced today that its
Matchframe Digital Intermediate division is currently providing full
digital intermediate services for Super 16MM productions.

H o l l y w o o d  I n t e r m e d i a t e, Inc. (H-Y-W-I.PK - News), a
provider of digital intermediate film mastering services, announced that
High Definition preview masters as part of its normal digital
intermediate service offerings and workflow.

Re: Editing Question....

2006-06-09 Thread Tom Ray




Martin Hepworth wrote:

Tom

Depends on what's call SA. SA will only mark the spam, any processing beyond
that is up to you..

--
Martin Hepworth 
Snr Systems Administrator

Solid State Logic
Tel: +44 (0)1865 842300

  

-Original Message-
From: Tom Ray [mailto:[EMAIL PROTECTED]
Sent: 09 June 2006 17:17
To: users@spamassassin.apache.org
Subject: Editing Question

Is there a way to edit the message that appears in your mail when it's
marked as spam.

This is the text I want to edit:

"Spam detection software, running on the system "mx02.detroitonline.com",
has
identified this incoming email as possible spam.  The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email.  If you have any questions, see
[EMAIL PROTECTED] for details."

Actually, the better way to phrase this is that I want each domain owner
able to edit their own version of that. I do virtual email hosting but I
run each domain under it's own user name so the Admin controls their
spam rules. I want them to be able to edit that with their own
information. Can we do that?



I guess I should also ask, where is this default text at?

Re: Why does this rule not hit?

2006-06-09 Thread Mike Jackson

Did you restart whatever service does the spam filtering after you added the 
rule?

- Original Message - 
From: "Rosenbaum, Larry M." <[EMAIL PROTECTED]>

To: 
Sent: Friday, June 09, 2006 12:04
Subject: Why does this rule not hit?

Why does this rule:

body ORNL_STOCKT_N2 /H o l l y w o o d  I n t e r m e d i a t e/i
scoreORNL_STOCKT_N2 6

not hit on a message containing the following text?
(Lint shows no errors, spamd was restarted.  The string appears in the
text several times.  Every paragraph is actually a single line.)  Spam
text follows:

H o l l y w o o d  I n t e r m e d i a t e provides a proprietary
technology of Digital Intermediate services to feature filmmakers for
post-production for film mastering and restoration. This technology
gives the filmmakers total creative control over the look of their
productions. Whether shooting on film or acquiring in HD or SD video, H
o l l y w o o d  I n t e r m e d i a t e puts a powerful cluster of
digital tools at the director's disposal to achieve stunning results on
the big screen. Matchframe Digital Intermediate, a division of H o l l y
w o o d  I n t e r m e d i a t e, Inc., packages a full array of
post-production services with negative handling expertise and
cost-effective 2K digital intermediate and 35mm film out systems. The
Digital Intermediate process eliminates current post-production
redundancies by creating a single high-resolution master file from which
all versions can be made, including all theatrical and High Definition
formats. By creating a single master file with resolution hig

Red H0t News:

H o l l y w o o d  I n t e r m e d i a t e Expands the Creative Palette
for Independent Filmmakers GLENDALE, CA--(MARKET WIRE)--May 31, 2006 --
H o l l y w o o d  I n t e r m e d i a t e, Inc. A provider of digital
intermediate film mastering services, announced today that its
Matchframe Digital Intermediate division is currently providing full
digital intermediate services for Super 16MM productions.

H o l l y w o o d  I n t e r m e d i a t e, Inc. (H-Y-W-I.PK - News), a
provider of digital intermediate film mastering services, announced that
High Definition preview masters as part of its normal digital
intermediate service offerings and workflow.

Re: Why does this rule not hit?

2006-06-09 Thread Theo Van Dinter

On Fri, Jun 09, 2006 at 03:04:40PM -0400, Rosenbaum, Larry M. wrote:
> Why does this rule:
> body ORNL_STOCKT_N2 /H o l l y w o o d  I n t e r m e d i a t e/i
> not hit on a message containing the following text?
> H o l l y w o o d  I n t e r m e d i a t e provides a proprietary

The double space in the body is replaced with a single space for body
rules, and your rule looks for the double space which doesn't exist so
the rule doesn't match.

-- 
Randomly Generated Tagline:
"I'm here with the two symbols of the republican party -- an elephant, and a
 big fat white guy who's threatened by change."  - Peter Griffin, Family Guy


pgp8PV1G5gjIo.pgp
Description: PGP signature

RE: Why does this rule not hit?

2006-06-09 Thread Rosenbaum, Larry M.

> From: Mike Jackson [mailto:[EMAIL PROTECTED]
> Sent: Friday, June 09, 2006 3:13 PM
>
> Did you restart whatever service does the spam filtering after you
added
> the
> rule?

Yes.  I also tested it from the command line.  It didn't hit then,
either.
 
> 
> - Original Message -
> From: "Rosenbaum, Larry M." <[EMAIL PROTECTED]>
> To: 
> Sent: Friday, June 09, 2006 12:04
> Subject: Why does this rule not hit?
> 
> 
> > Why does this rule:
> >
> > body ORNL_STOCKT_N2 /H o l l y w o o d  I n t e r m e d i a t
e/i
> > scoreORNL_STOCKT_N2 6
> >
> > not hit on a message containing the following text?
> > (Lint shows no errors, spamd was restarted.  The string appears in
the
> > text several times.  Every paragraph is actually a single line.)
Spam
> > text follows:
> >
> > H o l l y w o o d  I n t e r m e d i a t e provides a proprietary
> > technology of Digital Intermediate services to feature filmmakers
for
> > post-production for film mastering and restoration. This technology
> > gives the filmmakers total creative control over the look of their
> > productions. Whether shooting on film or acquiring in HD or SD
video, H
> > o l l y w o o d  I n t e r m e d i a t e puts a powerful cluster of
> > digital tools at the director's disposal to achieve stunning results
on
> > the big screen. Matchframe Digital Intermediate, a division of H o l
l y
> > w o o d  I n t e r m e d i a t e, Inc., packages a full array of
> > post-production services with negative handling expertise and
> > cost-effective 2K digital intermediate and 35mm film out systems.
The
> > Digital Intermediate process eliminates current post-production
> > redundancies by creating a single high-resolution master file from
which
> > all versions can be made, including all theatrical and High
Definition
> > formats. By creating a single master file with resolution hig
> >
> > Red H0t News:
> >
> > H o l l y w o o d  I n t e r m e d i a t e Expands the Creative
Palette
> > for Independent Filmmakers GLENDALE, CA--(MARKET WIRE)--May 31, 2006
--
> > H o l l y w o o d  I n t e r m e d i a t e, Inc. A provider of
digital
> > intermediate film mastering services, announced today that its
> > Matchframe Digital Intermediate division is currently providing full
> > digital intermediate services for Super 16MM productions.
> >
> > H o l l y w o o d  I n t e r m e d i a t e, Inc. (H-Y-W-I.PK -
News), a
> > provider of digital intermediate film mastering services, announced
that
> > High Definition preview masters as part of its normal digital
> > intermediate service offerings and workflow.
> >
> >

RE: Why does this rule not hit?

2006-06-09 Thread Dallas L. Engelken

> -Original Message-
> From: Theo Van Dinter [mailto:[EMAIL PROTECTED] 
> Sent: Friday, June 09, 2006 14:15
> To: users@spamassassin.apache.org
> Subject: Re: Why does this rule not hit?
> 
> On Fri, Jun 09, 2006 at 03:04:40PM -0400, Rosenbaum, Larry M. wrote:
> > Why does this rule:
> > body ORNL_STOCKT_N2 /H o l l y w o o d  I n t e r m e d 
> i a t e/i
> > not hit on a message containing the following text?
> > H o l l y w o o d  I n t e r m e d i a t e provides a proprietary
> 
> The double space in the body is replaced with a single space 
> for body rules, and your rule looks for the double space 
> which doesn't exist so the rule doesn't match.
> 

and if you dont like this behavior, you can use 'rawbody' instead of
'body' to hit it with 2 spaces between the words...

d

RE: Why does this rule not hit?

2006-06-09 Thread Rosenbaum, Larry M.

> From: Theo Van Dinter [mailto:[EMAIL PROTECTED]
> Sent: Friday, June 09, 2006 3:15 PM
> 
> On Fri, Jun 09, 2006 at 03:04:40PM -0400, Rosenbaum, Larry M. wrote:
> > Why does this rule:
> > body ORNL_STOCKT_N2 /H o l l y w o o d  I n t e r m e d i a t
e/i
> > not hit on a message containing the following text?
> > H o l l y w o o d  I n t e r m e d i a t e provides a proprietary
> 
> The double space in the body is replaced with a single space for body
> rules, and your rule looks for the double space which doesn't exist so
> the rule doesn't match.

Thanks.  That's a very handy thing to know.

The Future of Email is SQL






After considerable experimenting and thinking things through I thought 
I'd start a thread on the future of email to start planting the seeds of 
where MTA development needs to go. I'm convinced that someday soon we 
will all realize that MBOX and MAILDIR are obsolete technologies and 
that the future is going to be SQL based storage.

First - before everyone starts screaming about speed comparisons, I'm 
not going to go there. Every storage technology has it's advantages and 
disadvantages but I'm just going to say that SQL based mail storage is 
fast enough. The advantages of SQL has to do with power and not with 
speed. Those who would choose it would do so because they want to do new 
things that you can do with a database and can't do without one.

SQL has several advantages. You don't have t deal with the quirks of the 
underlying file system or OS. It takes care of all the locking issues 
and indexing and makes it so that multiple applications can seamlessly 
access the data. With an SQL backend email can be stored from the MTA, 
read from and IMAP client that accesses the same database, and the spam 
filtering engine will have access to the stored email as well.

To give you some examples of what could be done .

Suppose a spammer sends 1000 phishing spams to your users and then you 
figure out that the 1000 spams already delivered is spam. With a 
database you can do a query to retroactively delete spam that was 
already delivered to the mailboxes. This could also be used to 
retroactively delete viruses already delivered.

Spam filtering programs can lookup existing email in existing folders 
and compare it with new email already deliverd to help determine more 
accurately if a message is spam or not. For example, if the host server 
has a reputation for 100% ham then it can deliver new email without 
running it through Spam Assassin. If programs like Spamassassin can 
access existing email in existing folders it can evaluate new email 
using tricks no one has yet considered.

SQL databases allow for multiple masters and slaves and replication that 
lets you create a cluster that never fails under any conditions. It 
would be far easier to create a system that is always on and always 
backed up.

An SQL backend allows you to use a wide variety of tools, programming 
languages, operating systems in order for you to easily integrate more 
easily than non database systems.

And - this is important - once you have a database then new things that 
no one has yet thought of will be possible and new things we've never 
heard of will be developed because the new power will lend to the 
development of more tricks than you can do without database power.

My point here is - think outside the box. I'm going to be lobbying IMAP 
server developers to include SQL backends. exim could pipe data into a 
local delivery agent, or it can have features written to write directly 
to the SQL backend.

Thoughts . ?


-- 
## List details at http://www.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://www.exim.org/eximwiki/

Re: The Future of Email is SQL




My point here is - think outside the 
box. I'm going to be lobbying IMAP server developers to include SQL 
backends. exim could pipe data into a local delivery agent, or it can have 
features written to write directly to the SQL backend.Thoughts . 
?
Because I am an SQL dummy, I do have this question.  
Would aps like Mysql and Postgres be able to handle 10,000+ users with an 
average of 50 MB of email?  
 
I really don't know.
 
Also, does the body just get written to a table?
 
Enlighten me,

Re: The Future of Email is SQL







 wrote:

  
  
  
  
My point here is - think outside the box. I'm going to be lobbying IMAP
  
server developers to include SQL backends. exim could pipe data into a 
local delivery agent, or it can have features written to write directly
  
to the SQL backend.
  
Thoughts . ?
  
  Because I am an SQL dummy, I do have this
question.  Would aps like Mysql and Postgres be able to handle 10,000+
users with an average of 50 MB of email?  
   
  I really don't know.
   
  Also, does the body just get written to a table?
   
  Enlighten me,
   
  

That would be about 500 gigs of email. Fry's Electronics has drives
that size on special for $189. So - I'd say yes, should be fairly easy
to scale up to that size and beyond.

Re: Ignore first upstream client in report

2006-06-09 Thread Per olof Ljungmark


Per olof Ljungmark wrote:

Hi all,

I'm moving our content filtering from the frontend smtp-servers to an
inside relay. After a test, I can see that while it seems to work ok,
all spam is marked "Spam from local .

I have "trusted_networks" and "internal_networks" set in local.cf with 
CIDR annotation.


How can I tell SA (or amavisd) to ignore our incoming servers in the 
report and display the last external ip instead? Please note that there 
are no problems AFAIKS in the actual filtering, only in reporting:

SPAM FROM LOCAL [ip-of-our-mx] <[EMAIL PROTECTED]>

All are running postfix 2.2 or 2.3 and amavisd-new-2.4.1 / SA 3.1.3.

ext->MX smtp->smtp/amavis/SA/smtp->IMAP mail server
port25  1002425



Anyone?

Re: The Future of Email is SQL

2006-06-09 Thread Jason Marshall

That would be about 500 gigs of email. Fry's Electronics has drives that size 
on special for $189. So - I'd say yes, should be fairly easy to scale up to 
that size and beyond.


You really think one 500 gig disk is going to give you anywhere close to 
the performance you need to accomodate 500 active gigs of mailboxes?  If 
you had 30x 18 gig fibre-channel drives spread out over many controllers 
and many machines you might have half a chance of keeping up.


If you put, say, 5 of those disks on each of 6 servers, you'd still need a 
way to (reliably) aggregate those 30 disks into one large storage 
facility, and the database engine on all 6 of those servers would have to 
be aware that it was just 1/6 of the equation at all times.


There would be no redundancy at that point.  You could probably get a 7th 
server with a bunch of large slow disks and use that to back up the data 
on the real cluster.  But you'd never be able to move 'production' over to 
the backup server, there'd just be too little disk throughput for it to 
work.


And by the way, 10k users isn't a lot.  I have 3000 users, and I'd 
consider us to be a miniscule operation compared to many others out there. 
Scale this to 250k users and we're talking...


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
| Jason Marshall, [EMAIL PROTECTED] Spots InterConnect, Inc. Calgary, AB |
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

RE: Ignore first upstream client in report

2006-06-09 Thread Bowie Bailey

Per olof Ljungmark wrote:
> Per olof Ljungmark wrote:
> > Hi all,
> > 
> > I'm moving our content filtering from the frontend smtp-servers to
> > an inside relay. After a test, I can see that while it seems to
> > work ok, all spam is marked "Spam from local .
> > 
> > I have "trusted_networks" and "internal_networks" set in local.cf
> > with CIDR annotation. 
> > 
> > How can I tell SA (or amavisd) to ignore our incoming servers in the
> > report and display the last external ip instead? Please note that
> > there are no problems AFAIKS in the actual filtering, only in
> > reporting: SPAM FROM LOCAL [ip-of-our-mx] <[EMAIL PROTECTED]>
> > 
> > All are running postfix 2.2 or 2.3 and amavisd-new-2.4.1 / SA 3.1.3.
> > 
> > ext->MX smtp->smtp/amavis/SA/smtp->IMAP mail server
> > port25  1002425
> > 
> 
> Anyone?

As for SA, just make sure that your frontend servers are listed in
trusted_networks.

But the message you quote doesn't come from SA, so you may want to ask
on the Amavis list.

-- 
Bowie

Re: The Future of Email is SQL




>>That would be about 500 gigs of email. Fry's Electronics has drives 
that size on special for $189. So - I'd say yes, should be fairly easy to scale 
up to that size and beyond.
 
I believe it would be approx 200 Gigs

Re: The Future of Email is SQL

2006-06-09 Thread DAve


Marc Perkel wrote:



 wrote:


My point here is - think outside the box. I'm going to be lobbying IMAP
server developers to include SQL backends. exim could pipe data into a
local delivery agent, or it can have features written to write directly
to the SQL backend.

Thoughts . ?


We are looking at using DBMail as a mail archive for clients to pull 
lost and historic mail at a later date. But not for daily use.


Because I am an SQL dummy, I do have this question.  Would aps like 
Mysql and Postgres be able to handle 10,000+ users with an average of 
50 MB of email?  
I really don't know.
 
Also, does the body just get written to a table?
 
Enlighten me,
 

That would be about 500 gigs of email. Fry's Electronics has drives that 
size on special for $189. So - I'd say yes, should be fairly easy to 
scale up to that size and beyond.


If you are building a high performance mail server, even just a 
mailstore, you aren't buying hardware at Fry's. Our mail gateways take 
in 2gb of messages a day for 6k+ accounts. That is after we have 
rejected 70% of the connections.


Between two mail gateways and three toasters we have 14 disks that never 
stop seeking, never, 24/7/365. A consumer grade storage device would 
scream "mommy" and wet itself.


DAve

--
Three years now I've asked Google why they don't have a
logo change for Memorial Day. Why do they choose to do logos
for other non-international holidays, but nothing for
Veterans?

Maybe they forgot who made that choice possible.

Re: The Future of Email is SQL - What drives do you use?


| Between two mail gateways and three toasters we have 14 disks that never
| stop seeking, never, 24/7/365. A consumer grade storage device would
| scream "mommy" and wet itself.
|
| DAve

OK, I'm sorry for changing the subject but I have had good results with 18 and 
36 GB IBM SCSI
drives.

What do you use?

Re: Editing Question....

2006-06-09 Thread David Goldsmith

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Tom Ray wrote:
> 
> 
> Martin Hepworth wrote:
>> Tom
>>
>> Depends on what's call SA. SA will only mark the spam, any processing
>> beyond
>> that is up to you..
>>
>> -- 
>> Martin Hepworth Snr Systems Administrator
>> Solid State Logic
>> Tel: +44 (0)1865 842300
>>
>>  
>>> -Original Message-
>>> From: Tom Ray [mailto:[EMAIL PROTECTED]
>>> Sent: 09 June 2006 17:17
>>> To: users@spamassassin.apache.org
>>> Subject: Editing Question
>>>
>>> Is there a way to edit the message that appears in your mail when it's
>>> marked as spam.
>>>
>>> This is the text I want to edit:
>>>
>>> "Spam detection software, running on the system
>>> "mx02.detroitonline.com",
>>> has
>>> identified this incoming email as possible spam.  The original message
>>> has been attached to this so you can view it (if it isn't spam) or label
>>> similar future email.  If you have any questions, see
>>> [EMAIL PROTECTED] for details."
>>>
>>> Actually, the better way to phrase this is that I want each domain owner
>>> able to edit their own version of that. I do virtual email hosting but I
>>> run each domain under it's own user name so the Admin controls their
>>> spam rules. I want them to be able to edit that with their own
>>> information. Can we do that?
>>>
>>> 
> I guess I should also ask, where is this default text at?

The default 'report' template is defined in the 10_misc.cf file.

David Goldsmith

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.3rc2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEieXY417vU8/9QfkRApy3AKCA3zX5mFEE/q13kwUDNY1lCTzkXQCfRi/c
B5yB1QEQHVrzfZOY2NM/SJM=
=agx2
-END PGP SIGNATURE-

Re: The Future of Email is SQL - What drives do you use?

2006-06-09 Thread Jason Marshall


OK, I'm sorry for changing the subject but I have had good results with 18 and 
36 GB IBM SCSI
drives.

What do you use?


I generally use Seagate.  Used to use IBM/Hitachi and Fujitsu.  Still 
would if they were easier to find in stock around here.  Have used 
Quantums, and long long ago Micropolis.  Both should be avoided...


Found a pile of new 4 gig SCSI disks on ebay, and have been using those 
for linux system disks for the last couple years.  They're SCSI 2, great 
for booting from, they last forever...  Ah, remember the good old days 
when you could buy a disk that was the right size for the job, not 1400x 
bigger than what you need...


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
| Jason Marshall, [EMAIL PROTECTED] Spots InterConnect, Inc. Calgary, AB |
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: The Future of Email is SQL

2006-06-09 Thread Logan Shaw


On Fri, 9 Jun 2006, Marc Perkel wrote:

 wrote:


Because I am an SQL dummy, I do have this question.  Would aps like Mysql 
and Postgres be able to handle 10,000+ users with an average of 50 MB of 
email?  I really don't know.

 Also, does the body just get written to a table?


That would be about 500 gigs of email. Fry's Electronics has drives that size 
on special for $189. So - I'd say yes, should be fairly easy to scale up to 
that size and beyond.


That seems like a red herring considering 500GB of e-mail
still takes about 500GB of disk space whether it's a database
engine or the kernel's filesystem driver writing it to disk.
Yes, there will be differences in storage efficiency, but
they're minor.

And to answer 's question, yeah, the body would be written
to a table.  Because when you store things in a SQL database,
everything you store is written into a table.  For what it's
worth, you can store stuff like e-mail in a BLOB (Binary Large
OBject) or a similar type of field that is specifically meant
to be able to handle data of arbitrary length.

  - Logan

RE: The Future of Email is SQL

2006-06-09 Thread Gary W. Smith









Marc, 

 

We have had to approach this in a similar
fashion.  We have large volume email accounts under cyrus as well as a
custom spam filtering system (behind SA).  Here is the approach we did.

 

We have cyrus setup on multiple partitions
based upon the directories.  This allows us to upgrade individual sets of
directories based on load.  Though this approach isn’t the best it
works well.  We have over 500gb on a single server.

 

We have had a problem with spam, just like
everyone else.  The spam no longer hits many of our user accounts. 
Instead it is inserted into a database and they are sent a daily digest (or
they can look it up).  We started with a simple set of tables which in
testing grew very large (5gb) with our test set.  In production this would
have been 100gb.  We only retain 15 days…

 

To accomplish this we looking into
splitting up the data just like we did for cyrus.  We broke that single
table down into x tables (x being defined as a tweakable number – for prod
we use 200).  We use random allocation to put an email into one of the tables. 
This becomes important as the data is separated from some basic information
which allows us to keep these files on x number of spindles or network devices
and managed in a much simpler fashion.

 

We have been looking at imap based on db’s
as their backends and are still in the air on them as they don’t meet all
of our requirements right now (in their stable form) but going forward I think
that SQL emails might become our designed transport.  Our SQL servers for
handling this are clustered machines, each with about 600gb disk space, under
linux-ha and DRBD.  This is also then replicated to a matching offsite
database cluster.

 

I believe that there is a use for a
technology focused more around databases (actually there are some right now
just very specific to themselves and not really configurable) that will replace
existing named systems (such as uw-imap and cyrus).  I would guess that
these tools themselves might start that implementation within themselves (hint
hint) so we don’t have to turn to the alternative imap systems.

 

Anyway, this stuff exists and some of us
use certain concepts already applied.  Implementation is simple in many
cases.

 

 











From: Marc Perkel [mailto:[EMAIL PROTECTED] 
Sent: Friday, June 09, 2006 1:19
PM
To: users@spamassassin.apache.org
Subject: The Future of Email is
SQL



 

After considerable experimenting and thinking things through I thought I'd start a thread on the future of email to start planting the seeds of where MTA development needs to go. I'm convinced that someday soon we will all realize that MBOX and MAILDIR are obsolete technologies and that the future is going to be SQL based storage. First - before everyone starts screaming about speed comparisons, I'm not going to go there. Every storage technology has it's advantages and disadvantages but I'm just going to say that SQL based mail storage is fast enough. The advantages of SQL has to do with power and not with speed. Those who would choose it would do so because they want to do new things that you can do with a database and can't do without one. SQL has several advantages. You don't have t deal with the quirks of the underlying file system or OS. It takes care of all the locking issues and indexing and makes it so that multiple applications can seamlessly access the data. With an SQL backend email can be stored from the MTA, read from and IMAP client that accesses the same database, and the spam filtering engine will have access to the stored email as well. To give you some examples of what could be done . Suppose a spammer sends 1000 phishing spams to your users and then you figure out that the 1000 spams already delivered is spam. With a database you can do a query to retroactively delete spam that was already delivered to the mailboxes. This could also be used to retroactively delete viruses already delivered. Spam filtering programs can lookup existing email in existing folders and compare it with new email already deliverd to help determine more accurately if a message is spam or not. For example, if the host server has a reputation for 100% ham then it can deliver new email without running it through Spam Assassin. If programs like Spamassassin can access existing email in existing folders it can evaluate new email using tricks no one has yet considered. SQL databases allow for multiple masters and slaves and replication that lets you create a cluster that never fails under any conditions. It would be far easier to create a system that is always on and always backed up. An SQL backend allows you to use a wide variety of tools, programming languages, operating systems in order for you to easily integrate more easily than non database systems. And - this is important - once you have a database then new things that no one has yet thought of will be possible and new things we've never heard of will be developed becaus

Re: The Future of Email is SQL

On Fri, Jun 09, 2006 at 02:25:52PM -0600,  wrote:
> 
> My point here is - think outside the box. I'm going to be lobbying IMAP 
> server developers to include SQL backends. exim could pipe data into a 
> local delivery agent, or it can have features written to write directly 
> to the SQL backend.
> 
> Thoughts . ?
> 
> Because I am an SQL dummy, I do have this question.  Would aps like Mysql and 
> Postgres be able to handle 10,000+ users with an average of 50 MB of email?  

There are people happily running PostgreSQL with terrabyte databases.
It's really a question of how much concurrency you need.

One nice thing about databases is they make it possible to do things
like partition your tables by month/week/whatever. You can then move
older data onto larger partitions that use slower, cheaper drives.
-- 
Jim C. Nasby, Database Architect[EMAIL PROTECTED] 
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

Re: The Future of Email is SQL





Gary,

I'm trying to introduce the idea of a MySQL backend to Timo over at
Dovecot. He has done a little work in that direction already. But - I'm
throwing this idea out there right now just to get people thinking. I'm
hoping that in the next year as people think this through that some
serious development will occur. I think that as people say AH HA that
development will progress.

Gary W. Smith wrote:

  
  

  
  
  Marc, 
   
  We have had
to approach this in a similar
fashion.  We have large volume email accounts under cyrus as well as a
custom spam filtering system (behind SA).  Here is the approach we did.
   
  We have
cyrus setup on multiple partitions
based upon the directories.  This allows us to upgrade individual sets
of
directories based on load.  Though this approach isn’t the best it
works well.  We have over 500gb on a single server.
   
  We have had
a problem with spam, just like
everyone else.  The spam no longer hits many of our user accounts. 
Instead it is inserted into a database and they are sent a daily digest
(or
they can look it up).  We started with a simple set of tables which in
testing grew very large (5gb) with our test set.  In production this
would
have been 100gb.  We only retain 15 days…
   
  To
accomplish this we looking into
splitting up the data just like we did for cyrus.  We broke that single
table down into x tables (x being defined as a tweakable number – for
prod
we use 200).  We use random allocation to put an email into one of the
tables. 
This becomes important as the data is separated from some basic
information
which allows us to keep these files on x number of spindles or network
devices
and managed in a much simpler fashion.
   
  We have been
looking at imap based on db’s
as their backends and are still in the air on them as they don’t meet
all
of our requirements right now (in their stable form) but going forward
I think
that SQL emails might become our designed transport.  Our SQL servers
for
handling this are clustered machines, each with about 600gb disk space,
under
linux-ha and DRBD.  This is also then replicated to a matching offsite
database cluster.
   
  I believe
that there is a use for a
technology focused more around databases (actually there are some right
now
just very specific to themselves and not really configurable) that will
replace
existing named systems (such as uw-imap and cyrus).  I would guess that
these tools themselves might start that implementation within
themselves (hint
hint) so we don’t have to turn to the alternative imap systems.
   
  Anyway, this
stuff exists and some of us
use certain concepts already applied.  Implementation is simple in many
cases.
   
   
  
  
  
  
  From: Marc
Perkel [mailto:[EMAIL PROTECTED]] 
  Sent: Friday, June 09,
2006 1:19
PM
  To:
users@spamassassin.apache.org
  Subject: The Future of
Email is
SQL
  
   
  
  After considerable experimenting and thinking things through I thought 
  I'd start a thread on the future of email to start planting the seeds of 
  where MTA development needs to go. I'm convinced that someday soon we 
  will all realize that MBOX and MAILDIR are obsolete technologies and 
  that the future is going to be SQL based storage.
   
  First - before everyone starts screaming about speed comparisons, I'm 
  not going to go there. Every storage technology has it's advantages and 
  disadvantages but I'm just going to say that SQL based mail storage is 
  fast enough. The advantages of SQL has to do with power and not with 
  speed. Those who would choose it would do so because they want to do new 
  things that you can do with a database and can't do without one.
   
  SQL has several advantages. You don't have t deal with the quirks of the 
  underlying file system or OS. It takes care of all the locking issues 
  and indexing and makes it so that multiple applications can seamlessly 
  access the data. With an SQL backend email can be stored from the MTA, 
  read from and IMAP client that accesses the same database, and the spam 
  filtering engine will have access to the stored email as well.
   
  To give you some examples of what could be done .
   
  Suppose a spammer sends 1000 phishing spams to your users and then you 
  figure out that the 1000 spams already delivered is spam. With a 
  database you can do a query to retroactively delete spam that was 
  already delivered to the mailboxes. This could also be used to 
  retroactively delete viruses already delivered.
   
  Spam filtering programs can lookup existing email in existing folders 
  and compare it with new email already deliverd to help determine more 
  accurately if a message is spam or not. For example, if the host server 
  has a reputation for 100% ham then it can deliver new email without 
  running it through Spam Assassin. If programs like Spamassassin can 
  access existing email in existing folders it can evaluate new email 
  using tricks no one has yet considered.
   
  SQL database

Re: The Future of Email is SQL







Jim C. Nasby wrote:

  On Fri, Jun 09, 2006 at 02:25:52PM -0600,  wrote:
  
  
My point here is - think outside the box. I'm going to be lobbying IMAP 
server developers to include SQL backends. exim could pipe data into a 
local delivery agent, or it can have features written to write directly 
to the SQL backend.

Thoughts . ?

Because I am an SQL dummy, I do have this question.  Would aps like Mysql and Postgres be able to handle 10,000+ users with an average of 50 MB of email?  

  
  
There are people happily running PostgreSQL with terrabyte databases.
It's really a question of how much concurrency you need.

One nice thing about databases is they make it possible to do things
like partition your tables by month/week/whatever. You can then move
older data onto larger partitions that use slower, cheaper drives.
  


Perhaps the headers and other information that you would index be kept
in the database and the body of the message stored somewhere else,
perhaps even as files. 

I'm just trying to inspire thought and creativity here btw.

RE: The Future of Email is SQL

2006-06-09 Thread Greg Allen

-Original Message-
From: Marc Perkel [mailto:[EMAIL PROTECTED]
Sent: Friday, June 09, 2006 4:19 PM
To: users@spamassassin.apache.org
Subject: The Future of Email is SQL

Thoughts . ?

-

MS Exchange... one big Database

You can use Exmerge to do some of what you are looking to do (delete one
email to all users, export dates of email, etc.).

If someone sends 1,000 copies of the same email to all of the users on the
same organization (cc,bcc) the message is stored only once, with pointers to
it.

It may not do everything you are looking to do, not sure.

Just pointing out that Microsoft has already started down that route to some
extent and they may end up using SQL even.

Re: The Future of Email is SQL - What drives do you use?

2006-06-09 Thread DAve


 wrote:

| Between two mail gateways and three toasters we have 14 disks that never
| stop seeking, never, 24/7/365. A consumer grade storage device would
| scream "mommy" and wet itself.
|
| DAve

OK, I'm sorry for changing the subject but I have had good results with 18 and 
36 GB IBM SCSI
drives.

What do you use?



Fujitsu is nice, Seagate is better. WD and Maxtor make poor doorstops as 
they are too light, but they make a funny plunk sound when they hit water.


But it's the model of the drive and it's intended purpose that matters, 
not so much the label. Every manufacturer makes consumer grade equipment.


DAve


--
Three years now I've asked Google why they don't have a
logo change for Memorial Day. Why do they choose to do logos
for other non-international holidays, but nothing for
Veterans?

Maybe they forgot who made that choice possible.

RE: The Future of Email is SQL

2006-06-09 Thread Rob McEwen


>>MS Exchange... one big Database

Exactly...

And that is one reason why I wouldn't touch this SQL idea with a 10 foot
pole.. the fact that Exchange works this way only proves my point... I hear
all the time about Exchange servers crashing and the administrator having to
rebuild the database while the mail server is down for the next 10 hours.

The bottom line is that using a SQL DB backend as mail storage is putting
all your eggs in one basket.

I have a much simpler solution to accomplish the problem that this was idea
was originally attempting to solve... simply place the spams that are caught
in a folder on the mail server that is accessible via webmail. Then create a
separate program to periodically enumerate through the spam folder in all
the accounts on the server to delete spams over X days old.

If needed, you could still have a database with the basic info about the
spams (date received, subject line, recipients, from, message file name,
etc) to use for e-mailing "digests" to the user... and this DB's stability
wouldn't then have to be tied to the overall reliability/stability of mail
services.

Also keep in mind that SQL doesn't always mean better performance... I've
seen many web sites that deliver content dynamically from a SQL database
backend where there were noticeably large delays between page loads, for
example.

Rob McEwen
PowerView Systems
[EMAIL PROTECTED]

RE: The Future of Email is SQL

2006-06-09 Thread Greg Allen



> -Original Message-
> From: Rob McEwen [mailto:[EMAIL PROTECTED]
> Sent: Friday, June 09, 2006 6:16 PM
> To: users@spamassassin.apache.org
> Subject: RE: The Future of Email is SQL
>
>
>
> >>MS Exchange... one big Database
>
> Exactly...
>
> And that is one reason why I wouldn't touch this SQL idea with a 10 foot
> pole.. the fact that Exchange works this way only proves my
> point... I hear
> all the time about Exchange servers crashing and the
> administrator having to
> rebuild the database while the mail server is down for the next 10 hours.

Yup, I have worked on Exchange servers for years. 5.5 blew up all the time.
2000 not so much at all. I expect 2003 is failrly stable. But regardless, if
it does go... that group of users is down all day. I know some orgs are
using clusters on Exchange to help with that problem... but now you have a
cluster that only one guy knows how to work on. The guy who set it up. So,
if the cluster gets screwed somehow, you have to find the guy who set it up,
you then have to fix the cluster, and then the Exchange. Just shoot yourself
in the head and save some time.

I would rather use Exchange with seperate PST files for each user, and it
will let you do that. The reason most companies end up on the single HUGE
database it because Exchange requires that be down to share appointments,
tasks, etc.

Re: The Future of Email is SQL







Greg Allen wrote:

  
  
  
-Original Message-
From: Rob McEwen [mailto:[EMAIL PROTECTED]]
Sent: Friday, June 09, 2006 6:16 PM
To: users@spamassassin.apache.org
Subject: RE: The Future of Email is SQL





  
MS Exchange... one big Database

  

Exactly...

And that is one reason why I wouldn't touch this SQL idea with a 10 foot
pole.. the fact that Exchange works this way only proves my
point... I hear
all the time about Exchange servers crashing and the
administrator having to
rebuild the database while the mail server is down for the next 10 hours.

  
  
Yup, I have worked on Exchange servers for years. 5.5 blew up all the time.
2000 not so much at all. I expect 2003 is failrly stable. But regardless, if
it does go... that group of users is down all day. I know some orgs are
using clusters on Exchange to help with that problem... but now you have a
cluster that only one guy knows how to work on. The guy who set it up. So,
if the cluster gets screwed somehow, you have to find the guy who set it up,
you then have to fix the cluster, and then the Exchange. Just shoot yourself
in the head and save some time.

I would rather use Exchange with seperate PST files for each user, and it
will let you do that. The reason most companies end up on the single HUGE
database it because Exchange requires that be down to share appointments,
tasks, etc.

  

What I have in mind would be a far better database than Exchange. I'm assuming a really good database like MySQL or Oracle and I'm assuming that in the future that databases will get even better. Spamassassin has switched from DB files to MySQL and I think that kind of evolution will continue. So - I'm trying to plant that idea of a SQL future.

Re: Mail somehow bypassing spamassassin entirely showing up in my Inbox

2006-06-09 Thread Daryl C. W. O'Shea


Arias Hung wrote:

On Thu, 08 Jun 2006, Daryl C. W. O'Shea delivered in simple text monotype:



As for the copy_config timeouts... what kind of system load are you 
seeing.  10, 50, 500, or higher?  The current 20 seconds alarm is 
twice the original alarm timeout, but if you've got a high enough load 
it could still be a problem.  You could increase this value to 
something practically unusable, like 300, but I'd be really surprised 
(and would like to about) if the timeout isn't being caused by insane 
load or excessive swapping.


Actually, with the load at just around 5-6 I'm notcing spam starting to 
seep through.  When my snapshot utility kicks in twice
a day to take snapshots of my filesystem that seems to unleash a the 
largest torrents of unchecked spam.


fs snapshots would make swap effectively useless.  You'll be waiting all 
day for disk I/O.



So... how much memory do you have in this machine, how much is free, 
and how much (hopefully none or little) swap is being used.  If swap 
is being used, how much of the spamd processes are being swapped out 
(check will the system is idle after it's been busy for a bit).


1 gig.  Here's a sample of my free/spam:

# free -m
 total   used   free sharedbuffers cached
Mem:  1010982 28  0 94149
-/+ buffers/cache:738272
Swap: 1953516   1436


If you look at  'top'  I'm sure you'll see that a good portion of the 
spamd processes have been swapped out whenever you see this happen.




This is with the alarm timeout value increased to 40 in spamd as you 
suggested.


Yeah, that's not really going to help.  It's going to take a LONG time 
if you're swap thrashing.




Does this ring any bells?


Yup.  I'm almost certain that this is definitely caused by the spamd 
processes being swapped in and out.




Thanks for you continued help on this.


No problem.  Bill is in the mail!  :)


Daryl

RE: The Future of Email is SQL

2006-06-09 Thread John D. Hardin

On Fri, 9 Jun 2006, Rob McEwen wrote:

> >>MS Exchange... one big Database
> 
> Exactly...
> 
> And that is one reason why I wouldn't touch this SQL idea with a
> 10 foot pole.. the fact that Exchange works this way only proves
> my point... I hear all the time about Exchange servers crashing
> and the administrator having to rebuild the database while the
> mail server is down for the next 10 hours.
> 
> The bottom line is that using a SQL DB backend as mail storage is
> putting all your eggs in one basket.

Not to mention you get the same problem that everyone complains about
with the Windows Registry: everything is buried in this black box
storage format that you can only access with specific tools - you lose
the ability to access and process email messages with the rich suite
of simple text processing tools that are available, and the ability to
read your email with something as simple as a text editor.

Granted, there is less of the "impenetrable black box" situation with
a SQL database than there is with the Registry, but the same concepts
and limitations apply.

--
 John Hardin KA7OHZICQ#15735746http://www.impsec.org/~jhardin/
 [EMAIL PROTECTED]FALaholic #11174pgpk -a [EMAIL PROTECTED]
 key: 0xB8732E79 - 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  The first time I saw a bagpipe, I thought the player was torturing
  an octopus. I was amazed they could scream so loudly.
-- cat_herder_5263 on Y! SCOX
---
 9 days until SWMBO's Birthday

Re: Gmail spam

2006-06-09 Thread jdow


Off hand you could not convince me that this message ever got near
gmail servers.

{^_^}
- Original Message - 
From: "Jason Staudenmayer" <[EMAIL PROTECTED]>



Is anyone else getting spam from gmail? The ones I'm getting are very
lengthy but doesn't look like bayes poison.


Microsoft Mail Internet Headers Version 2.0
Received: from mail2.adventureaquarium.com ([10.0.0.205]) by
MAIL-I.adventureaquarium.com with Microsoft SMTPSVC(5.0.2195.6713);
Thu, 8 Jun 2006 08:05:21 -0400
Received: (qmail 31386 invoked from network); 8 Jun 2006 12:05:21 -
Received: from [EMAIL PROTECTED] by
mail2.adventureaquarium.com by uid 503 with qmail-scanner-1.20 
(clamdscan: 0.88.2/1467. spamassassin: 3.1.1.
Clear:RC:0(66.148.73.132):SA:0(2.2/7.5):. 
Processed in 0.48126 secs); 08 Jun 2006 12:05:21 -

X-Spam-Status: No, hits=2.2 required=7.5
X-Qmail-Scanner-Mail-From: [EMAIL PROTECTED] via
mail2.adventureaquarium.com
X-Qmail-Scanner: 1.20 (Clear:RC:0(66.148.73.132):SA:0(2.2/7.5):.
Processed in 0.48126 secs)
Received: from unknown (HELO 192.168.0.4) (66.148.73.132)
 by mail2.adventureaquarium.com with SMTP; 8 Jun 2006 12:05:21 -
Received: from crysholgh.com (9.13.1/9.13.1) id XAA37462; Thu, 08 Jun
2006 05:05:20 -0800
Message-Id: <[EMAIL PROTECTED]>
From: "Marcelino Crews" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Subject: this weeks stock pick KMAG - build a strong position now 
X-Mailer: Opera/6.05 (Windows 2000; U) [fi]

Date: Thu, 08 Jun 2006 05:05:20 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="Boundary-00=_9HReE4jIy7jpiF0"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
To: [EMAIL PROTECTED]
Subject: this weeks stock pick KMAG - build a strong position now 



Maybe gmail has an open relay? Or does this look like something else?

Jason

Re: Gmail spam

2006-06-09 Thread jdow


Repeat - it NEVER WENT NEAR gmail. That part is pure forgery.

{^_^}
- Original Message - 
From: "Jason Staudenmayer" <[EMAIL PROTECTED]>



I see ... I'll have to see why my qmail didn't drop it for those address
issues.

Thanks

-Original Message-
From: Jamie L. Penman-Smithson [mailto:[EMAIL PROTECTED] 


On 9 Jun 2006, at 13:56, Jason Staudenmayer wrote:

Is anyone else getting spam from gmail? The ones I'm getting are very
lengthy but doesn't look like bayes poison.


It's _not from_ GMail.



Received: from unknown (HELO 192.168.0.4) (66.148.73.132)
  by mail2.adventureaquarium.com with SMTP; 8 Jun 2006 12:05:21 -
Received: from crysholgh.com (9.13.1/9.13.1) id XAA37462; Thu, 08 Jun
2006 05:05:20 -0800
Message-Id: <[EMAIL PROTECTED]>
From: "Marcelino Crews" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Subject: this weeks stock pick KMAG - build a strong position now




Maybe gmail has an open relay? Or does this look like something else?


No, you should be looking at this header:


Received: from unknown (HELO 192.168.0.4) (66.148.73.132)
  by mail2.adventureaquarium.com with SMTP; 8 Jun 2006 12:05:21 -


This message was received from [66.148.73.132] with no rDNS and using  
a private non-routable IP in HELO.


The IP in question is owned by HopOne:

NetRange:   66.148.64.0 - 66.148.127.255
CIDR:   66.148.64.0/18
OrgName:HopOne Internet Corporation
OrgID:  HOPO
Address:1010 Wisconsin Avenue N.W.
City:   Washington
StateProv:  DC
PostalCode: 20007-3603
Country:US

It doesn't match the SPF record for gmail.com either:

_spf.google.com.300 IN  TXT "v=spf1  
ip4:216.239.56.0/23 ip4:64.233.160.0/19 ip4:66.249.80.0/20  
ip4:72.14.192.0/18 ?all"


The sender address is forged, as is common.

IOW it should have been rejected outright before it even got to SA,  
either because it has no rDNS, or because it used an invalid address  
literal (1.2.3.4 instead of [1.2.3.4]), or because it used a private  
non-routable IP in HELO.


-j

Re: The Future of Email is SQL

On Fri, Jun 09, 2006 at 02:50:03PM -0700, Marc Perkel wrote:
> Gary,
> 
> I'm trying to introduce the idea of a MySQL backend to Timo over at 
> Dovecot. He has done a little work in that direction already. But - I'm 
> throwing this idea out there right now just to get people thinking. I'm 
> hoping that in the next year as people think this through that some 
> serious development will occur. I think that as people say AH HA that 
> development will progress.

Before you start getting stuck with MySQL you should read
http://sql-info.de/mysql/gotchas.html. You'd be much better off with a
database that's actually standards compliant.

Probably the best bet would be to offer support for SQLite and
PostgreSQL. That allows small users to have the 0 maintenance of SQLite
while big users get the scaleability of PostgreSQL.
-- 
Jim C. Nasby, Database Architect[EMAIL PROTECTED] 
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

Re: The Future of Email is SQL

On Fri, Jun 09, 2006 at 06:16:15PM -0400, Rob McEwen wrote:
> 
> >>MS Exchange... one big Database
> 
> Exactly...
> 
> And that is one reason why I wouldn't touch this SQL idea with a 10 foot
> pole.. the fact that Exchange works this way only proves my point... I hear
> all the time about Exchange servers crashing and the administrator having to
> rebuild the database while the mail server is down for the next 10 hours.

Just because MS couldn't figure out how to do this correctly doesn't
mean it can't be done.
-- 
Jim C. Nasby, Database Architect[EMAIL PROTECTED] 
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

Re: The Future of Email is SQL

2006-06-09 Thread Steve Thomas

While this is quite an interesting topic, I have to ask why it's on the
spamassassin list. Message stores aren't spamassassin specific and this is
already a pretty high-volume list. Does this discussion really belong
here?

St-

RE: The Future of Email is SQL

2006-06-09 Thread Gary W. Smith

It's getting there, albeit slowly.  I think that if you rule out any up
and coming application but it's just not there yet we wouldn't have an
opensource community...  

We have a variety of reasons for using MySQL, most of them aren't good
ones though but it's something we've been able to work with for some
time.


> -Original Message-
> From: Jim C. Nasby [mailto:[EMAIL PROTECTED]
> Sent: Friday, June 09, 2006 9:05 PM
> To: Marc Perkel
> Cc: Gary W. Smith; users@spamassassin.apache.org
> Subject: Re: The Future of Email is SQL
> 
> On Fri, Jun 09, 2006 at 02:50:03PM -0700, Marc Perkel wrote:
> > Gary,
> >
> > I'm trying to introduce the idea of a MySQL backend to Timo over at
> > Dovecot. He has done a little work in that direction already. But -
I'm
> > throwing this idea out there right now just to get people thinking.
I'm
> > hoping that in the next year as people think this through that some
> > serious development will occur. I think that as people say AH HA
that
> > development will progress.
> 
> Before you start getting stuck with MySQL you should read
> http://sql-info.de/mysql/gotchas.html. You'd be much better off with a
> database that's actually standards compliant.
> 
> Probably the best bet would be to offer support for SQLite and
> PostgreSQL. That allows small users to have the 0 maintenance of
SQLite
> while big users get the scaleability of PostgreSQL.
> --
> Jim C. Nasby, Database Architect[EMAIL PROTECTED]
> Give your computer some brain candy! www.distributed.net Team #1828
> 
> Windows: "Where do you want to go today?"
> Linux: "Where do you want to go tomorrow?"
> FreeBSD: "Are you guys coming, or what?"

Re: The Future of Email is SQL