[Assp-test] Seg Fault

[Colin Waring]

For the past couple of weeks ASSP has been dying around the same time, though 
not consistently every day.

The logs haven't shown anything - they just stop. I changed ASSP to not run as 
a daemon and this weekend have caught this error on the command line.

2018-08-18 08:30:07 [Worker_1] Warning: got unexpected signal SEGV in 
Worker_1: package - Net::SMTP, file - sub Net::SMTP::DESTROY_SSLNS, line - 
10!
Segmentation fault

Perl is v5.22.1
Net::SMTP is up to date with CPAN:

/usr/local/share/perl/5.22.1/Net/SMTP.pm
Installed: 3.11
CPAN:  3.11  up to date

The most recent log for Worker_1 is almost 30 minutes prior and related to 
AFC being called. The lines immediately before related to MaxAUTHErrors:

2018-08-18 08:30:03 m1-77402-01879 [Worker_5] [MaxAUTHErrors] 181.214.206.111 
too many (26) AUTH errors from 181.214.206.0
2018-08-18 08:30:03 m1-77402-01879 [Worker_5] 181.214.206.111 Message-Score: 
added 60 (autValencePB) for AUTHErrors, total score for this message is now 60
2018-08-18 08:30:03 m1-77402-01879 [Worker_5] 181.214.206.111 info: start 
damping on closing connection (12)

The only thing possibly consistent with the timing is that block reports are 
run at 8am, though they have been for a long time. I don't know of anything 
else recurring even close to that time.

Any suggestions?
All the best,
Colin.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Log entries for resend

[Colin Waring]

I've noticed the log entries are inaccurate and it logs as "successful sent" 
even if it is not successful.

This causes log searches to be inaccurate as searching for "resend" doesn't 
bring up the error.

2018-07-26 09:30:13 [Worker_1] Error: can't open requested file .eml in 
any collection folder
2018-07-26 09:30:16 [Worker_1] Info: successful sent file 
/usr/local/assp/resendmail/.eml to 1.1.1.1:1 (smtpDestination)

Can the second line be updated so that it states the send failed?

Also less important, the correct grammar should be "successfully sent".

Thanks,
Colin.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Can't use an undefined value as a subroutine reference at sub main::ThreadMaintMain2 line 63

[Colin Waring]

Thanks, that’s firing up now and I’ll see what happens next time I catch the 
error.

All the best,
Colin.

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 16 May 2018 07:18
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Can't use an undefined value as a subroutine reference 
at sub main::ThreadMaintMain2 line 63

Colin,

I'm unable to reproduce this behavior.

I've uploaded a modified version (18136) to the test folder in SVN.
This version will tell us what happens.

Thomas





Von:    "Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:15.05.2018 10:10
Betreff:[Assp-test] Can't use an undefined value as a subroutine 
reference at sub main::ThreadMaintMain2 line 63



Hi,



I’ve caught this today:



2018-05-15 08:03:31 [Main_Thread] Saving config

2018-05-15 08:03:31 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged

2018-05-15 08:03:31 [Main_Thread] Adminupdate: file 
'/usr/local/assp/files/blockreportuser.txt' for config 'BlockReportFile' was 
changed

2018-05-15 08:03:32 [Main_Thread] Info: added schedule : BlockReport - for : 
*@domain.tld=>*=>1<mailto:*@domain.tld=%3e*=%3e1>=> - at : 0 0,4,8,12,16,20 * * 
* - next run is at : 2018-05-15 12:00:00

2018-05-15 08:03:32 [Worker_1] Info: notification message queued to sent to 
monitoraddr...@ourdomain.tld<mailto:monitoraddr...@ourdomain.tld>

2018-05-15 08:03:32 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.

2018-05-15 08:03:32 [Main_Thread] SyncCFG: start synchronization of 
BlockReportFile

2018-05-15 08:03:32 [Worker_1] Info: auto restart died worker Worker_1

2018-05-15 08:03:32 [Worker_1] Info: cleaned command 'syncConfigSend' from 
commandqueue

2018-05-15 08:04:11 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!

2018-05-15 08:04:56 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!



There then seems to be no traffic until 08:05:00 (approx. 90s)



It is highly unlikely that BlockReportFile was changed at this time. The line 
from BlockReportFile that is quoted used to work but I can see it is now 
missing the “# next run” so I’m suspecting the 4 hour schedule is the issue 
here. I know it used to be right because I questioned whether the number of 
days could be less than 1 when it was initially set up.



Even more odd is that I don’t get this error every four hours – the last time 
it happened was on the 10th so there must be more to it than the entry in the 
file:



2018-05-10 08:02:46 [Main_Thread] Saving config

2018-05-10 08:02:46 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged

2018-05-10 08:02:46 [Main_Thread] Adminupdate: file 
'/usr/local/assp/files/blockreportuser.txt' for config 'BlockReportFile' was 
changed

2018-05-10 08:02:47 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.

2018-05-10 08:02:47 [Worker_1] Info: notification message queued to sent to 
support.dolphinict.co...@email.uk.autotask.net<mailto:support.dolphinict.co...@email.uk.autotask.net>

2018-05-10 08:02:47 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.

2018-05-10 08:02:47 [Main_Thread] Info: added schedule : BlockReport - for : 
*@domain.tld<mailto:*@domain.tld> =>*=>1=> - at : 0 0,4,8,12,16,20 * * * - next 
run is at : 2018-05-10 12:00:00

2018-05-10 08:02:47 [Worker_1] Info: auto restart died worker Worker_1

2018-05-10 08:02:47 [Main_Thread] SyncCFG: start synchronization of 
BlockReportFile

2018-05-10 08:03:04 [Worker_10001] SyncCFG: request to synchronize 
BlockReportFile

2018-05-10 08:03:13 [Worker_10001] SyncCFG: successfully sent config for 
BlockReportFile to 10.0.5.219:25

2018-05-10 08:03:46 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!



Traffic didn’t stop that time.

I’m not sure it’s a significant problem, but it’s an error nonetheless.

All the best,

Colin.



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>
https://lists.sourceforge.net/lists/list

[Assp-test] Can't use an undefined value as a subroutine reference at sub main::ThreadMaintMain2 line 63

[Colin Waring]

Hi,

I've caught this today:

2018-05-15 08:03:31 [Main_Thread] Saving config
2018-05-15 08:03:31 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged
2018-05-15 08:03:31 [Main_Thread] Adminupdate: file 
'/usr/local/assp/files/blockreportuser.txt' for config 'BlockReportFile' was 
changed
2018-05-15 08:03:32 [Main_Thread] Info: added schedule : BlockReport - for : 
*@domain.tld=>*=>1=> - at : 0 0,4,8,12,16,20 * * * - next run is at : 
2018-05-15 12:00:00
2018-05-15 08:03:32 [Worker_1] Info: notification message queued to sent to 
monitoraddr...@ourdomain.tld
2018-05-15 08:03:32 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.
2018-05-15 08:03:32 [Main_Thread] SyncCFG: start synchronization of 
BlockReportFile
2018-05-15 08:03:32 [Worker_1] Info: auto restart died worker Worker_1
2018-05-15 08:03:32 [Worker_1] Info: cleaned command 'syncConfigSend' from 
commandqueue
2018-05-15 08:04:11 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!
2018-05-15 08:04:56 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!

There then seems to be no traffic until 08:05:00 (approx. 90s)

It is highly unlikely that BlockReportFile was changed at this time. The line 
from BlockReportFile that is quoted used to work but I can see it is now 
missing the "# next run" so I'm suspecting the 4 hour schedule is the issue 
here. I know it used to be right because I questioned whether the number of 
days could be less than 1 when it was initially set up.

Even more odd is that I don't get this error every four hours - the last time 
it happened was on the 10th so there must be more to it than the entry in the 
file:

2018-05-10 08:02:46 [Main_Thread] Saving config
2018-05-10 08:02:46 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged
2018-05-10 08:02:46 [Main_Thread] Adminupdate: file 
'/usr/local/assp/files/blockreportuser.txt' for config 'BlockReportFile' was 
changed
2018-05-10 08:02:47 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.
2018-05-10 08:02:47 [Worker_1] Info: notification message queued to sent to 
support.dolphinict.co...@email.uk.autotask.net
2018-05-10 08:02:47 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.
2018-05-10 08:02:47 [Main_Thread] Info: added schedule : BlockReport - for : 
*@domain.tld =>*=>1=> - at : 0 0,4,8,12,16,20 * * * - next run is at : 
2018-05-10 12:00:00
2018-05-10 08:02:47 [Worker_1] Info: auto restart died worker Worker_1
2018-05-10 08:02:47 [Main_Thread] SyncCFG: start synchronization of 
BlockReportFile
2018-05-10 08:03:04 [Worker_10001] SyncCFG: request to synchronize 
BlockReportFile
2018-05-10 08:03:13 [Worker_10001] SyncCFG: successfully sent config for 
BlockReportFile to 10.0.5.219:25
2018-05-10 08:03:46 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!

Traffic didn't stop that time.
I'm not sure it's a significant problem, but it's an error nonetheless.
All the best,
Colin.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

So your domain is thockar.com therefore the forged domain would be 
thockar.com-1.me

My example domain was a .co.uk therefore the forged domain was 
example.co.uk-1.me

Whoever registered uk-1.me also registered com-1.me because the DNS records 
include CNAMEs that point to the uk-1.me

The only way I can see to catch this would be to have the sender/from/reply-to 
checked to see if the domain contains any line from the local domains file. If 
the entry appears anywhere other than at the end of the address then score. It 
would have to work only on the part after the @ because many mailing lists 
include sender addresses in the left hand side as a way of message tracking.

For my purposes, I cannot see any reason why any of my domains would appear in 
part in anyone else’s domains – however I can see cases where not everyone’s 
domains are unique enough so there would have to be an over-ride where specific 
domains could be excluded should that be necessary.

Hopefully I’ve managed to explain what’s in my head well enough?
All the best,
Colin.


From: Thomas Eckardt <thomas.ecka...@thockar.com>
Sent: 21 April 2018 10:20
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

Or would it be ''thockar.uk-1.me' ?

In either case - this is hard to catch. The bombHeaderRe may help, if there are 
only some local domains hosted.

Thomas





Von:"Thomas Eckardt" 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:21.04.2018 10:41
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains




Only to be clear - for my domain the domainname would be 'thockar.com.uk-1.me' 
- right?



Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:21.04.2018 09:51
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains



In this case, the actual domain on the reply-to header was uk-1.me – this 
exists and is registered. The domain has wildcard DNS so *.uk-1.me will return 
valid DNS records – both A and MX. I suspect that the domain has been 
registered for the express intention of sending these kinds of phishing emails 
so I’ve added *@*.uk-1.me<mailto:*@*.uk-1.me> to blackListedDomains but it 
wouldn’t take much for them to change domains.



As a result, the reply-to address of localdomain.co.uk-1.me appears valid to 
all checks. The only thing that could tell ASSP that this is a phishing address 
is that the hostname contains an entry from localdomains with a bit on the end.



It just so happens that this particular message also had multiple from headers 
– something that you have updated ASSP to be able to detect now. We will now 
catch any similar emails on that basis however it is still possible that such a 
phishing email would get past if it did not have multiple from headers.



As these kinds of emails tend to be targeted and manually crafted for high 
value amounts I would guess it won’t take long for a miscreant to figure that 
out with a few tests.



All the best,

Colin.



From: Thomas Eckardt 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
Sent: 21 April 2018 08:18
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



>None of the addresses are actually @domain.tld

I'm right ? The used domains never ends with a valid TLD - so the domains never 
exists? Or at least - they ends with a valid TLD, but domains not exists?

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:20.04.2018 21:49
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Yes there is so this particular message gets caught which is great.

There is no guarantee that all emails with the -1.me also have multiple from 
headers, also the -1.me can change but it is always -something.tld on the end.

All the best,

Colin.



From: Thomas Eckardt 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
Sent: 20 April 2018 17:54
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

In this case, the actual domain on the reply-to header was uk-1.me – this 
exists and is registered. The domain has wildcard DNS so *.uk-1.me will return 
valid DNS records – both A and MX. I suspect that the domain has been 
registered for the express intention of sending these kinds of phishing emails 
so I’ve added *@*.uk-1.me<mailto:*@*.uk-1.me> to blackListedDomains but it 
wouldn’t take much for them to change domains.

As a result, the reply-to address of localdomain.co.uk-1.me appears valid to 
all checks. The only thing that could tell ASSP that this is a phishing address 
is that the hostname contains an entry from localdomains with a bit on the end.

It just so happens that this particular message also had multiple from headers 
– something that you have updated ASSP to be able to detect now. We will now 
catch any similar emails on that basis however it is still possible that such a 
phishing email would get past if it did not have multiple from headers.

As these kinds of emails tend to be targeted and manually crafted for high 
value amounts I would guess it won’t take long for a miscreant to figure that 
out with a few tests.

All the best,
Colin.

From: Thomas Eckardt <thomas.ecka...@thockar.com>
Sent: 21 April 2018 08:18
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

>None of the addresses are actually @domain.tld

I'm right ? The used domains never ends with a valid TLD - so the domains never 
exists? Or at least - they ends with a valid TLD, but domains not exists?

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:20.04.2018 21:49
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains



Yes there is so this particular message gets caught which is great.

There is no guarantee that all emails with the -1.me also have multiple from 
headers, also the -1.me can change but it is always -something.tld on the end.

All the best,

Colin.



From: Thomas Eckardt 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
Sent: 20 April 2018 17:54
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



But there should be a scoring because of multiple Fom: and/or Sender: headers-

Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:20.04.2018 16:42
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Hi Thomas,



I’ve run the message through the analyser and although a great feature to have 
it is not going to catch these emails.



None of the addresses are actually @domain.tld



The Reply-to: is @domain.tld-1.me so the extra -1.me bypasses the spoofing 
check.



The DoNoFrom: option is catching the multiple from headers which is great.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 20 April 2018 15:24
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



Colin,

solved build 18107 the problem for you?


changed:
...
'DoNoSpoofing4From','Do NoSpoofing for from:'
'Do the NoSpoofing check also for header 'from:', 'sender:', 'reply-to:' and 
'errors-to:' addresses.

Thomas





Von:"cw" <colin.war...@gmail.com<mailto:colin.war...@gmail.com>>
An:"ASSP development mailing list" 
<Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>>
Datum:14.04.2018 09:47
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Hi Thomas,

Looks like a good feature. I'll have to double check the headers for this 
message. I think the domains in all three from headers actually exist but have 
no relation to the recipient.

As the smtp address & from headers are a legitimate but compromised account the 
only header that would fail a legitimate domain check would be the reply to 
header.

These are carefully crafted phishing emails that are targeted, I've seen them 
sent to many accounts departments pretending to be from company directors 
requesting bank payments of up to £10,000. Of course the accounts department 
goes straight to said direc

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

Yes there is so this particular message gets caught which is great.
There is no guarantee that all emails with the -1.me also have multiple from 
headers, also the -1.me can change but it is always -something.tld on the end.
All the best,
Colin.

From: Thomas Eckardt <thomas.ecka...@thockar.com>
Sent: 20 April 2018 17:54
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

But there should be a scoring because of multiple Fom: and/or Sender: headers-

Thomas





Von:    "Colin Waring" <co...@dolphinict.co.uk>
An:"ASSP development mailing list" <assp-test@lists.sourceforge.net>
Datum:20.04.2018 16:42
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains



Hi Thomas,



I’ve run the message through the analyser and although a great feature to have 
it is not going to catch these emails.



None of the addresses are actually @domain.tld



The Reply-to: is @domain.tld-1.me so the extra -1.me bypasses the spoofing 
check.



The DoNoFrom: option is catching the multiple from headers which is great.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 20 April 2018 15:24
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



Colin,

solved build 18107 the problem for you?


changed:
...
'DoNoSpoofing4From','Do NoSpoofing for from:'
 'Do the NoSpoofing check also for header 'from:', 'sender:', 'reply-to:' and 
'errors-to:' addresses.

Thomas





Von:"cw" <colin.war...@gmail.com<mailto:colin.war...@gmail.com>>
An:"ASSP development mailing list" 
<Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>>
Datum:14.04.2018 09:47
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Hi Thomas,

Looks like a good feature. I'll have to double check the headers for this 
message. I think the domains in all three from headers actually exist but have 
no relation to the recipient.

As the smtp address & from headers are a legitimate but compromised account the 
only header that would fail a legitimate domain check would be the reply to 
header.

These are carefully crafted phishing emails that are targeted, I've seen them 
sent to many accounts departments pretending to be from company directors 
requesting bank payments of up to £10,000. Of course the accounts department 
goes straight to said director who comes to us wanting to know why we aren't 
blocking them.

All the best,
Colin

On Sat, 14 Apr 2018, 08:26 Thomas Eckardt, 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>> wrote:
> I thought this would not be caught by nospoofing because that would only 
> match if the RHS ended in the entry from localdomains.

OK.

And what if the 'DoNoFrom' feature would work like this:

Check for Existing and Valid From: and Sender: Header Tag and Address (DoNoFrom)

If enabled, the MIME header is checked for valid From: and Sender: header tags.
This header check fails and faults are counted, if both headers (From: and 
Sender:) are missing - or if any of these headers contains not a valid email 
address - or if multiple of the same headers are found.
The scoring value nofromValencePB is added for each detected fault.


In your example:

Reply-to: Sender Name 
<n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>>

To: recipi...@recipientdomain.tld<mailto:recipi...@recipientdomain.tld>

From: Sender Name <f...@domain.tld<mailto:f...@domain.tld>>

From: Sender Name <f...@domain2.tld<mailto:f...@domain2.tld>>

From: Sender Name 
<actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld<mailto:actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld>>

'nofromValencePB' would be added two times - one time for each additionally 
From: header.




Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:13.04.2018 20:55
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Thank you for the reply Thomas,



Being able to include sender:, reply to: and errors-to: would be handy in my 
opinion



However, in this case the local domain was not in any of the from: fields 
whatsoever. By using 
n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>, this hits 
a stupid bug in Outlook where in some places it will only display 
n...@recipientdomain.

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

Hi Thomas,

I’ve run the message through the analyser and although a great feature to have 
it is not going to catch these emails.

None of the addresses are actually @domain.tld

The Reply-to: is @domain.tld-1.me so the extra -1.me bypasses the spoofing 
check.

The DoNoFrom: option is catching the multiple from headers which is great.

All the best,
Colin.

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 20 April 2018 15:24
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

Colin,

solved build 18107 the problem for you?


changed:
...
'DoNoSpoofing4From','Do NoSpoofing for from:'
  'Do the NoSpoofing check also for header 'from:', 'sender:', 'reply-to:' and 
'errors-to:' addresses.

Thomas





Von:"cw" <colin.war...@gmail.com<mailto:colin.war...@gmail.com>>
An:"ASSP development mailing list" 
<Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>>
Datum:14.04.2018 09:47
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains




Hi Thomas,

Looks like a good feature. I'll have to double check the headers for this 
message. I think the domains in all three from headers actually exist but have 
no relation to the recipient.

As the smtp address & from headers are a legitimate but compromised account the 
only header that would fail a legitimate domain check would be the reply to 
header.

These are carefully crafted phishing emails that are targeted, I've seen them 
sent to many accounts departments pretending to be from company directors 
requesting bank payments of up to £10,000. Of course the accounts department 
goes straight to said director who comes to us wanting to know why we aren't 
blocking them.

All the best,
Colin

On Sat, 14 Apr 2018, 08:26 Thomas Eckardt, 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>> wrote:
> I thought this would not be caught by nospoofing because that would only 
> match if the RHS ended in the entry from localdomains.

OK.

And what if the 'DoNoFrom' feature would work like this:

Check for Existing and Valid From: and Sender: Header Tag and Address (DoNoFrom)

If enabled, the MIME header is checked for valid From: and Sender: header tags.
This header check fails and faults are counted, if both headers (From: and 
Sender:) are missing - or if any of these headers contains not a valid email 
address - or if multiple of the same headers are found.
The scoring value nofromValencePB is added for each detected fault.


In your example:

Reply-to: Sender Name 
<n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>>

To: recipi...@recipientdomain.tld<mailto:recipi...@recipientdomain.tld>

From: Sender Name <f...@domain.tld<mailto:f...@domain.tld>>

From: Sender Name <f...@domain2.tld<mailto:f...@domain2.tld>>

From: Sender Name 
<actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld<mailto:actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld>>

'nofromValencePB' would be added two times - one time for each additionally 
From: header.




Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:13.04.2018 20:55
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains



Thank you for the reply Thomas,



Being able to include sender:, reply to: and errors-to: would be handy in my 
opinion



However, in this case the local domain was not in any of the from: fields 
whatsoever. By using 
n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>, this hits 
a stupid bug in Outlook where in some places it will only display 
n...@recipientdomain.tld<mailto:n...@recipientdomain.tld>. The 
-1.me<http://1.me/> is completely fictional and varies from message to message. 
I thought this would not be caught by nospoofing because that would only match 
if the RHS ended in the entry from localdomains.



All the best,

Colin.



From: Thomas Eckardt 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
Sent: 13 April 2018 16:55
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



Colin,

'DoNoSpoofing4From' should do it - but it is'nt. Only the first 'From:' address 
is currently checked and this will not prevent this attack.
But it is possible to include 'sender: , reply-to: and errors-to:' in to this 
check - which would catch this mails

Re: [Assp-test] RebuildSDB not running

[Colin Waring]

Hi John,

You need to capture the output of the perl module installer and find out why 
all those modules are failing to install. Fix that. Alternatively, try 
installing each module manually and see the errors.

I’m running Ubuntu 16 and the installer would install most of those if the 
system was set up right.

All the best,
Colin.

From: John Wolf 
Sent: 13 April 2018 18:51
To: ASSP development mailing list 
Subject: [Assp-test] RebuildSDB not running

Hello All,
A month or so ago I created a new virtual machine in Virtual box.  It is a 
virtual Ubuntu 16.04 server currently running ASSP version 2.6.1  *Fortress*  
build 17355 .  I don't know if I missed something during the install, it seems 
to be working ok except the rebuildsdb process aborts out.  The log shows the 
following:
Server Name:

sfilter



ASSP host UUID:

d96937e9-2ed4-11e8-8f8d-e4fd4c4d0c10



Server OS:

linux



Server IP:

127.0.1.1



used DNS Servers:

192.168.xxx.3 192.168.xxx.254

Local DNS Servers in use



defined DNS Servers:

192.168..3 192.168.xxx.254



DNS Servers query time:

min: 0.000 , avg: 0.031 , max: 0.226



Perl Version:

5.022001

Perl.org

assp-process-memory:

current: 1407 MB

min: 1075 MB

max: 1407 MB

Spamdb version:

used:

2_14315_UAX#29_UAX#15_WordStem2.02

required:

2_14315_UAX#15

HMMdb version:

used:

n/a

required:

2_14315_UAX#15

code integrity signature:

expected:

D052878A93FA57BC3AAF9774BF1407E1845BD98E

current:

D052878A93FA57BC3AAF9774BF1407E1845BD98E

ASSP Version:

2.6.1(17355)

show current local change log

show last available change 
log


release

beta





 Apr-13-18 11:58:25 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:25 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:25 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:25 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:28 m1-35100-03802 [Worker_2] 66.220.155.145 
>
 to: adw...@wselectronics.com info: 
PB-IP-Score for '66.220.155.0' is 0, added 10 in this session

 Apr-13-18 11:58:30 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:30 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:30 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:30 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:35 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:35 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:35 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:35 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:40 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:40 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:40 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:40 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:45 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:45 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:45 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:45 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:50 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:50 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:50 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:50 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:55 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:55 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub 

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

Thank you for the reply Thomas,

Being able to include sender:, reply to: and errors-to: would be handy in my 
opinion

However, in this case the local domain was not in any of the from: fields 
whatsoever. By using 
n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>, this hits 
a stupid bug in Outlook where in some places it will only display 
n...@recipientdomain.tld<mailto:n...@recipientdomain.tld>. The -1.me is 
completely fictional and varies from message to message. I thought this would 
not be caught by nospoofing because that would only match if the RHS ended in 
the entry from localdomains.

All the best,
Colin.

From: Thomas Eckardt <thomas.ecka...@thockar.com>
Sent: 13 April 2018 16:55
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

Colin,

'DoNoSpoofing4From' should do it - but it is'nt. Only the first 'From:' address 
is currently checked and this will not prevent this attack.
But it is possible to include 'sender: , reply-to: and errors-to:' in to this 
check - which would catch this mails.

What do you think?

Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:13.04.2018 17:17
Betreff:[Assp-test] Multiple From headers/regex based on localdomains



Hi,



I’ve a couple of fun ones at the moment. Basically I’m getting reports of 
phishing emails that get past everything.



The headers are like this:



Reply-to: Sender Name 
<n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>>

To: recipi...@recipientdomain.tld<mailto:recipi...@recipientdomain.tld>

From: Sender Name <f...@domain.tld<mailto:f...@domain.tld>>

From: Sender Name <f...@domain2.tld<mailto:f...@domain2.tld>>

From: Sender Name 
<actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld<mailto:actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld>>



These bypass no spoofing as none of the from/SMTP header domains are actually 
the recipient domain. Annoyingly, Outlook chooses the Reply-to address to 
display so it appears almost legitimate.



I’m aware that the RFCs allow multiple from headers, though I can’t see of any 
legitimate reason for this so I was considering blocking or increasing spam 
score based on this – is this possible with ASSP at the moment or not?



The second thing I was looking at doing was coming up with a regex. 
Essentially, all recipient domains are in localdomains.txt so I’d want a regex 
that would take all lines from localdomains. If the reply to or smtp from 
address is a line from localdomains with anything else after it, then bin it. I 
accept that there may in some extremely obscure cases be a clash with a 
legitimate domain but do not believe that to be likely. I’ll have a look next 
week as to if I can figure out a way to do it but if there’s something obvious 
that you could let me know that’d be great.



All the best,

Colin.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/assp-test





DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known 
virus in this email!
***
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

So,

As suspected the rebuild debug shows nothing useful at this stage.

I can however now tell where the content of hmmdb is coming from – it is being 
populated whenever someone reports a message through the email interface.

The only files I have in tmpDB currently are:

-rw-r--r-- 1 root root 118557988 Jan  8 06:46 rbtmp.hamHMM.chains
-rw-r--r-- 1 root root  94224002 Jan  8 06:46 rbtmp.hamHMM.totals
-rw-r--r-- 1 root root 240631755 Jan  8 06:47 rbtmp.spamHMM.chains
-rw-r--r-- 1 root root 185125930 Jan  8 06:47 rbtmp.spamHMM.totals


So I’m missing rbtmp.hamHMM and rbtmp.spamHMM

I had a look at the code and saw that the populate part runs the database 
import routine against the hash HMMresObj yet the only place the hash is 
populated is:

$HMMresObj=tie %HMMres,'BerkeleyDB::Hash',
 (-Filename => "$DBDir/rb_HMMres.bdb" ,
  -Flags => DB_CREATE,
  -Env => $BDBEnv);

So, how does the database get populated if BDB is off?

That’s about as far as I can get at the moment I think..

Incidentally I have noticed that spamdb.helo.rb.tmp gets created in the assp 
working directory not tmpDB – I’m not sure whether it is supposed to be there?

All the best,
Colin.


From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 07 January 2018 22:43
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Rebuild has completed:

mysql> select * from hmmdb;
+--++-+
| pkey | pvalue | pfrozen |
+--++-+
| ***COUNT***  | 3  |   0 |
| ***DB-VERSION*** | 2_14315_UAX#29_UAX#15_WordStem2.02 |   0 |
| ***bayesnorm***  | 0.54300466416  |   0 |
+--++-+
3 rows in set (0.00 sec)

So nothing in mysql. ASSP status is all green and I can see the above data by 
using the edit list button next to hmmdb.

Could DBCacheMaxAge have anything to do with this? It was set to 10.

I’m re-running rebuild with the debug file created and will have to check in 
the morning.



From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 07 January 2018 21:08
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Hi Thomas,

I’ve checked and RebuildTestMode is not set.

mysql> select count(*) from hmmdb;
+--+
| count(*) |
+--+
|  5194934 |
+--+
1 row in set (3.35 sec)

The count hasn’t changed overnight so it is definitely not updating.

So I’ve dropped hmmdb, spamdb and spamdbhelo. Run a full update on all the 
servers including perl modules and then restarted everything. Tables recreated 
and now a rebuild is running to hopefully set them up afresh.

Fingers crossed that solves it and hopefully no other tables are affected.

All the best,
Colin.
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 19:06
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Colin, did you set RebuildTestMode  For me, it looks like.

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Remove these tables - they were possibly created many many years ago. I can't 
remember.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 19:29
Betreff:Re: [Assp-test] Meltdown/Spectre



Hi Thomas,



Maybe I’m misunderstanding what populating is? Is populating when the temporary 
db generated by the rebuild are loaded into the mysql server?



I was therefore looking at the mysql server to confirm if any new data was 
being put in it.



Is there any debugging I can turn up to get more information on what is 
happening at that point? I’m not sure if rebuilddebug.txt would give more 
information, I imagine it’d certainly slow down other parts of the rebuild.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 17:34
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



>2018-01-06 22:00:00 Maxbytes: 20,000
ok nearly two hours - that's long - takes on my system ~ 30 min
>2018-01-06 23:51:13 start popul

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

Rebuild has completed:

mysql> select * from hmmdb;
+--++-+
| pkey | pvalue | pfrozen |
+--++-+
| ***COUNT***  | 3  |   0 |
| ***DB-VERSION*** | 2_14315_UAX#29_UAX#15_WordStem2.02 |   0 |
| ***bayesnorm***  | 0.54300466416  |   0 |
+--++-+
3 rows in set (0.00 sec)

So nothing in mysql. ASSP status is all green and I can see the above data by 
using the edit list button next to hmmdb.

Could DBCacheMaxAge have anything to do with this? It was set to 10.

I’m re-running rebuild with the debug file created and will have to check in 
the morning.



From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 07 January 2018 21:08
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Hi Thomas,

I’ve checked and RebuildTestMode is not set.

mysql> select count(*) from hmmdb;
+--+
| count(*) |
+--+
|  5194934 |
+--+
1 row in set (3.35 sec)

The count hasn’t changed overnight so it is definitely not updating.

So I’ve dropped hmmdb, spamdb and spamdbhelo. Run a full update on all the 
servers including perl modules and then restarted everything. Tables recreated 
and now a rebuild is running to hopefully set them up afresh.

Fingers crossed that solves it and hopefully no other tables are affected.

All the best,
Colin.
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 19:06
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Colin, did you set RebuildTestMode  For me, it looks like.

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Remove these tables - they were possibly created many many years ago. I can't 
remember.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 19:29
Betreff:Re: [Assp-test] Meltdown/Spectre



Hi Thomas,



Maybe I’m misunderstanding what populating is? Is populating when the temporary 
db generated by the rebuild are loaded into the mysql server?



I was therefore looking at the mysql server to confirm if any new data was 
being put in it.



Is there any debugging I can turn up to get more information on what is 
happening at that point? I’m not sure if rebuilddebug.txt would give more 
information, I imagine it’d certainly slow down other parts of the rebuild.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 17:34
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



>2018-01-06 22:00:00 Maxbytes: 20,000
ok nearly two hours - that's long - takes on my system ~ 30 min
>2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
>check is now disabled!

>2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
>Bayesian check is now enabled!
there is something wrong - 5 seconds duration with a hardcoded delay of 5 
seconds for 2.5 million records

>2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 
>records!

>2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
>records!
same here, 5.4 million records in less than a second - this is impossible

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Where do you get these MySQL tables/records from ? There is no option (and also 
NO CODE) in assp to tie the temporary HMM tables to mysql. And even if this 
would be possible - mysql is too slow to build the HMM. There are only two 
options in assp to hold the temp HMM tables, BerkeleyDB and memory.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 17:51
Betreff:Re: [Assp-test] Meltdown/Spectre





So a report in from last nights’ rebuild

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

Hi Thomas,

I’ve checked and RebuildTestMode is not set.

mysql> select count(*) from hmmdb;
+--+
| count(*) |
+--+
|  5194934 |
+--+
1 row in set (3.35 sec)

The count hasn’t changed overnight so it is definitely not updating.

So I’ve dropped hmmdb, spamdb and spamdbhelo. Run a full update on all the 
servers including perl modules and then restarted everything. Tables recreated 
and now a rebuild is running to hopefully set them up afresh.

Fingers crossed that solves it and hopefully no other tables are affected.

All the best,
Colin.
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 19:06
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Colin, did you set RebuildTestMode  For me, it looks like.

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Remove these tables - they were possibly created many many years ago. I can't 
remember.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 19:29
Betreff:Re: [Assp-test] Meltdown/Spectre



Hi Thomas,



Maybe I’m misunderstanding what populating is? Is populating when the temporary 
db generated by the rebuild are loaded into the mysql server?



I was therefore looking at the mysql server to confirm if any new data was 
being put in it.



Is there any debugging I can turn up to get more information on what is 
happening at that point? I’m not sure if rebuilddebug.txt would give more 
information, I imagine it’d certainly slow down other parts of the rebuild.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 17:34
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



>2018-01-06 22:00:00 Maxbytes: 20,000
ok nearly two hours - that's long - takes on my system ~ 30 min
>2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
>check is now disabled!

>2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
>Bayesian check is now enabled!
there is something wrong - 5 seconds duration with a hardcoded delay of 5 
seconds for 2.5 million records

>2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 
>records!

>2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
>records!
same here, 5.4 million records in less than a second - this is impossible

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Where do you get these MySQL tables/records from ? There is no option (and also 
NO CODE) in assp to tie the temporary HMM tables to mysql. And even if this 
would be possible - mysql is too slow to build the HMM. There are only two 
options in assp to hold the temp HMM tables, BerkeleyDB and memory.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 17:51
Betreff:Re: [Assp-test] Meltdown/Spectre





So a report in from last nights’ rebuild.

Logs are:



2018-01-06 22:00:00 Maxbytes: 20,000
2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
check is now disabled!

2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
Bayesian check is now enabled!
2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 records!

2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
records!
2018-01-06 23:52:22 Total processing time: 6,742 second(s)
2018-01-06 23:52:22 Total processing data: 975.63 Mbyte



So that’s about 20 minutes quicker with nearly double the data processed. 
Marginally more Spamdb records and a reduction of HMM records by 2 million.



Still about half the speed of yours though.

All the best,

Colin.



From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 06 January 2018 20:48
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



I’ll try upping Maxbytes  to 2

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

Hi Thomas,

Maybe I’m misunderstanding what populating is? Is populating when the temporary 
db generated by the rebuild are loaded into the mysql server?

I was therefore looking at the mysql server to confirm if any new data was 
being put in it.

Is there any debugging I can turn up to get more information on what is 
happening at that point? I’m not sure if rebuilddebug.txt would give more 
information, I imagine it’d certainly slow down other parts of the rebuild.

All the best,
Colin.

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 17:34
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

>2018-01-06 22:00:00 Maxbytes: 20,000
ok nearly two hours - that's long - takes on my system ~ 30 min
>2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
>check is now disabled!

>2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
>Bayesian check is now enabled!
there is something wrong - 5 seconds duration with a hardcoded delay of 5 
seconds for 2.5 million records

>2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 
>records!

>2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
>records!
same here, 5.4 million records in less than a second - this is impossible

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Where do you get these MySQL tables/records from ? There is no option (and also 
NO CODE) in assp to tie the temporary HMM tables to mysql. And even if this 
would be possible - mysql is too slow to build the HMM. There are only two 
options in assp to hold the temp HMM tables, BerkeleyDB and memory.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 17:51
Betreff:Re: [Assp-test] Meltdown/Spectre



So a report in from last nights’ rebuild.

Logs are:



2018-01-06 22:00:00 Maxbytes: 20,000
2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
check is now disabled!

2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
Bayesian check is now enabled!
2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 records!

2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
records!
2018-01-06 23:52:22 Total processing time: 6,742 second(s)
2018-01-06 23:52:22 Total processing data: 975.63 Mbyte



So that’s about 20 minutes quicker with nearly double the data processed. 
Marginally more Spamdb records and a reduction of HMM records by 2 million.



Still about half the speed of yours though.

All the best,

Colin.



From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 06 January 2018 20:48
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



I’ll try upping Maxbytes  to 2 and see what happens. I’ve also turned off 
usedb4rebuild to see what happens in relation to your other message.



As far as hmmdb goes, I checked everything over and can’t see anything wrong 
although the numbers don’t add up to the ones in the log. The db entries don’t 
have dates against them so I’m not sure how I would check to see if they are 
recent.



-rw-r--r-- 1 root root 0 Jan  5 22:00 BDB-error.txt

-rw-r--r-- 1 root root434175 Jan  5 22:00 __db.001

-rw-r--r-- 1 root root   3325951 Jan  5 22:00 __db.002

-rw-r--r-- 1 root root  65544191 Jan  5 22:13 __db.003

-rw-r--r-- 1 root root663552 Jan  6 00:12 rb_Helo.bdb

-rw-r--r-- 1 root root 334389248 Jan  6 00:08 rb_spam.bdb

-rw-r--r-- 1 root root 332099584 Jan  6 00:13 rbtmp.hamHMM.bdb

-rw-r--r-- 1 root root 168296448 Jan  6 00:13 rbtmp.hamHMM.totals.bdb

-rw-r--r-- 1 root root 339763200 Jan  6 00:13 rbtmp.spamHMM.bdb

-rw-r--r-- 1 root root 335945728 Jan  6 00:13 rbtmp.spamHMM.totals.bdb

-rw-r--r-- 1 root root 12288 Jan  5 23:21 trashlist.bdb



mysql> select count(*) from hmmdb;

|  5194934 |



mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |





From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 06 January 2018 06:54
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



> 

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

So a report in from last nights’ rebuild.

Logs are:



2018-01-06 22:00:00 Maxbytes: 20,000
2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
check is now disabled!

2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
Bayesian check is now enabled!
2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 records!

2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
records!
2018-01-06 23:52:22 Total processing time: 6,742 second(s)
2018-01-06 23:52:22 Total processing data: 975.63 Mbyte


So that’s about 20 minutes quicker with nearly double the data processed. 
Marginally more Spamdb records and a reduction of HMM records by 2 million.

Still about half the speed of yours though.
All the best,
Colin.

From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 06 January 2018 20:48
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

I’ll try upping Maxbytes  to 2 and see what happens. I’ve also turned off 
usedb4rebuild to see what happens in relation to your other message.

As far as hmmdb goes, I checked everything over and can’t see anything wrong 
although the numbers don’t add up to the ones in the log. The db entries don’t 
have dates against them so I’m not sure how I would check to see if they are 
recent.

-rw-r--r-- 1 root root 0 Jan  5 22:00 BDB-error.txt
-rw-r--r-- 1 root root434175 Jan  5 22:00 __db.001
-rw-r--r-- 1 root root   3325951 Jan  5 22:00 __db.002
-rw-r--r-- 1 root root  65544191 Jan  5 22:13 __db.003
-rw-r--r-- 1 root root663552 Jan  6 00:12 rb_Helo.bdb
-rw-r--r-- 1 root root 334389248 Jan  6 00:08 rb_spam.bdb
-rw-r--r-- 1 root root 332099584 Jan  6 00:13 rbtmp.hamHMM.bdb
-rw-r--r-- 1 root root 168296448 Jan  6 00:13 rbtmp.hamHMM.totals.bdb
-rw-r--r-- 1 root root 339763200 Jan  6 00:13 rbtmp.spamHMM.bdb
-rw-r--r-- 1 root root 335945728 Jan  6 00:13 rbtmp.spamHMM.totals.bdb
-rw-r--r-- 1 root root 12288 Jan  5 23:21 trashlist.bdb

mysql> select count(*) from hmmdb;
|  5194934 |

mysql> mysql> select count(*) from hmmham;
|  1248444 |

mysql> select count(*) from hmmhamtot;
|  1123064 |

mysql> select count(*) from hmmspam;
|  1654660 |

mysql> select count(*) from hmmspamtot;
|  1495532 |


From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 06 January 2018 06:54
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

> I’m wondering why I have so many more records when Maxbytes is less and the 
> total data is less.

This is caused by HTML mails - mostly SPAM mails.

You may have a look in to some spam mails with a size of 20.000 and more bytes. 
You'll find some, which are starting with alot of HTML header stuff (CSS and 
script and so on). Most times this content is longer than 6000 byte (your 
MaxByte setting).
I saw mails with a size of 25.000 bytes and 10 words of human readable content.
ASSP tries to get the human readable content of HTML mails for analyzing, but 
if this is not possible, it uses the available data.
The CSS and header content is very different in every mail. Even assp 
normalizes this content anyway, this leads in to much more different HMMdb and 
spamDB records - most of them are useless for spam detection.

Have a look in to the GUI for - Use this HTML Parser (HTMLParser).
I use HTML::Strip.

My current setting for MaxBytes (20.000) is only a long time running try out. I 
want to see, how the detection works from 20.000 to 50.000 bytes setting in 
10.000 bytes steps. Each setting is used for ~1 month. MaxBytes 50.000 has 
passed the test and was perfect - like expected - because 100% of spam mails 
(without an attachment) are perfectly analyzed and detected. How ever, this 
setting leads in to a ~25% performance penalty for the rebuild task (in 
relation to 20.000) using my corpus.

>CPU Model: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz

An nice CPU - but with ASSP's single threaded rebuild task it is slower than my 
older Intel(R) Xeon(R) CPU X5680 @ 3.33GHz. 
http://cpuboss.com/cpus/Intel-Xeon-X5680-vs-Intel-Xeon-E5-2640-v2

Collin, don't care about the overall rebuild speed. It runns at night and it 
does'nt hurt, if it takes an hour more or less. Two steps are time critical: 
populating spamDB and populating HMMdb. As you said "The db part looks to be 
fine". But wait 
It looks like, there is something wrong with the temporary rebuild databases 
used for HMM. This can be also the cause for a very very slow rebuild. >>> The 
rebuild was actually quicker a while back, maybe 40m

>2018-01-05 00:07:42 Start populating Hidden Markov Model. HMM-check is 
>disabled for this time!

>2018-01-05 00:07:43 Total processing time: 7,663 second(s)

This is ONE second time difference - totaly impossible - even if HMMdb is hold 
in RAM 

Is it right, that you use 

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

I’ll try upping Maxbytes  to 2 and see what happens. I’ve also turned off 
usedb4rebuild to see what happens in relation to your other message.

As far as hmmdb goes, I checked everything over and can’t see anything wrong 
although the numbers don’t add up to the ones in the log. The db entries don’t 
have dates against them so I’m not sure how I would check to see if they are 
recent.

-rw-r--r-- 1 root root 0 Jan  5 22:00 BDB-error.txt
-rw-r--r-- 1 root root434175 Jan  5 22:00 __db.001
-rw-r--r-- 1 root root   3325951 Jan  5 22:00 __db.002
-rw-r--r-- 1 root root  65544191 Jan  5 22:13 __db.003
-rw-r--r-- 1 root root663552 Jan  6 00:12 rb_Helo.bdb
-rw-r--r-- 1 root root 334389248 Jan  6 00:08 rb_spam.bdb
-rw-r--r-- 1 root root 332099584 Jan  6 00:13 rbtmp.hamHMM.bdb
-rw-r--r-- 1 root root 168296448 Jan  6 00:13 rbtmp.hamHMM.totals.bdb
-rw-r--r-- 1 root root 339763200 Jan  6 00:13 rbtmp.spamHMM.bdb
-rw-r--r-- 1 root root 335945728 Jan  6 00:13 rbtmp.spamHMM.totals.bdb
-rw-r--r-- 1 root root 12288 Jan  5 23:21 trashlist.bdb

mysql> select count(*) from hmmdb;
|  5194934 |

mysql> mysql> select count(*) from hmmham;
|  1248444 |

mysql> select count(*) from hmmhamtot;
|  1123064 |

mysql> select count(*) from hmmspam;
|  1654660 |

mysql> select count(*) from hmmspamtot;
|  1495532 |


From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 06 January 2018 06:54
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

> I’m wondering why I have so many more records when Maxbytes is less and the 
> total data is less.

This is caused by HTML mails - mostly SPAM mails.

You may have a look in to some spam mails with a size of 20.000 and more bytes. 
You'll find some, which are starting with alot of HTML header stuff (CSS and 
script and so on). Most times this content is longer than 6000 byte (your 
MaxByte setting).
I saw mails with a size of 25.000 bytes and 10 words of human readable content.
ASSP tries to get the human readable content of HTML mails for analyzing, but 
if this is not possible, it uses the available data.
The CSS and header content is very different in every mail. Even assp 
normalizes this content anyway, this leads in to much more different HMMdb and 
spamDB records - most of them are useless for spam detection.

Have a look in to the GUI for - Use this HTML Parser (HTMLParser).
I use HTML::Strip.

My current setting for MaxBytes (20.000) is only a long time running try out. I 
want to see, how the detection works from 20.000 to 50.000 bytes setting in 
10.000 bytes steps. Each setting is used for ~1 month. MaxBytes 50.000 has 
passed the test and was perfect - like expected - because 100% of spam mails 
(without an attachment) are perfectly analyzed and detected. How ever, this 
setting leads in to a ~25% performance penalty for the rebuild task (in 
relation to 20.000) using my corpus.

>CPU Model: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz

An nice CPU - but with ASSP's single threaded rebuild task it is slower than my 
older Intel(R) Xeon(R) CPU X5680 @ 3.33GHz. 
http://cpuboss.com/cpus/Intel-Xeon-X5680-vs-Intel-Xeon-E5-2640-v2

Collin, don't care about the overall rebuild speed. It runns at night and it 
does'nt hurt, if it takes an hour more or less. Two steps are time critical: 
populating spamDB and populating HMMdb. As you said "The db part looks to be 
fine". But wait 
It looks like, there is something wrong with the temporary rebuild databases 
used for HMM. This can be also the cause for a very very slow rebuild. >>> The 
rebuild was actually quicker a while back, maybe 40m

>2018-01-05 00:07:42 Start populating Hidden Markov Model. HMM-check is 
>disabled for this time!

>2018-01-05 00:07:43 Total processing time: 7,663 second(s)

This is ONE second time difference - totaly impossible - even if HMMdb is hold 
in RAM 

Is it right, that you use BerkeleyDB for the rebuild? If so -

check the 'tmpDB/rebuildDB/BDB-error.txt' file. It should be zero byte long!

In doubt: shutdown assp, clean the folder  'tmpDB/rebuildDB/', start assp, run 
a rebuild.


Thomas



Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:05.01.2018 21:14
Betreff:Re: [Assp-test] Meltdown/Spectre







From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 05 January 2018 17:16
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



>>time 7,663 seconds, data 486.61 Mbyte

>This is very slow. To be honest - I'm lost for words!

>My rebuild results are:
Mine are very different

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 05 January 2018 17:16
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

>>time 7,663 seconds, data 486.61 Mbyte

>This is very slow. To be honest - I'm lost for words!

>My rebuild results are:
Mine are very different

2018-01-04 22:00:00 Maxbytes: 6,000
2018-01-05 00:03:00 start populating Spamdb with 2,466,760 records - Bayesian 
check is now disabled!
2018-01-05 00:07:42 Start populating Hidden Markov Model. HMM-check is disabled 
for this time!
2018-01-05 00:07:43 Total processing time: 7,663 second(s)
2018-01-05 00:07:43 Total processing data: 486.61 Mbyte
2018-01-05 00:08:37 Uploading Griplist via Direct Connection

The db part looks to be fine considering the times and the extra records that 
mine added. I’m wondering why I have so many more records when Maxbytes is less 
and the total data is less.

My two MX have directly mounted Gluster replicas running off a Fibre channel 
SAN and the rebuild only runs on one.

I have a 4GB tmpDB mounted as tmpfs:

tmpfs  4.0G  1.3G  2.8G  32% /usr/local/assp/tmpDB

Hardware for each is Citrix XenServer 7.2 running on HP DL servers
CPU Model: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
112GB RAM in each with 12GB allocated to each VM
Hard drives aren’t SSD but are on a 1+0 array – I forget how many drives are in 
it but there’s a few. SAN is a Dell Powervault, I’d need to check on the spec.

The VMs are Ubuntu 16.04.3 LTS
16 cores allocated in 4 socket with 4 cores per socket

Primary
top - 20:02:52 up 82 days,  3:40,  1 user,  load average: 0.41, 0.18, 0.11
Tasks: 241 total,   1 running, 240 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 12318500 total,   180648 free,  6131216 used,  6006636 buff/cache
KiB Swap:  8253436 total,  7765076 free,   488360 used.  5702644 avail Mem

Secondary/rebuild
top - 20:02:30 up 66 days,  6:59,  2 users,  load average: 0.05, 0.05, 0.07
Tasks: 250 total,   1 running, 249 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.1 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 12318500 total,   448412 free,  7276144 used,  4593944 buff/cache
KiB Swap:  8253436 total,  6071240 free,  2182196 used.  3396112 avail Mem

ASSP uses 2.3g memory
Clamd about 1G
Gluster 2.2G

Perl is v5.22.1. I believe 5.26 is coming in 18.04 LTS at the end of April 
according to the release schedule. I’ll plan an upgrade sometime after that.

The rebuild was actually quicker a while back, maybe 40m but one of the version 
changes must have had an impact. I couldn’t say which though as I only really 
keep an eye on the amount of data processed and the norm/confidence.

>From my point of view the real bottleneg for the rebuild task is, that only 
>one core (thread) is used by this >task, even there are 12 or more available.
>Because of this (my bad) software design, the speed of a single core matters 
>too much. I think about for >a while to change this. I hope, I'll get this 
>fixed/improved in 2018.

Improvements are always welcome to make a great product even better 

I hope 2018 is good to you.
All the best,
Colin.




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:05.01.2018 16:01
Betreff:Re: [Assp-test] Meltdown/Spectre



Hi Thomas,



Thank you for the input – I do recall previously discussing ISP mode and 
realising that it was for bigger deployments than ours.



We have three servers. Two handling inbound and one specifically for Office 365 
relaying. The two inbound probably do about 50,000 messages per day between 
them according to infostats.



CPU Usage on both frontends is 1.62% avg and 1.49% avg respectively. I only 
have a single MySQL db (general load average is around 0.1 ) and I’ve been 
watching the hypervisor reports on its performance. I did set up a Gluster sync 
between the two frontends so they have access to the same corpus without having 
to do it over the network – that helped with performance however I’ve never 
been able to get the rebuild run to be particularly quick (Last night’s was 
total processing time 7,663 seconds, data 486.61 Mbyte). I haven’t brought it 
up here because it doesn’t really have much of an effect and it is likely in my 
setup rather than an ASSP issue.



So I think I’ll get away with it on my setup, hopefully this information will 
be helpful to other people who are trying to figure out if they’ll be impacted.



All the best,

Colin Waring.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 05 January 2018 13:49
To: ASSP development mailing list 
&l

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

Hi Thomas,

Thank you for the input – I do recall previously discussing ISP mode and 
realising that it was for bigger deployments than ours.

We have three servers. Two handling inbound and one specifically for Office 365 
relaying. The two inbound probably do about 50,000 messages per day between 
them according to infostats.

CPU Usage on both frontends is 1.62% avg and 1.49% avg respectively. I only 
have a single MySQL db (general load average is around 0.1 ) and I’ve been 
watching the hypervisor reports on its performance. I did set up a Gluster sync 
between the two frontends so they have access to the same corpus without having 
to do it over the network – that helped with performance however I’ve never 
been able to get the rebuild run to be particularly quick (Last night’s was 
total processing time 7,663 seconds, data 486.61 Mbyte). I haven’t brought it 
up here because it doesn’t really have much of an effect and it is likely in my 
setup rather than an ASSP issue.

So I think I’ll get away with it on my setup, hopefully this information will 
be helpful to other people who are trying to figure out if they’ll be impacted.

All the best,
Colin Waring.

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 05 January 2018 13:49
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

I remember an ISP issue, who used 10 assp instances with one enterprise MySQL 
backend cluster, sharing all tables for all instances.
In havy workload times (100.000 or even more mails per hour), the MySQL server 
was brought to its end - no matter how many physical resouces were made 
available. Even holding the complete assp DB in the DB-server RAM has not 
solved the problem.
With 100.000 mails per hour and  ~50 DB queries per mail (HMMdb and spamDB), 
the DB server has to process at least 5 million queries in one hour.
If we exclude HMMdb and spamDB, depending on the configuration, there can be 
additionaly 10 to 20 DB queries per mail (for all the other DB-tables). Even 
this can lead in to a very high DB workload!
The URIBL-check can also be very resource expensive (read and write !!!). 
Assume a mail with 100 different URIs is seen the first time - 100 
unsuccessfull cache DB-queries, followed by 100 DNS queries, followed by 100 
cache DB-writes.

To prevent this issue, assp V2 has a buildin ISP mode for HMMdb and spamDB.
In short:
- the corpus of all instances is synchronized to a master instance (rsync for 
example)
- HMMdb and spamDB are hold in memory in each instance and each worker
- HMMdb and spamDB are build on the master system and are distributed as files 
to all other instances using an external script (methode of your choice)
- all other tables are shared traditionaly - but each instance uses a 
configurable DB cache to prevent repeated DB-queries for the same results (for 
example IP checks, helo )

This ISP mode requires at least 16GB RAM per instance, if a maximum of 15 SMTP 
workers is used. Using more than 15 workers in an instance, produces a large 
overhead without any performance improvement.

Collin, I don't know the workload and configuration of your systems - but the 
math is simple.

An possible solution between the standard mode and the ISP mode can be:
- each assp instance has its own DB backend
- all DB-backends are bidirectional synchronized (asynchron) to a 
DB-master-server-cluster

Depending on the overall workload, the DB-master-server-cluster must be an 
enterprise cluster or something like that.
If we assume 10 assp instances, each record change in one instance will lead in 
to one store and nine write sync ops at the master cluster!

If we assume five DB-write ops per mail -> 100 000 mail/h in all instances -> 
500 000 store ops/h + 4.5M sync ops/h at the master cluster.
Yes - the workload at the cluster will be very high, but it is no longer time 
critical and will balance over all the time.
The disadvantage is, that the tables in all instances are never 100% sychron 
and the last instance "winns" in writing the same DB-record. The async state of 
the tables in all DB-backends increases with the overall workload.

You may also think about a ring synchronization between the 10 assp 
DB-backends. The cluster will not be required and the DB-backends will have a 
manageable workload - but the delay of syncing a single record and the data 
inconsitency over all instances will be increased.

Thomas






Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:05.01.2018 10:45
Betreff:[Assp-test] Meltdown/Spectre



Hi All,



I’m wondering if anyone has updated their ASSP/db backends and monitored the 
performance impact yet.



I’m currently wo

[Assp-test] Meltdown/Spectre

[Colin Waring]

Hi All,

I'm wondering if anyone has updated their ASSP/db backends and monitored the 
performance impact yet.

I'm currently working on assessing just how bad this is going to be with how 
many systems I've got to coordinate hypervisor/OS/microcode updates on so I'm 
checking around with everyone to see who's already got some answers.

All the best,
Colin Waring.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP ramps up CPU usage after a time

[Colin Waring]

There is indeed a logout option.

Top left just above the left hand menu. The advice has always been to ensure 
that you log out (then press cancel twice) when finished rather than closing 
the browser, along with only using the root user when absolutely necessary.

All the best,
Colin Waring.


-Original Message-
From: James Moe [mailto:ji...@sohnen-moe.com] 
Sent: 15 October 2017 22:59
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] ASSP ramps up CPU usage after a time

On 10/02/2017 11:39 AM, James Moe wrote:
>
> opensuse v42.2
> linux v4.4.87-18.29-default x86_64
> assp  v 2.5.5(17223)
> perl 5.18.2
> 
> After some up time, usually in the range of 1 - 3 weeks, ASSP starts 
> consuming considerably more CPU time.
>
  Finally tracked down what is actually happening here.
  Apparently one of the worker threads runs the web interface when it is 
accessed from a browser. As soon as I logged into the interface the usage went 
from 0.6% to 11% using Worker_1 thread.
  If at some point I need to log in again (the browser closed erasing all 
cookies), another worker thread is assigned to the interface, increasing the 
CPU usage to 23%.
  Since there is no "logout" option, I presume this would continue until ASSP 
ran out of workers to handle the interface?

--
James Moe
moe dot james at sohnen-moe dot com
520.743.3936
Think.

--
Check out the vibrant tech community on one of the world's most engaging tech 
sites, Slashdot.org! http://sdm.link/slashdot 
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Colin Waring]

16256 works acceptably but shuts down once or twice a day. 16270 or 16274_1 
gave me problems with delays.

I suspect the shutting down is a symptom of a different problem as it has 
happened for a while.

On 30 Sep 2016 17:57, Thomas Eckardt  wrote:
Hmm ... not OK.

for my records:

build 16256 is running fine
builds 16270 and higher make problems

right?

Thomas





Von:cw 
An: ASSP development mailing list 
Datum:  30.09.2016 17:19
Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
servers



I've had to roll back now unfortunately as I'm getting email problems
again
:(

On Fri, Sep 30, 2016 at 3:50 PM, cw  wrote:

> Mixed results on this. So far no problems with running workers being
> logged but the GUI has become incredibly unresponsive. By unresponsive I
> mean I waited a good couple of minutes for the shutdown_list page to
load.
> The dot on the main page is red yet the workers page is all green.
> Scratch that, it has refreshed again and I have a worker stuck:
> Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
> filename name (stuck)
> 30s later and it is healthy again..
>
> On the server I haven't upgraded the shutdown_list page comes up within
> seconds. I'm not sure whether to leave it running or whether this is
> evidence of the same kind of unresponsiveness that cause me to have to
roll
> back earlier this week.
>
> On Fri, Sep 30, 2016 at 3:29 PM, cw  wrote:
>
>> I wish I'd spotted this before writing out the other message. I'll give
>> it a test now for you.
>>
>> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> Collin, this should no longer happen using the updated 2.5.2 16274_1
at
>>> CVS /test
>>>
>>> Thomas
>>>
>>>
>>>
>>> Von:cw 
>>> An: ASSP development mailing list

>>> Datum:  29.09.2016 16:40
>>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>>> servers
>>>
>>>
>>>
>>> Hi Thomas,
>>> I moved up to 16270 following this thread of discussion but then had a
>>> day
>>> working away. I've come back to huge issues with delays, mails not
going
>>> through and many, many of these in the logs:
>>>
>>> Info: unable to detect any running worker for a new connection - wait
>>> (max
>>> 30 seconds)
>>>
>>> When I say many, I have over 21,000 lines in today's log file. I also
>>> found
>>> the GUI unresponsive or not connecting at all and ASSP restarting
quite
>>> regularly.
>>>
>>> I've dropped back to 16256 and things are instantly better. Do you
think
>>> going up to 16273 might improve things over 16270 or am I better
holding
>>> off for now?
>>> All the best,
>>> Colin.
>>>
>>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
>>> 
>>> wrote:
>>>
>>> > I just released 2.5.2 build 16273 at CVS test folder
>>> >
>>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>>> >
>>> > This release should make a very large difference for SSL/TLS mails
sent
>>> by
>>> > hosts that uses small SSL-frame size.
>>> >
>>> > Tell me your test results.
>>> >
>>> >
>>> > Thomas
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Von:K Post 
>>> > An: ASSP development mailing list
>> >
>>> > Datum:  28.09.2016 19:42
>>> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses
/
>>> > servers
>>> >
>>> >
>>> >
>>> > But I want a postman driving a Ferarri with monster truck tires that
>>> can
>>> > roll over the traffic (and if wishes are being granted, I'd prefer
the
>>> car
>>> > in a deep blue instead of classic red).
>>> >
>>> > We regularly see people attaching large files or a bunch of smaller
>>> ones
>>> > that add up to a big email, I'm talking lots and lots of different
>>> people
>>> > from outside the organization sending to us, and this happens on a
>>> daily
>>> > basis.  It's especially popular with photos and huge scans
multi-page
>>> > 600dpi (which people don't understand can be done at low
resolution).
>>> > Often it's people sending in scanned official documents for us to
>>> review
>>> > an
>>> > help them.  They're not our staff, they're the people we help.  They
>>> have
>>> > a
>>> > tendency of not following any instructions, and ignore the fact that
we
>>> > have a web based system for the process.  We can't control it and
the
>>> > powers that be don't want us lowering the 30 MB threshold across the
>>> > board.  Lot of these people use gmail.com addresses and google
allows
>>> for
>>> > up to 25 MB - https://support.google.com/mail/answer/6584
>>> >
>>> > I think it's really interesting that google seems to use this
>>> inefficient
>>> > small packet size for SSL, allows for 25MB emails, is a big
proponent
>>> of
>>> > SSL, and at the same time doesn't allow mails to take more 

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Colin Waring]

I have been running IO::Socket::SSL 2.0.33 though have just updated to 2.0.38. 
I don't think this is going to be related as I have seen this issue for a long 
time and will undoubtedly have had previous versions of OpenSSL.

Don't forget that I see the issue from more than just Google.

I'm quite pushed for time at the moment. Ken, what did you do specifically to 
grab the necessary debugs? - save me having to stop and think :)

All the best,
Colin Waring.


Colin Waring
Technical Manager
Dolphin ICT Limited
T
+44 (0)151 438 2246 Ext 2003
www.dolphinict.co.uk
co...@dolphinict.co.uk
US15a, Armstrong House, First Avenue, Robin Hood Airport, Doncaster, DN9 3GA





Dolphin ICT Limited. NOTICE & DISCLAIMER Dolphin ICT Limited, a private limited 
company, with company registration number 6206916, registered in the United 
Kingdom, the registered office of which is at US15a, Armstrong House, First 
Avenue, Robin Hood Airport, Doncaster, DN9 3GA VAT registration number GB 918 
1896 88. 



-Original Message-
From: K Post [mailto:nntp.p...@gmail.com]
Sent: 27 September 2016 04:53
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

I have IO::Socket::SSL 2.036 installed instead of 2.020.  Could this have 
anything to do with any of this?

On Mon, Sep 26, 2016 at 11:49 PM, K Post <nntp.p...@gmail.com> wrote:

> THANK YOU again for taking all the time on this.  It's nuts that this 
> only seems to happen (to me and others reporting) with TLS on and mail 
> sent through google servers.
>
> I've confirmed the version of Convert::Scalar to be 1.11
>
> I'll get you a debug log privately, but here's what I'm seeing with 
> the latest version:
>
> 11mb attachment, tls on, newest version, but without the 
> $main::neverQueueSize = 4194304; line took 620 seconds.  That's better 
> than the 772seconds that saw before I but still pretty terrible - and 
> of course, that's only one test.
>
> I see a message which I assume is now expected:
> message is too large ( SIZE 15700413 byte > neverQueueSize 1200
> byte) to be queued for further internal processing! Skipping DKIM, 
> Plugins and charset conversion. for that message
>
> I saw a X-ASSP-KEEP line in the header too.  Don't know what that means.
> Haven't seen that before.
>
> Once I added the $main::neverQueueSize = 4194304; line to 
> ASSP_Correct.pm, speed improves for sure.  It took 327 seconds.  Still 
> really slow considering that without TLS it only takes 19 seconds.
> Similar line noting the 4MB size limit Removing the full message 
> analysis seems like a shame especially since it doesn't seem to even 
> stutter if TLS is off.
>
> So more questions for your consideration
> 1) What is TLS doing that slows things down so much for GOOGLE mails 
> only (or at least only google that I've seen be slow)
> 2) What encryption related modules need checking?
> 3) Why would things be fine on your old Windows 2003 rig, but clearly 
> not okay on my (presumably) faster machine
> 4) What is similar between my machine and the others who reported TLS 
> problems with Google.  I know one at least was a Linux rig.
>
>
>
>
>
>
> On Mon, Sep 26, 2016 at 4:02 AM, Thomas Eckardt < 
> thomas.ecka...@thockar.com> wrote:
>
>> First, thank you for the debug file.
>>
>> There is one big problem. The debug file explains the general 
>> behavior of the slowing down connection while the data size is growing.
>> It not explains, why this should only happens at connections from 
>> gmail.com and only if TLS is used.
>>
>> looking at the following timeline - the *** lines are from me and are 
>> showing the count of read-socketcalls within this second
>>
>> 
>> Sep-23-16 21:14:37 [Worker_2] > IO::Socket::INET=GLOB(0x11c1e3bc) (6)<DATA[CR][LF]
>> Sep-23-16 21:14:37 [Worker_2] > Sep-23-16 21:14:38 [Worker_2] > Sep-23-16 21:14:39 [Worker_2] > (each 1440 byte) 164 ...
>> Sep-23-16 21:14:40 [Worker_2] > (each 1440 byte) 167 ...
>> Sep-23-16 21:14:41 [Worker_2] > (each 1440 byte) 108 ...
>> Sep-23-16 21:14:42 [Worker_2] > (each 1440 byte) 95 ...
>> Sep-23-16 21:14:43 [Worker_2] > (each 1440 byte) 82 ...
>> Sep-23-16 21:14:44 [Worker_2] > (each 1440 byte) 74 ...
>> Sep-23-16 21:15:09 [Worker_2] > (each 1440 byte) 43 ...
>> Sep-23-16 21:15:39 [Worker_2] > (each 1440 byte) 35 ...
>> Sep-23-16 21:16:39 [Worker_2] > (each 1440 byte) 21 ...
>> Sep-23-16 21:18:39 [Worker_2] > (each 1440 byte) 12 ...
>> Sep-23-16 21:22:41 msg79676-04975 209.85.223.177 
>> <nntp.p...@gmail.com>
>> to:
>> testtls@[[ OUR DOMAIN ]].org info: message is too large (

Re: [Assp-test] invalidFormatHeloRe

[Colin Waring]

Hi Thomas,

Thanks for the reply. That was actually a typo in my email, I did mean 
validFormatHeloRe not invalidFormatHeloRe.

The file on SourceForge is out of date. It hasn't been updated in nearly three 
years and still has w{2,6} in it - I checked this prior to posting as my way of 
keeping things up to date is comparing them with 
http://assp.cvs.sourceforge.net/viewvc/assp/assp2

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 09 September 2016 08:38
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] invalidFormatHeloRe

>invalidFormatHeloRe

No - 'validFormatHeloRe' makes this rule.

This regex was change at the begin of this year (I think) - the default is

file:files/validhelo.txt

validhelo.txt:

^(?:\w[\w\.\-]*\.\w{2,64})$
^[a-fA-F0-9]{1,4}:([a-fA-F0-9:]{1,4}){1,}(?:(?:\.\d+){3})?$


Thomas




Von:cw <colin.war...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  08.09.2016 13:26
Betreff:[Assp-test] invalidFormatHeloRe



Hi,



I’m not an expert at Regexs otherwise I’d look at this myself.



I’ve had someone emailing me about problems getting mail through and at first 
glance it was due to an invalid HELO. At second glance, the HELO is actually 
valid and points to a domain that has a valid DNS record. The HELO is 
server.kalo.digital



This fails the default regex for invalidFormatHeloRe because the regex
stipulates that the last part of the HELO has to be between 2 and 6
characters long. This doesn’t take into account the more recent TLDs that
have been forced on the Internet of which .digital is one being 7
characters.



I can’t find anything in RFC1123 that specifically states the number of
characters for the TLD so is this a problem with the Regex rather than the
usage of TLDs with more than 6 characters?



All the best,

Colin Waring.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Colin Waring]

I have to say I've seen this and I posted about it back in January.

https://sourceforge.net/p/assp/mailman/message/34783916/

Back then I saw problems with Gmail, Yahoo Mail and SMTPRoutes. Since then I've 
occasionally fielded calls from different people saying that emails aren't 
coming through and the solution has been to add the IP to noTLSip. The problem 
was much more significant back in January because I was getting lots of 
complaints whereas now it is only occasional.

I'm on a completely different architecture to you.

Ubuntu 14.04.4 LTS, OpenSSL 1.0.1f (latest from apt), Perl v5.18.2, Net::SSLeay 
1.74, IO::Socket::SSL 2.033, Net::SMTP::SSL 1.03

I've been using cpanm and cpanoutdated to manage module updates, checking from 
within cpan I can see that a number of modules haven't been done that way so 
I'm running upgrade from within CPAN itself to get things up to date. One of 
the updates is Net:SSLeay 1.77 so I'll see what that does.

All the best,
Colin Waring.


Colin Waring
Technical Manager
Dolphin ICT Limited
T
+44 (0)151 438 2246 Ext 2003
www.dolphinict.co.uk
co...@dolphinict.co.uk
US15a, Armstrong House, First Avenue, Robin Hood Airport, Doncaster, DN9 3GA





Dolphin ICT Limited. NOTICE & DISCLAIMER Dolphin ICT Limited, a private limited 
company, with company registration number 6206916, registered in the United 
Kingdom, the registered office of which is at US15a, Armstrong House, First 
Avenue, Robin Hood Airport, Doncaster, DN9 3GA VAT registration number GB 918 
1896 88. 



-Original Message-
From: K Post [mailto:nntp.p...@gmail.com]
Sent: 01 August 2016 23:06
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: [Assp-test] Inbound TLS from gmail.com addresses / servers

I originally thought that we had a problem with all TLS inbound email.  As it 
turns out, my conclusion appears to have been wrong.


   - There are some SLOW servers outside that are just plain slow (nothing
   I can do there),

   - TLS seems to work reasonably fast with most inbound mail, though
   significantly slower than without TLS  (5 seconds for an 11mb file without
   tls, vs 45 seconds with TLS on)

   - GMAIL.com inbound TLS emails are SLOW, no matter what settings I tweak


With inbound gmail.com message. if I have TLS off, an 11mb attachment is 
delivered through ASSP in under 5 seconds.  With TLS on it takes close to
10 minutes, which gets close to gmail's limit.

I've tested with Outlook.com and that same 11mb attachment comes in through 
ASSP with TLS on in about 45 seconds.

Sending a 30mb attachment from gmail FAILS because it takes too long. gmail 
will try for I believe 10 minutes to send a message, then it quits and retries. 
 After a couple tries, it sends an NDR.

This is a Windows 2012 R2 server, latest ASSP dev, OpenSSL 1.0.2h installed 
from slproweb.com/products/Win32OpenSSL.html (though I've also tried with the 
OpenSSL I downloaded a while back from the ASSP sourceforge site.
 net::ssleay 1.74 (openssl 1.0.2g).  I'm almost certain that the OpenSSL 
installation is not used by ASSP, but I've not been able to get confirmation of 
that here.

Just updated IO::Socket::SSL to 2.033.
Net::SMTP:SSL 1.02.

CPU usage as reported by assp is 4.78%.  It's not on the fastest machine in the 
world (it's a hypver-v guest on a decent machine), but it seems speedy enough.  
24gb ram.  We've got similar physical hosts running Exchange as a guest without 
any speed issues whatsoever.

Any other info I can provide to help figure this out?

Disabling TLS for any gmail inbound mail isn't a feasible option, plus I don't 
know if it really is just google, or just the way that google connects which 
others might too...

Thank you all.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] More MX and A record lookup issues

[Colin Waring]

You need debug logs and set something up to monitor your DNS traffic. You need 
to be certain whether the issue is with ASSP handling DNS or your DNS setup. 
This information is the only thing that will really let you track your issue 
down.

All the best,
Colin Waring.

-Original Message-
From: K Post [mailto:nntp.p...@gmail.com] 
Sent: 19 May 2015 14:57
To: ASSP development mailing list
Subject: [Assp-test] More MX and A record lookup issues

Running 15135 on a Windows 2012 box.

I've got a message that was ultimately erroneously rejected due to total score. 
 Contributing to this score is ASSP being (for some reason) unable to find A or 
MX records for the sending IP.  This isn't the first time I've seen this.  My 
last suggestion of potentially having ASSP retry dns lookups if neither A or MX 
returns anything was dismissed as crazy.  I don't know what else to suggest.  
Here's what I'm seeing:

In analyze everything looks great:
• domain bounce.e.hautelook.com (in Mail From:) has a valid MX record:
bounce-mx.exacttarget.com
• domainMX bounce-mx.exacttarget.com has a valid A record: 66.231.91.54 • 
domain e.nordstromrack.com (in From , Reply-To) has a valid MX record:
reply-mx.s6.exacttarget.com
• domainMX reply-mx.s6.exacttarget.com has a valid A record: 198.245.82.46 • 
198.245.83.134 SenderBase: status=white SenderBase, data=[CN=US, 
ORG=EXACTTARGET, DOM=hautelook.com, BLS=, HNM=Y, CIDR=20, HN= 
mta6.e.hautelook.com] Senderbase should have given a bonus, the A and MX record 
is there, so it shouldn't have counted against the message.

But in the message in the corpus, I see:
X-ASSP-Message-Score: 10 (MX missing: bounce.e.hautelook.com (Mail From:))
X-ASSP-IP-Score: 10 (MX missing: bounce.e.hautelook.com (Mail From:))
X-ASSP-Message-Score: 15 (A record missing: bounce.e.hautelook.com (Mail
From:))
X-ASSP-IP-Score: 15 (A record missing: bounce.e.hautelook.com (Mail From:)) 
Senderbase doesn't seem to have run either

I see nothing else to indicate that the machine is having DNS problems of any 
kind.  It's looking to a set of internal DNS servers that are fast and reliable 
- they're used for all of our servers and none of them have any dns issues.

It's not light exacttarget, a major mailing company used by big companies, 
temporarily removed the A and MX records for this hostname.

Any idea of what could be going on and how to correct it?  Could it be that 
this is happening to others but I'm the only one going through almost every 
questionally blocked message by hand (hate this part)??


Thanks
--
One dashboard for servers and applications across Physical-Virtual-Cloud Widest 
out-of-the-box monitoring support with 50+ applications Performance metrics, 
stats and reports that give you Actionable Insights Deep dive visibility with 
transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
--
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] speed of adding records to spamdb table

[Colin Waring]

My assumption would be that is the estimate of the number of seconds it will 
take the process to complete.

Our rebuild takes about 10 seconds to populate the database, you do need to do 
some network tuning and make sure your database is optimised for purpose, I 
can't help you with MS SQL though.

All the best,
Colin Waring.

-Original Message-
From: K Post [mailto:nntp.p...@gmail.com] 
Sent: 23 April 2015 15:25
To: ASSP development mailing list
Subject: [Assp-test] speed of adding records to spamdb table

Working to get the rebuild process to complete.  Win32.  MS SQL DB.

Are these speeds normal??  I'm a little confused to the sec numbers, It's not 
5,000+ seconds, I don't think. and I don't know why secs would be decreasing.  
confused.

Apr-23-15 10:21:37 Added 176152 of 998035 records for table spamdb - finished 
in 5081 sec
Apr-23-15 10:21:38 Added 176346 of 998035 records for table spamdb - finished 
in 5078 sec
Apr-23-15 10:21:40 Added 176540 of 998035 records for table spamdb - finished 
in 5081 sec
Apr-23-15 10:21:42 Added 176928 of 998035 records for table spamdb - finished 
in 5077 sec
Apr-23-15 10:21:47 Added 177704 of 998035 records for table spamdb - finished 
in 5073 sec
Apr-23-15 10:21:48 Added 177785 of 998035 records for table spamdb - finished 
in 5075 sec
Apr-23-15 10:21:49 Added 177940 of 998035 records for table spamdb - finished 
in 5074 sec
Apr-23-15 10:21:53 Added 178560 of 998035 records for table spamdb - finished 
in 5071 sec
Apr-23-15 10:21:55 Added 178870 of 998035 records for table spamdb - finished 
in 5069 sec
Apr-23-15 10:21:57 Added 179180 of 998035 records for table spamdb - finished 
in 5068 sec
Apr-23-15 10:21:59 Added 179490 of 998035 records for table spamdb - finished 
in 5066 sec
Apr-23-15 10:22:01 Added 179800 of 998035 records for table spamdb - finished 
in 5065 sec
--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own 
process in accordance with the BPMN 2 standard Learn Process modeling best 
practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ 
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ClamAV win32 Sane

[Colin Waring]

The subject tests shouldn't require AFC at all, as the subject comes early on 
in the message clamav should catch it normally.

I'm not sure if there's a debug option for the scanning or afc. You could turn 
on general debug, run the test then turn it off again.

For clamd itself you might need to make sure the logging is configured for 
Windows:

LogFile C:/ClamAv/Logs/clamd.log
LogTime yes
LogClean yes
LogFileMaxSize 0

The latter two won't be needed for normal operation as they will produce larger 
log files.

All the best,
Colin Waring.

-Original Message-
From: K Post [mailto:nntp.p...@gmail.com] 
Sent: 16 March 2015 15:28
To: ASSP development mailing list
Subject: Re: [Assp-test] ClamAV win32 Sane

Thank you Colin!!

I have almost the same settings as yours.   The only differnce is DoASSP_AFC
is set to both.  I tried yesterday with AFC off though, and it's still not 
caught.

When tests 1 and 3 get caught, it does appear that the sane signatures are 
catching them:

Mar-14-15 16:06:08 msg63566-10522 209.85.220.175 
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org ClamAV:
scanned 2232 bytes in whitelisted message - FOUND 
Sanesecurity.TestSig_Type4_Bdy.3.UNOFFICIAL
Mar-14-15 16:06:08 msg63566-10522 209.85.220.175 
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
Message-Score: added 50 (vdValencePB) for virus detected:
'Sanesecurity.TestSig_Type4_Bdy.3.UNOFFICIAL', total score for this message is 
now 35
Mar-14-15 16:06:08 msg63566-10522 [VIRUS] 209.85.220.175 
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org [spam 
found] (virus detected: 'Sanesecurity.TestSig_Type4_Bdy.3.UNOFFICIAL') [3rd in 
body] - messages/discarded/3rd_in_body--67.txt;

But, yeah, when it's only the subject that has the test, I see AFC pluggin 
being called, but no hit!

Not sure where else to look or what else to try.  It's certainly not the end of 
the world, but I worry based on the Sane guy saying how important this one is - 
that headers are often what's in the signature files.


On Mon, Mar 16, 2015 at 5:34 AM, Colin Waring co...@dolphinict.co.uk
wrote:

 Your log looks to me like the settings simply aren't calling Clam to 
 scan the message rather than clam missing the message.

 I have ScanWL, ScanNP, ScanLocal, ScanCC and UseAvClamd enabled and 
 you need to make sure that AvClamdPort is correct for your system.
 DoASSP_AFC is set to enabled but only set to do attachments. If you 
 haven't got the main clam settings enabled, you'll need to make sure 
 that ASSP_AFCSelect is set to one of the options that scans the whole message.

 2015-03-15 15:34:57 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com info: found message size announcement:
 1.56 kByte
 2015-03-15 15:34:57 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com IP 209.85.214.176 matches 
 whiteListedIPs - with 209.85.128.0/17
 2015-03-15 15:34:57 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com [SMTP Reply] 250 OK
 2015-03-15 15:34:57 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld [SMTP 
 Reply]
 250 Accepted
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld [SMTP 
 Reply]
 354 Enter message, ending with . on a line by itself
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld Whitelisted 
 sender address: sen...@gmail.com for recipient recipi...@domain.tld
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld 
 DKIM-Signature found
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld info: 
 domain gmail.com has published a DMARC record
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld ClamAV:
 scanned 1774 bytes in whitelisted message - FOUND
 Sanesecurity.TestSig_Type4_Hdr.2.UNOFFICIAL(740814f660dc883f8fe4646084
 30ae9f:1774)
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld Message-Score:
 added 50 (vdValencePB) for virus detected:
 'Sanesecurity.TestSig_Type4_Hdr.2.UNOFFICIAL(740814f660dc883f8fe464608
 430ae9f:1774)', total score for this message is now 50
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out] 
 [VIRUS]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld [spam 
 found] (virus detected:
 'Sanesecurity.TestSig_Type4_Hdr.2.UNOFFICIAL(740814f660dc883f8fe464608
 430ae9f:1774)') 
 [rrg63Uhj2UCyECcruX7D83A4qd5UA5vnlgwJp6b6fmPZpObZJAbftehuhRAXFby] - 
 /usr/local/assp/store/quarantine/rrg63Uhj2UCyECcruX7D83A4qd5UA5vnlgwJp
 6b6fmPZpObZJA--571715.eml;
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS

Re: [Assp-test] ClamAV win32 Sane

[Colin Waring]

Howdy,

I think you need to pull some logs for both ASSP and clam. I've run the tests 
on my install and they all got blocked properly.

I'm not using Windows though so can't help with the setup.

All the best,
Colin Waring

On 14 Mar 2015 20:07, K Post nntp.p...@gmail.com wrote:
Correction, the first 2 sane tests slip through, 3rd IS trapped.

On Sat, Mar 14, 2015 at 4:05 PM, K Post nntp.p...@gmail.com wrote:

 I've got the sane signatures installed on a windows box with ASSP.

 Has anyone tried these tests?
 http://sanesecurity.com/support/signature-testing/

 I've tried this with and without the AFS plugin.  Same results.  All 3
 messages arrive.

 UseAVClamD is on
 DoFileScan is off

 When I run tests from http://www.emailsecuritycheck.net/, some of the
 tests are coming through as well, but some are caught.

 Any suggestions would be appreciated.

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ClamAV win32 Sane

[Colin Waring]

I'll look into them for you but it'll be tomorrow before I do.

All of them got blocked, though I did see the same effect on gmail from the 
HTML one.

All the best,
Colin Waring

On 15 Mar 2015 18:32, K Post nntp.p...@gmail.com wrote:
Colin-
really, I'm just interested in the results of the 2nd test in your log.  I
managed to get the html email one to be trapped - apparently sending html
mail from gmail is a bit different.  From outlook it trapped it.

The one where the spam string is in the subject however, doesn't seem to be
caught though.  It looks like one of our bombre is scoring the long
subject.  I don't now why that would stop a detection though.  It does look
like the ASSP_AFC is being called (it was enabled for this test).


Mar-15-15 14:27:37 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
Received-RWL: listed from list.dnswl.org; client-ip=209.85.220.177
Mar-15-15 14:27:37 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
Message-Score: added -2 for 209.85.220.0 in griplist (0.14), total score
for this message is now -42
Mar-15-15 14:27:37 msg44055-12284 [DKIM] 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org [scoring]
DKIM signature failed - none - sender policy is: neutral - author policy
is: neutral
Mar-15-15 14:27:37 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
Message-Score: added 10 (dkimValencePB) for DKIM none, total score for this
message is now -32
Mar-15-15 14:27:38 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org info:
SenderBase - query using SenderBase
Mar-15-15 14:27:38 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
SenderBase -- used Senderbase -- country:US orgname:GOOGLE domain:google.com
Mar-15-15 14:27:39 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org HMM is
not available - hmmdb is still locked by a rebuild task
Mar-15-15 14:27:40 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org Bayesian
Check [monitoring] - Prob: 1.0 = spam
Mar-15-15 14:27:40 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org [Plugin]
calling plugin ASSP_AFC
Mar-15-15 14:27:40 msg44055-12284 [MessageOK] 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org message
ok [rrg63Uhj2UCyECcruX7D83A4qd5UA5vnlgwJp6b6fmPZpObZJAbftehuhRAXFby] -
messages/okmail/rrg63Uhj2UCyECcruX7D83A4qd5UA5vnlgwJp6b6fmPZpObZJA--73.txt


I've got the sanesecurity.ftm database there, last modified 9/3/14

Thank you for your help!
--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Localdomains stopping working

[Colin Waring]

Thanks again for the reply,

I've stayed away from that because I always intended to have a central logging 
server thus would need syslog for that, it just hasn't happened yet!

I'll have to look into LDAP, it makes sense that you could use a group in the 
flat files and then manage everything through that. 

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 11 March 2015 07:37
To: ASSP development mailing list
Subject: Re: [Assp-test] Localdomains stopping working

Monitoring runs on localhost

You should have a look in to the assp-monitor.pl script. This script emulates a 
SYSLOG server. If syslog is configured in assp to send the log to ths 
assp-monitor SYSLOG server, the script will watch permanently if assp is 
running or not. You have to modify the script for your local needs, like: IP, 
Port, timing values, restart command and so on. But this is easy to see.
The advantage of this script is, that assp is monitored even the instance is 
idle for hours.

Some of our configuration files are generated externally, such as
localdomains

In this case assp rereads the file every 5 minutes (per default). Here we 
have the five minutes - and the reload is normal. Make sure your external 
collection script makes no mistake! 

I just set up different users so we could stop using root, clicked logout 
and got the login prompt.

You have to click cancel in the login prompt - this should be shown in the 
login prompt window.

The sequence in maillog.txt is like this:

Mar-11-15 07:59:52 [Main_Thread] Admin connection from user root on host 
***; page:/logout; session-ID:31d32662563be88bd596b72bb20bcb3c;
Mar-11-15 07:59:52 [Main_Thread] Logout from admin interface requested for 
user 'root' at '**'
Mar-11-15 07:59:52 [Main_Thread] Terminated WEB session 
31d32662563be88bd596b72bb20bcb3c for user 'root' at ''
Mar-11-15 07:59:52 [Main_Thread] Terminated WEB session 
6eb2b017b825cd3defc7c48c441ab01b for user 'root' at ''
Mar-11-15 07:59:52 [Main_Thread] Terminated WEB session 
3e8252de5c6b289718e69c86a8b68ad1 for user 'root' at ''

 Would there be a preferred way to have any updates sent to ASSP rather 
than overwriting the file?

I prefer using LDAP and the Groups feature for registering and classifying 
domains, IP's and users.
The concept of assp allows to have a central LDAP server where all 
domains, groups, IP's and users are registered.
As a result, the usage of the assp GUI is only required for major 
configuration changes - all other domain , IP and user based changes have 
to be only done in the LDAP directory.

Thomas





Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  10.03.2015 20:21
Betreff:Re: [Assp-test] Localdomains stopping working



Actually this raises a few other questions (sorry!).

Monitoring runs on localhost and the script basically calls the telnet 
command then searches the output for Connected. The web admin is 
configured to use https so the monitoring command should never actually 
set up a session with ASSP. I'll need to do a bit more with the script to 
change it to look for a particular response on port 3.

Some of our configuration files are generated externally, such as 
localdomains which comes from a combination of different systems. Would 
there be a preferred way to have any updates sent to ASSP rather than 
overwriting the file? I know this isn't causing the problem as the first 
thing I did was stop the scripts involved.

I just set up different users so we could stop using root, clicked logout 
and got the login prompt. When I tried to log back in I got user root is 
currently logged on from host 10.0.5.51 - no new sessions will be accepted 
until root has logged off. So it looks like even though I clicked logout 
the session didn't get cleaned up properly.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 10 March 2015 16:04
To: ASSP development mailing list
Subject: Re: [Assp-test] Localdomains stopping working

It doesn't authenticate and doesn't attempt to do anything with the
connection.
I wouldn't have thought that an unauthenticated connection would be 
able
to have any impact


The reason is the root login without an logout. assp caches the complete 
web communication for the root account. Because it is doing this, no other 
login is allowed while root has an active login.
Now for example - if the monitor (5) runs on the same system or is 
connected from the same IP (NAT) like a root-web session it may possible 
(should not, but who knows) that the monitor connection is misinterpreted. 

There is simply no web connection code in assp, that expects a non-browser 
session.
The web code of assp is written for browsers and it is not perfect in 
terms of security if http is used. For this reason https should be used

Re: [Assp-test] Localdomains stopping working

[Colin Waring]

Hi Thomas,

Thank you for the very in depth responses. You're a star as always. I'll give 
them a proper review later.

My first thought is that the monitoring script that I use only checks that it 
can open a connection. It doesn't authenticate and doesn't attempt to do 
anything with the connection. I wouldn't have thought that an unauthenticated 
connection would be able to have any impact on the configuration as that seems 
like a significant security issue.

The monitoring script runs every 60s not five minutes, I did previously look at 
SNMP but couldn't get any results so I'll add that to the high priority list. I 
use that script as it has other monitors in such as queue length, MTA 
monitoring and some system admin tasks.

We will definitely stop using the root login though. Strange how we haven't 
seen any issues at all until last week.

All the best,
Colin Waring

On 10 Mar 2015 10:38, Thomas Eckardt thomas.ecka...@thockar.com wrote:
Colin - I find it hard to believe. You brought home the bacon. :):):)

NEVER EVER use the web listerner 5 to monitor assp - this can lead in
to unexpected config changes or config reloads - in worth case you can
lose parts or the complete configuration.



These are very BASIC IT rules - and they also applies to assp:

Don't login to assp as 'root'. Use 'root' only, if you need to access
restricted configuration parameters.
NEVER forget to use the 'logout' button in the GUI - especially NOT if
root is logged on!


2015-03-09 09:38:34 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 106
records
2015-03-09 09:43:33 [Main_Thread] Adminupdate: [root 192.168.11.13] file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains'

2015-03-09 21:37:10 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 104
records
2015-03-09 21:42:11 [Main_Thread] Adminupdate: [root 192.168.11.13] file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains'

exactly 5 minutes difference - Colin, can you remember about this 5
minutes - is it an accidental circumstance, that the monitor to port 5
is running every 5 minutes ??
But - it is NOT a  accidental circumstance, that the last root web-session
was not logged out!

all has been said

2015-03-09 00:04:33 [Main_Thread] Info: added schedule : BlockReport -
for : *@domain.tld=*=1= - at : 0 0,4,8,12,16,20 * * * - next run is at
: 2015-03-09 04:00:00

this is normal - the MaintThread has changed the file after the
blockreport is done

2015-03-09 02:42:11 [Main_Thread] Option list file:
'/usr/local/assp/files/droplist.txt' reloaded (droplist) with 658 records

this is normal - the MaintThread has download the file

This is a huge problem, as localdomains errors cause mail to be
incorrectly rejected and leads to serious complaints. If I can't resolve
this within the next few days I'm likely to have to switch to a different
product which I really don't want to do.

good luck


Thomas






Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  10.03.2015 10:05
Betreff:Re: [Assp-test] Localdomains stopping working



Hi again,

This looks to be a more serious issue now affecting other config files. It
appears that ASSP reloads the flat files and gets the entries wrong.
192.168.11.X is my home office subnet that is allowed access to the admin
interface via VPN. This brings up two things.

1) At first glance it looks like ASSP is incorrectly and sometimes
partially reloading the localdomains file whenever a setting is changed
via the admin interface. Localdomains.txt did not change at all yesterday
yet we have differing numbers of entries indicating the file was only
partially loaded.
2) The first entry at 00:34:50 is impossible. The router for 192.168.11.X
was turned off at approximately 22:30 and not turned back on until 07:00
therefore there could not have been any admin update from the 192.168.11.X
subnet.
3) None of these coincide with actual connections to the admin interface.
There are no logs preceding that say IP 192.168.11.X matches
allAdminConnectionsFrom. The only admin connections to this instance were
at 2015-03-08 14:42:01 from .11 and 2015-03-09 08:02:14 from .13

2015-03-09 00:34:50 [Main_Thread] Adminupdate: [root 192.168.11.11] file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains' was
changed
2015-03-09 00:34:50 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 139
records
2015-03-09 09:38:34 [Main_Thread] Adminupdate: [root 192.168.11.13] file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains' was
changed
2015-03-09 09:38:34 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 106
records
2015-03-09 09:43:33 [Main_Thread] Adminupdate: [root 192.168.11.13] file
'/usr/local/assp/files

Re: [Assp-test] Localdomains stopping working

[Colin Waring]

Actually this raises a few other questions (sorry!).

Monitoring runs on localhost and the script basically calls the telnet command 
then searches the output for Connected. The web admin is configured to use 
https so the monitoring command should never actually set up a session with 
ASSP. I'll need to do a bit more with the script to change it to look for a 
particular response on port 3.

Some of our configuration files are generated externally, such as localdomains 
which comes from a combination of different systems. Would there be a preferred 
way to have any updates sent to ASSP rather than overwriting the file? I know 
this isn't causing the problem as the first thing I did was stop the scripts 
involved.

I just set up different users so we could stop using root, clicked logout and 
got the login prompt. When I tried to log back in I got user root is currently 
logged on from host 10.0.5.51 - no new sessions will be accepted until root has 
logged off. So it looks like even though I clicked logout the session didn't 
get cleaned up properly.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 10 March 2015 16:04
To: ASSP development mailing list
Subject: Re: [Assp-test] Localdomains stopping working

It doesn't authenticate and doesn't attempt to do anything with the
connection.
I wouldn't have thought that an unauthenticated connection would be 
able
to have any impact


The reason is the root login without an logout. assp caches the complete web 
communication for the root account. Because it is doing this, no other login is 
allowed while root has an active login.
Now for example - if the monitor (5) runs on the same system or is 
connected from the same IP (NAT) like a root-web session it may possible 
(should not, but who knows) that the monitor connection is misinterpreted. 
There is simply no web connection code in assp, that expects a non-browser 
session.
The web code of assp is written for browsers and it is not perfect in terms of 
security if http is used. For this reason https should be used and if anyhow 
possible a Client-SSL-certificate authentication should be configured mandatory.

You're a star as always.

No, I'm a gyp artist.
Call me Betelgeuse :):)


Colin, do a telnet to assp port 3 (webStatPort) and press two times enter - 
you'll get the right answer - 'healthy'
or the bad one - 'not healthy'. Both answers are configurable. I think your 
monitor don't need to know more.

Thomas



Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  10.03.2015 13:30
Betreff:Re: [Assp-test] Localdomains stopping working



Hi Thomas,

Thank you for the very in depth responses. You're a star as always. I'll give 
them a proper review later.

My first thought is that the monitoring script that I use only checks that it 
can open a connection. It doesn't authenticate and doesn't attempt to do 
anything with the connection. I wouldn't have thought that an unauthenticated 
connection would be able to have any impact on the configuration as that seems 
like a significant security issue.

The monitoring script runs every 60s not five minutes, I did previously look at 
SNMP but couldn't get any results so I'll add that to the high priority list. I 
use that script as it has other monitors in such as queue length, MTA 
monitoring and some system admin tasks.

We will definitely stop using the root login though. Strange how we haven't 
seen any issues at all until last week.

All the best,
Colin Waring

On 10 Mar 2015 10:38, Thomas Eckardt thomas.ecka...@thockar.com wrote:
Colin - I find it hard to believe. You brought home the bacon. :):):)

NEVER EVER use the web listerner 5 to monitor assp - this can lead in to 
unexpected config changes or config reloads - in worth case you can lose parts 
or the complete configuration.



These are very BASIC IT rules - and they also applies to assp:

Don't login to assp as 'root'. Use 'root' only, if you need to access 
restricted configuration parameters.
NEVER forget to use the 'logout' button in the GUI - especially NOT if root is 
logged on!


2015-03-09 09:38:34 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 106 
records
2015-03-09 09:43:33 [Main_Thread] Adminupdate: [root 192.168.11.13] 
file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains'

2015-03-09 21:37:10 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 104 
records
2015-03-09 21:42:11 [Main_Thread] Adminupdate: [root 192.168.11.13] 
file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains'

exactly 5 minutes difference - Colin, can you remember about this 5 minutes - 
is it an accidental circumstance, that the monitor to port 5 is running 
every 5 minutes ??
But - it is NOT a  accidental

Re: [Assp-test] Localdomains stopping working

[Colin Waring]

 I'm likely to have to switch to a different product which I 
really don't want to do.

All the best,
Colin Waring.

-Original Message-
From: Colin [mailto:colin.war...@gmail.com] 
Sent: 03 March 2015 17:44
To: ASSP development mailing list
Subject: [Assp-test] Localdomains stopping working

Howdy,

We've had this a couple of times in the last week or so:

2015-03-03 15:17:15 [Main_Thread] Saving config
2015-03-03 15:17:15 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged
2015-03-03 15:17:15 [Main_Thread] Adminupdate: [root ] file 
'/usr/local/assp/files/localdomains.txt' for config 'localDomains' was changed
2015-03-03 15:17:15 [Main_Thread] Option list file: 
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with
104 records

On the face of it, looks fine as it loads all the entries but after this point 
ASSP acts as though the file is empty. All inbound mail gets bounced with:

[SMTP Error] 530 Relaying not allowed (enable smtp authentication on your email 
client)

I've verified with the MTA that this isn't an MTA error, ASSP is generating 
this before passing the connection on to it.

The localdomains.txt file is updated automatically by a script so that could be 
the trigger for the reload.

Any thoughts?


--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored 
by Intel and developed in partnership with Slashdot Media, is your hub for all 
things parallel software development, from weekly thought leadership blogs to 
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test



--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.4.4 build 15067

[Colin Waring]

Thanks for the explanation Thomas,

Most of the changes I've been making are aimed at redundancy over performance. 
For example I intended to build a MySQL cluster and put it behind a load 
balancer so that we can handle the DB server going offline for maintenance etc.

I do have one issue that I've never been sure about whether it is performance 
related. Quite regularly, ASSP will accept connections and hold them for 
anywhere from a few seconds up to 10-20 seconds and then carry on. It is 
noticeable enough that when I'm using the web admin to change between a few 
settings I'll quite often see it. Most of the time it doesn't cause any 
problems as it always finishes processing after the delay.

Every now and then though it doesn't come back. ASSP won't respond to any 
shutdown commands so I have to kill the process, remove the pid file and start 
it back up manually. My monitoring scripts only kick in if they can't connect 
to port 25/5.

I suspect this won't help as that doesn't look to be performance related and 

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 09 March 2015 05:38
To: ASSP development mailing list
Subject: Re: [Assp-test] fixes in assp 2.4.4 build 15067

Colin,

If I understand more of how experimental this is and the next step is 
to HA the database

At this time the code is very experimental and very special. It aims to fix 
SMTP performance problems for an ISP, which holds around 20.000 domains.
The concept of the central RDB (for HMM and Bayesian) backend is not fast 
enough to process several hundred thousands or million mails a day.
If 100.000 mails have to be processed with HMM and/or Bayesian in a day, this 
will lead in to 6.000.000 - 60.000.000 SQL queries a day (only for HMM).
What DB engine (cluster) is able to do this? And this is only the average 
calculation - what about the peaks?
The code is currently specialized for the environment of this single ISP and is 
not generic enough to go public. 
There are currently no changes made to enhance the implementation of other 
features, like blockreporting or anything else.

Thomas





Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  08.03.2015 15:18
Betreff:Re: [Assp-test] fixes in assp 2.4.4 build 15067



Hi Thomas,

I'd be very interested to know more details on the ultimate aim with the ISP 
option. I support the idea of subscription for the higher end as it will help 
create funding for you past donations.

Is the aim of the addition to add support for extended scalability or do you 
have ideas for the future to make additional features available? If you 
remember we exchanged emails a while back about some of the features that I 
could see benefiting a larger setup and we are looking into how to implement 
things at the moment.

I've already implemented clustered file systems and the next step is to HA the 
database. The biggest concern for me in scaling up is the block reports being 
generated on each server individually.

If I understand more of how experimental this is and what could go wrong then I 
may be able to help with testibg.

All the best,
Colin Waring

On 8 Mar 2015 12:39, Thomas Eckardt thomas.ecka...@thockar.com wrote:
Hi all,

fixed in assp 2.4.4 build 15067:

- on some windows systems 'Win32::Unicode' was detected as unavailable, even it 
was correctly installed

- the alpha index was not working in build 15059

- HMM was not working, if 'spamdb' was set to a plain file, placed in a 
subfolder like: db/spamdb


added:

- This build contains experimental code to setup assp in very large ISP 
environments, with a very high workload
  caused by HMM, Bayesian and DNS.
  Such a setup requires an enormous and expensive amount of hardware resources, 
a very high knowledge in
  system design and OS scripting.
  minimum requirements:
  - assp: 64Bit OS, all SSD, 16GB RAM, 8 CPU cores, 64Bit Perl (multiple 
larger systems expected)
  - external high available enterprise database server
  - high available and very fast DNS-servers

  This ISP setup option is subject to become a payed licensed feature.


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known 
virus in this email!
***

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored 
by Intel and developed in partnership with Slashdot Media, is your hub for all 
things parallel software development, from weekly thought leadership blogs to 
news, videos

Re: [Assp-test] fixes in assp 2.4.4 build 15067

[Colin Waring]

Hi Thomas,

I'd be very interested to know more details on the ultimate aim with the ISP 
option. I support the idea of subscription for the higher end as it will help 
create funding for you past donations.

Is the aim of the addition to add support for extended scalability or do you 
have ideas for the future to make additional features available? If you 
remember we exchanged emails a while back about some of the features that I 
could see benefiting a larger setup and we are looking into how to implement 
things at the moment.

I've already implemented clustered file systems and the next step is to HA the 
database. The biggest concern for me in scaling up is the block reports being 
generated on each server individually.

If I understand more of how experimental this is and what could go wrong then I 
may be able to help with testibg.

All the best,
Colin Waring

On 8 Mar 2015 12:39, Thomas Eckardt thomas.ecka...@thockar.com wrote:
Hi all,

fixed in assp 2.4.4 build 15067:

- on some windows systems 'Win32::Unicode' was detected as unavailable,
even it was correctly installed

- the alpha index was not working in build 15059

- HMM was not working, if 'spamdb' was set to a plain file, placed in a
subfolder like: db/spamdb


added:

- This build contains experimental code to setup assp in very large ISP
environments, with a very high workload
  caused by HMM, Bayesian and DNS.
  Such a setup requires an enormous and expensive amount of hardware
resources, a very high knowledge in
  system design and OS scripting.
  minimum requirements:
  - assp: 64Bit OS, all SSD, 16GB RAM, 8 CPU cores, 64Bit Perl
(multiple larger systems expected)
  - external high available enterprise database server
  - high available and very fast DNS-servers

  This ISP setup option is subject to become a payed licensed feature.


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
***

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Unsupported bDat

[Colin Waring]

Hi Thomas,

If you have a test version please feel free to send it over. I'm starting to 
get a lot of complaints on this one - I thought it was just one sender at first 
but it looks like we're going to get a lot of grief over this one!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 30 June 2014 16:03
To: ASSP development mailing list
Subject: Re: [Assp-test] Unsupported bDat

Had the same trouble - but I think I found the BUG - just testing. The Problem 
is only related to whitelisted and or noprocessing mails.

Thomas




Von:Daniel Miller dmil...@amfes.com
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  30.06.2014 16:23
Betreff:[Assp-test] Unsupported bDat



Having trouble sending an attachment - never seen this error before:

Jun-30-1407:14:3837678-11881[Worker_1][TLS-in][TLS-out]*98.167.72.49**dmil...@amfes.com*info:foundmessagesizeannouncement:3.26MByte
Jun-30-1407:14:3837678-11881[Worker_1][TLS-in][TLS-out]*98.167.72.49**dmil...@amfes.com*messageproxiedwithoutprocessing-messagesize(3416095)isabove50(npSizeOut
 

http://bubba.amfes.lan:5/#npSizeOut).
Jun-30-1407:14:4237681-13661[Worker_1][TLS-in][TLS-out][unsupported_bDAt]*98.167.72.49*bDAtnotallowed
 



--
Daniel

--
Open source business process management suite built on Java and Eclipse Quickly 
connect people, data, and systems into organized workflows Winner of BOSSIE, 
CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft 
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known 
virus in this email!
***




--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Unsupported bDat

[Colin Waring]

Sorry I though you said you had fixed it and were just testing. Can we 
downgrade to an earlier version to get away from this bug? I have one client 
that is affected massively by this for some reason.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 30 June 2014 17:07
To: ASSP development mailing list
Subject: Re: [Assp-test] Unsupported bDat

If you have a test version please feel free to send it over.

This makes currently no sense - there is a BUG that I need to find and to fix. 
If I think I have fixed it, I'll release the code.

Thomas





Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  30.06.2014 17:46
Betreff:Re: [Assp-test] Unsupported bDat



Hi Thomas,

If you have a test version please feel free to send it over. I'm starting to 
get a lot of complaints on this one - I thought it was just one sender at first 
but it looks like we're going to get a lot of grief over this one!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 30 June 2014 16:03
To: ASSP development mailing list
Subject: Re: [Assp-test] Unsupported bDat

Had the same trouble - but I think I found the BUG - just testing. The Problem 
is only related to whitelisted and or noprocessing mails.

Thomas




Von:Daniel Miller dmil...@amfes.com
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  30.06.2014 16:23
Betreff:[Assp-test] Unsupported bDat



Having trouble sending an attachment - never seen this error before:

Jun-30-1407:14:3837678-11881[Worker_1][TLS-in][TLS-out]*98.167.72.49**dmil...@amfes.com*info:foundmessagesizeannouncement:3.26MByte
Jun-30-1407:14:3837678-11881[Worker_1][TLS-in][TLS-out]*98.167.72.49**dmil...@amfes.com*messageproxiedwithoutprocessing-messagesize(3416095)isabove50(npSizeOut
 


http://bubba.amfes.lan:5/#npSizeOut).
Jun-30-1407:14:4237681-13661[Worker_1][TLS-in][TLS-out][unsupported_bDAt]*98.167.72.49*bDAtnotallowed
 




--
Daniel

--
Open source business process management suite built on Java and Eclipse Quickly 
connect people, data, and systems into organized workflows Winner of BOSSIE, 
CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft 
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 


individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
Open source business process management suite built on Java and Eclipse
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Open source business process management suite built on Java and Eclipse
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Open source business process management suite

Re: [Assp-test] fixes in assp 2.4.2 build 14181

[Colin Waring]

Thank you so much for getting the fix out quickly on this one. 

I'm going to forward an email I sent earlier this month - I'm hoping that your 
fix may have resolved that issue too but I wanted to make sure now that I 
appear to be able to send to the SF lists again!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 30 June 2014 19:33
To: ASSP List
Subject: [Assp-test] fixes in assp 2.4.2 build 14181

Hi all,

fixed in assp 2.4.2 build 14181:

- mails archived by ASSP_ARC.pm had an additionaly trailing '.'

- for some mails an exception 'Odd number of elements in hash assignment at
  ...Perl/site/lib/Mail/SPF/Server.pm line 210.' was thrown

- with an installed version 1.994 of IO::Socket::SSL , the SMTP-SSL
listener(s) was only working in plain text

- message scoring was not working for local and outgoing mails, read the 
'changed' section

changed:

- message scoring was switched off in the code for local and outgoing mails, it 
is now enabled and configurable -
  read the 'added' section

- on very slow IP connections to the Web-Interface, it was possible that the 
transfered data were incomplete
  because of a hardcoded content-transfer-timeout of 30 seconds
  This timeout value is now controlled with the hidden configuration variable 
'WebTrafficTimeout', which has
  a default value of 60 seconds
 
added:

'DoLocalPenaltyMessage','Message Scoring Mode for Local and Outgoing Mails', 
'If this feature is selected, the total score for all checks during a local or 
outgoing message is used to  determine if the email is Spam. If the combined 
score is greater than the Local Low MessageLimit
 (LocalPenaltyMessageLow) and less than or equal the Local High MessageLimit 
(LocalPenaltyMessageLimit)  the message will not be blocked but tagged. If the 
combined score is greater than the Local High MessageLimit  
(LocalPenaltyMessageLimit), the message will be blocked.


'LocalPenaltyMessageLow','Low MessageLimit for Local and Outgoing Mails'
'MessageMode will not block local and outgoing messages whose score exceeds 
this threshold during the message but  will tag them.  For example: 40'


'LocalPenaltyMessageLimit','High MessageLimit for Local and Outgoing Mails'
'MessageMode will block local and outgoing messages whose score exceeds this 
threshold during the message.
 For example: 50'

Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known 
virus in this email!
***




--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Crashes today

[Colin Waring]

Anyone else seeing ASSP crashing a lot today?

 

Each time is preceeded by emails from quotes@somethingorother like this:

 

2014-06-02 12:10:00 m1-07354-06606 [Worker_5] [SSL-out] 109.228.10.136
quo...@professional-crm.co.uk to: recipi...@domain.tld info: start damping
on closing connection (12)

2014-06-02 12:11:30 m1-07490-12287 [Worker_2] [TLS-out] 109.228.22.120
quo...@letscompare-seo.co.uk [SMTP Reply] 250 OK

2014-06-02 12:11:33 m1-07490-12287 [Worker_2] [TLS-out] 109.228.22.120
quo...@letscompare-seo.co.uk to: recipi...@domain.tld [SMTP Reply] 250
Accepted

2014-06-02 12:11:33 m1-07490-12287 [Worker_2] [TLS-out] 109.228.22.120
quo...@letscompare-seo.co.uk to: recipi...@domain.tld recipient delayed:
recipi...@domain.tld

2014-06-02 12:11:33 m1-07490-12287 [Worker_2] [TLS-out] 109.228.22.120
quo...@letscompare-seo.co.uk to: recipi...@domain.tld [SMTP Status] 451
4.7.1 Greylisting, Please try again after 1 minute

2014-06-02 12:11:56 m1-07516-07642 [Worker_2] [TLS-out] 109.228.30.116
quo...@compare-frankingmachines.co.uk [SMTP Reply] 250 OK

2014-06-02 12:12:05 m1-07516-07642 [Worker_2] [TLS-out] 109.228.30.116
quo...@compare-frankingmachines.co.uk to: recipi...@domain.tld [SMTP
Reply] 250 Accepted

2014-06-02 12:12:05 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk [SMTP Reply] 250 OK

2014-06-02 12:12:08 m1-07516-07642 [Worker_2] [TLS-out] 109.228.30.116
quo...@compare-frankingmachines.co.uk to: recipi...@domain.tld recipient
delayed: recipi...@domain.tld

2014-06-02 12:12:08 m1-07516-07642 [Worker_2] [TLS-out] 109.228.30.116
quo...@compare-frankingmachines.co.uk to: recipi...@domain.tld [SMTP
Status] 451 4.7.1 Greylisting, Please try again after 1 minute

2014-06-02 12:12:15 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk to: recipi...@domain.tld [SMTP Reply]
250 Accepted

2014-06-02 12:12:21 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk to: recipi...@domain.tld [SMTP Reply]
354 Enter message, ending with . on a line by itself

2014-06-02 12:12:25 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk to: recipi...@domain.tld
DomainKey-Signature found

2014-06-02 12:12:25 m1-07544-12617 [Worker_1] [TLS-out] 109.228.4.25
quo...@compare-webdesign.co.uk [SMTP Reply] 250 OK

2014-06-02 12:12:26 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk to: recipi...@domain.tld Message-Score:
added 25 for DNSBL: neutral, 109.228.2.102 listed in
bb.barracudacentral.org, total score for this message is now 25

 

I'm snowed under as we're moving out entire infrastructure to a new platform
at the moment, has anyone else crafted any rules to stop these? Source
addresses are similar but I'm not sure about blocking an entire /16 

--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their 
applications. Written by three acclaimed leaders in the field, 
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Email::MIME problem

[Colin Waring]

Strangely, downgrading to 1.911 and upgrading to the latest ASSP did not
work for me. I have already had some overnight reports of corrupted mail.

I am just upgrading to 14144 now so hopefully that will resolve the problem.
With 14144 should we use the latest Email::MIME?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 23 May 2014 16:39
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

clean the PTRCache after upgrade

Thomas




Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 17:27
Betreff:Re: [Assp-test] Email::MIME problem



I've gone through my servers and replaces Email::MIME 1.926 with 1.911.

I'll get the latest version of ASSP running shortly. PTR cache is turned off
already, presumably I noticed problems with the cache at some point.

I have ASSP_AFC enabled but charset conversation not enabled.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 10:23
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

Colin,

so you may wish to check the code introduced after 14097

I'm doing this for several days now.

With Email::MIME 1.926 I (and others) get wired behavior - even if I use the
same mail multiple times.

There are two places where assp uses Email::MIME to modify an  email, the
charset conversion and the ASSP_AFC plugin (in case of spam/bad attachment
only).
Both could be disabled having the same result.

I have an idea what could happen - but I hope I'm wrong. I'll have to look
in to Perl and Carp source code - eval and exception handling.

workaround - use Email::MIME 1.911

Colin , btw. 14097 has a big issue with the PTR resolving/caching - switch
this check off!

Thomas

 





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 09:56
Betreff:Re: [Assp-test] Email::MIME problem



Hi Thomas,

This sounds like exactly the issue I reported on the 6th.

I found that the issue was not present in version 14097 and earlier so you
may wish to check the code introduced after 14097.

I've been busy with other things so haven't been able to do any more
troubleshooting on it and am still running 14097 myself without the issue.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 06:52
To: development mailing list ASSP
Subject: [Assp-test] Email::MIME problem

Hi all,

I got a report from Philipp about the problem were mails are delivered
incomplete/destroyed.

Philipp wrote:

...
I recently had the problem that some mails were forwarded incorrectly by
ASSP
2.4.2 build 14141 (and 14130, too).

That is, the mail was received and saved to file correctly, but when it was
forwarded to the destination MTA, it started in the midst of the content,
removed ALL header lines and added the ASSP-Headers at the end of the mail.
It was the same for noProcessing-mails, too, thus I excluded problems by
spam processing.

After long and painful debugging, I concluded that the problem must be the 



Header-Parsing of some multipart mails, but not all of them and I still
don't know which of those exactly, sorry!

Since I have another server with more or less the same configuration (at
least no differences that would influence noProcessing-mails) and there is
no sign of this problem, I concentrated on differences between those two
servers.

The only real differences were on some perl module version numbers. While
the faulty server had Email::MIME in version 1.926, the other one in version
1.911. Thus I downgraded this module on the faulty server and it seems as it
solved the problem.


I've released Email::MIME 1.911 as ZIP in the /lib folder on SF and SF-CVS.
To install it, copy the extracted ZIP in to the assp/lib folder and restart
assp. 
I currently don't know why and where the problem in assp is. It will take a
while to analyze the problem.

Thomas



DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 



individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***





--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform 
available
Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu

Re: [Assp-test] Email::MIME problem

[Colin Waring]

Thanks Thomas, muchly appreciated.

I have put a few tests through with 14144 and the latest Email::MIME and
they seem fine.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 24 May 2014 12:13
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

I am just upgrading to 14144 now so hopefully that will resolve the 
problem.

It will !

With 14144 should we use the latest Email::MIME?

Yes, the latest.

IMHO Email::MIME was not directly involved - only the bad 'bondary' was 
possibly - but is no longer a problem with any version of this module

The reason was a  'not common' but nice coding (by me) some years ago, 
used to call Perl's sv_grow for large mails (eg. noprocessing by size). 
This was no longer working with the new permanent opened UDP-DNS sockets - 
very strange and very hard to find - even the Perl souce code does not 
clearly explain what happens.
The assp install script also installs the Convert::Scalar module (since 
years), which consumes more memory, but is commonly used to do the 
sv_grow. If installed, it is used now by assp for this function. 

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  24.05.2014 12:15
Betreff:Re: [Assp-test] Email::MIME problem



Strangely, downgrading to 1.911 and upgrading to the latest ASSP did not
work for me. I have already had some overnight reports of corrupted mail.

I am just upgrading to 14144 now so hopefully that will resolve the 
problem.
With 14144 should we use the latest Email::MIME?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 23 May 2014 16:39
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

clean the PTRCache after upgrade

Thomas




Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 17:27
Betreff:Re: [Assp-test] Email::MIME problem



I've gone through my servers and replaces Email::MIME 1.926 with 1.911.

I'll get the latest version of ASSP running shortly. PTR cache is turned 
off
already, presumably I noticed problems with the cache at some point.

I have ASSP_AFC enabled but charset conversation not enabled.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 10:23
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

Colin,

so you may wish to check the code introduced after 14097

I'm doing this for several days now.

With Email::MIME 1.926 I (and others) get wired behavior - even if I use 
the
same mail multiple times.

There are two places where assp uses Email::MIME to modify an  email, the
charset conversion and the ASSP_AFC plugin (in case of spam/bad attachment
only).
Both could be disabled having the same result.

I have an idea what could happen - but I hope I'm wrong. I'll have to look
in to Perl and Carp source code - eval and exception handling.

workaround - use Email::MIME 1.911

Colin , btw. 14097 has a big issue with the PTR resolving/caching - switch
this check off!

Thomas

 





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 09:56
Betreff:Re: [Assp-test] Email::MIME problem



Hi Thomas,

This sounds like exactly the issue I reported on the 6th.

I found that the issue was not present in version 14097 and earlier so you
may wish to check the code introduced after 14097.

I've been busy with other things so haven't been able to do any more
troubleshooting on it and am still running 14097 myself without the issue.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 06:52
To: development mailing list ASSP
Subject: [Assp-test] Email::MIME problem

Hi all,

I got a report from Philipp about the problem were mails are delivered
incomplete/destroyed.

Philipp wrote:

...
I recently had the problem that some mails were forwarded incorrectly by
ASSP
2.4.2 build 14141 (and 14130, too).

That is, the mail was received and saved to file correctly, but when it 
was
forwarded to the destination MTA, it started in the midst of the content,
removed ALL header lines and added the ASSP-Headers at the end of the 
mail.
It was the same for noProcessing-mails, too, thus I excluded problems by
spam processing.

After long and painful debugging, I concluded that the problem must be the 




Header-Parsing of some multipart mails, but not all of them and I still
don't know which of those exactly, sorry!

Since I have another server with more or less the same configuration (at
least no differences that would influence noProcessing-mails) and there is
no sign

Re: [Assp-test] Email::MIME problem

[Colin Waring]

I've gone through my servers and replaces Email::MIME 1.926 with 1.911.

I'll get the latest version of ASSP running shortly. PTR cache is turned off
already, presumably I noticed problems with the cache at some point.

I have ASSP_AFC enabled but charset conversation not enabled.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 23 May 2014 10:23
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

Colin,

so you may wish to check the code introduced after 14097

I'm doing this for several days now.

With Email::MIME 1.926 I (and others) get wired behavior - even if I use the
same mail multiple times.

There are two places where assp uses Email::MIME to modify an  email, the
charset conversion and the ASSP_AFC plugin (in case of spam/bad attachment
only).
Both could be disabled having the same result.

I have an idea what could happen - but I hope I'm wrong. I'll have to look
in to Perl and Carp source code - eval and exception handling.

workaround - use Email::MIME 1.911

Colin , btw. 14097 has a big issue with the PTR resolving/caching - switch
this check off!

Thomas

 





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 09:56
Betreff:Re: [Assp-test] Email::MIME problem



Hi Thomas,

This sounds like exactly the issue I reported on the 6th.

I found that the issue was not present in version 14097 and earlier so you
may wish to check the code introduced after 14097.

I've been busy with other things so haven't been able to do any more
troubleshooting on it and am still running 14097 myself without the issue.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 06:52
To: development mailing list ASSP
Subject: [Assp-test] Email::MIME problem

Hi all,

I got a report from Philipp about the problem were mails are delivered
incomplete/destroyed.

Philipp wrote:

...
I recently had the problem that some mails were forwarded incorrectly by
ASSP
2.4.2 build 14141 (and 14130, too).

That is, the mail was received and saved to file correctly, but when it was
forwarded to the destination MTA, it started in the midst of the content,
removed ALL header lines and added the ASSP-Headers at the end of the mail.
It was the same for noProcessing-mails, too, thus I excluded problems by
spam processing.

After long and painful debugging, I concluded that the problem must be the 


Header-Parsing of some multipart mails, but not all of them and I still
don't know which of those exactly, sorry!

Since I have another server with more or less the same configuration (at
least no differences that would influence noProcessing-mails) and there is
no sign of this problem, I concentrated on differences between those two
servers.

The only real differences were on some perl module version numbers. While
the faulty server had Email::MIME in version 1.926, the other one in version
1.911. Thus I downgraded this module on the faulty server and it seems as it
solved the problem.


I've released Email::MIME 1.911 as ZIP in the /lib folder on SF and SF-CVS.
To install it, copy the extracted ZIP in to the assp/lib folder and restart
assp. 
I currently don't know why and where the problem in assp is. It will take a
while to analyze the problem.

Thomas



DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 


individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***





--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform 
available
Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu/SauceLabs
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run

Re: [Assp-test] rebuildspamdb error ASSP version 2.4.2(14132)

[Colin Waring]

You need to make sure that all the files in your lib and plugins folders are up 
to date. Wordstem will be one of them and no doubt others will be out of date.

All the best,
Colin Waring

On 17 May 2014 18:02, Daniel K. Du Vall dduv...@1peter4-10.org wrote:

 I have run rebuildspamdb as suggested but still getting this error. Have I 
 missed updating something else somewhere or maybe something else? 

 May-17-14 11:54:50 [init] Spamdb has 103,090 records 
 May-17-14 11:54:50 [init] Warning: the current Spamdb is possibly 
 incompatible to this version of ASSP. Please run a rebuildspamdb. current: 
 2_14094_5.014002_UAX#29_WordStem1.23 - required: 
 2_14094_5.014002_UAX#29_WordStem1.27 
 May-17-14 11:54:50 [init] Start analyze whitelist 
 May-17-14 11:54:50 [init] Whitelist has 4,049 records 
 May-17-14 11:54:50 [init] The Hidden-Markov-Model-DB has 887,742 records. 
 May-17-14 11:54:50 [init] Warning: the current HMMdb is possibly incompatible 
 to this version of ASSP. Please run a rebuildspamdb. current: 
 2_14094_5.014002_UAX#29_WordStem1.23 - required: 
 2_14094_5.014002_UAX#29_WordStem1.27 
 May-17-14 11:54:50 [init] Info: saving Stats in file asspstats.sav 
 May-17-14 11:54:50 [init] Info: saving ScoreStats in file asspscorestats.sav 


 Daniel Du Vall 

 --
  
 Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE 
 Instantly run your Selenium tests across 300+ browser/OS combos. 
 Get unparalleled scalability from the best Selenium testing platform 
 available 
 Simple to use. Nothing to install. Get started now for free. 
 http://p.sf.net/sfu/SauceLabs 
 ___ 
 Assp-test mailing list 
 Assp-test@lists.sourceforge.net 
 https://lists.sourceforge.net/lists/listinfo/assp-test 
--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu/SauceLabs
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] No info in Virus Report

[Colin Waring]

Any reason you're using Filescan instead of ticking UseAVClamd and setting
that up?

For filescan you'd need to set up the regex for good/bad/responds but using
clamd direct should be far more effective.

-Original Message-
From: Christian Leicht [mailto:use...@schani.com] 
Sent: 14 May 2014 12:56
To: assp-test@lists.sourceforge.net
Subject: [Assp-test] No info in Virus Report

Hello all,

i use in ASSP 2.4.2(14097) ClamAV via Command Line to check the Mails.
If someone get a Virus Mail, ASSP block an send the virusreport.txt to
receiver an webmaster.

But in the end of the Report are no information about the Mail sender or
virus  .
What?s wrong?

Here is my Comand Line (FileScanCMD)

/usr/bin/clamscan -v -d /var/lib/clamav/ --stdout --tempdir=/tmp
--copy=/usr/share/assp/virusemails --bytecode=no --bytecode-unsigned=no
--detect-pua=yes --phishing-sigs=yes --phishing-scan-urls=yes
--heuristic-scan-precedence=no --algorithmic-detection=yes --scan-pe=yes
--scan-elf=yes --scan-pdf=yes --scan-html=yes --scan-archive=yes
--detect-broken=yes FILENAME


In the virusreport.txt

Virus entdeckt: 


BITTE KURZ DURCHLESEN

Eine an Sie adressierte E-Mail wurde von unserem Virenscanner als
gefaehrlich eingestuft.
Weitere Details zur betroffenen Email finden Sie am Ende dieser Email.

Um weiteren Schaden fuer Sie auszuschliessen, wurde die betroffene E-Mail
nicht an Sie weitergeleitet sondern bereits auf dem Server in Quarantaene
verschoben.

Sollten Sie eine dringende Email erwarten, setzen Sie sich Bitte mit
  in Verbindung:
x
x
x


Falls Ihnen der Absender persoenlich bekannt ist, sollten Sie sich mit ihm
in Verbindung setzen und ihn darauf hinweisen, dass sein PC wahrscheinlich
von einem Virus befallen ist.

WARNING
Our anti-spam and anti-virus system has detected a virus or phishing attack
within an email sent to you.  Should you feel that this was in error, or
have questions please feel free to contact support and supply them with a
copy of this email.


###
Details zur Email:
The following are details from the email:
.
.




Thanks for help

Christian


use...@schani.com
---
Web: http://www.leicht.info
Mail: use...@schani.com 


--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform
available Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu/SauceLabs
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu/SauceLabs
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Serious issue, messages being passed to MTA incomplete

[Colin Waring]

Hi All,

 

I've started getting quite a few reports this morning that emails are coming
through unreadable.

 

This does not affect all messages, it seems to affect specific senders but
there are a number of senders affected and a number of recipients affected.

 

The messages affected are received fine and ASSP stores them complete in the
mail store folder.

 

When they are passed to the MTA, all of the headers are missing. Exim adds
its own information from what it is passed (sender, recipient and then adds
its own received header). 

 

The content of the message seems to be encoded content that I cannot locate
anywhere in any file saved by ASSP. My guess is that the whole message is
somehow getting encoded, including the headers and sent to the MTA as the
body of the message.

 

I've copied one example in below, though I've snipped off most of the actual
message for both the ASSP file and the Exim file. I can send the actual
unmodified versions privately if necessary. I have downgraded back to 14097
as the problem was not evident before upgrading again at the weekend.

 

Example mail as stored by ASSP:

 

X-Assp-Version: 2.4.2(14123) on mail.smtphost.co.uk

X-Assp-Server-TLS: yes

X-Assp-Delay: not delayed (1.1.1.1 in whitebox ); 6 May 2014

09:41:19 +0100

X-Assp-Whitelisted: Yes (whitelisted - found valid Message-ID

signature)

X-Assp-ID: mail.smtphost.co.uk m1-65679-08364

X-Assp-Session: A3FC150C

Received: from smtp5-out2.enta.net ([1.1.1.1] helo=smtp5-out2.enta.net)

by mail.smtphost.co.uk with SMTP (2.4.2); 6 May 2014 09:41:19 +0100

Received: from remote.host.name (2.2.reverse.dns [2.2.2.2])

(using TLSv1 with cipher AES128-SHA (128/128 bits))

(No client certificate requested)

by smtp5.enta.net (Postfix) with ESMTPS id 9EC182FEC

for recipi...@domain.tld; Tue,  6 May 2014 09:41:13 +0100 (BST)

Received: from internal.server.local ([fe80::1c65:2f34:31e3:8b5e]) by

internal.server.local ([fe80::1c65:2f34:31e3:8b5e%12]) with mapi; Tue, 6 May

2014 09:41:12 +0100

From: Sender Name sen...@domain.tld

To: 'Recipient Name' recipi...@domain.tld

Date: Tue, 6 May 2014 09:41:11 +0100

Subject: RE:

Thread-Index: AQHPaQYupn6DVC7MxUCc/owC0R/mqJszOmYggAAArwCAAABvwP///9UQ

Message-ID:
1880BEC6C4C252419D55D1057416134B969916E81D@internal.server.local

References: 4d1a14ad-d493-4976-80b6-5f87f0552a03@recipient.server.local

181DA9A7F6A0EB448A8426590E9E7D9629713D54@recipient.server.local

1880BEC6C4C252419D55D1057416134B969916E81C@internal.server.local

sig.3203bdec83.181da9a7f6a0eb448a8426590e9e7d9629713...@recipient.server.lo
cal

In-Reply-To:
sig.3203bdec83.181da9a7f6a0eb448a8426590e9e7d9629713...@recipient.server.lo
cal

Accept-Language: en-US, en-GB

Content-Language: en-US

X-MS-Has-Attach: yes

X-MS-TNEF-Correlator:

acceptlanguage: en-US, en-GB

Content-Type: multipart/mixed;

 
boundary=_002_1880BEC6C4C252419D55D1057416134B969916E81DKSSDC1kalcres_

MIME-Version: 1.0

 

--_002_1880BEC6C4C252419D55D1057416134B969916E81DKSSDC1kalcres_

Content-Type: text/plain; charset=iso-8859-1

Content-Transfer-Encoding: quoted-printable

 

Hmm weird

 

Does it work now?

 

#Message Continues

 

 

Logs from ASSP:

 

2014-05-06 10:39:53 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2  [SMTP
Reply] 250 OK

2014-05-06 10:39:53 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld [SMTP Reply] 250 Accepted

2014-05-06 10:39:54 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld [SMTP Reply] 250 Accepted

2014-05-06 10:39:54 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld [SMTP Reply] 354 Enter message,
ending with . on a line by itself

2014-05-06 10:39:54 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld Message-Score: added -15
(pbwValencePB) for In Penalty White Box, total score for this message is now
-15

2014-05-06 10:39:54 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld [scoring] SPF: none ip=2.2.2.2
mailfrom=sen...@domain.tld helo=smtp4-out1.enta.net

2014-05-06 10:39:54 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld Whitelisted sender address:
sen...@domain.tld for recipient recipi...@domain.tld

2014-05-06 10:39:54 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld Whitelisted sender address:
sen...@domain.tld for recipient recipie...@domain.tld

2014-05-06 10:40:02 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld [Plugin] calling plugin
ASSP_AFC

2014-05-06 10:40:02 m1-69193-10044 [Worker_3] [TLS-out] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld message proxied without
processing (no bad attachments)

2014-05-06 10:40:02 m1-69193-10044 [Worker_3] [TLS-out] [MessageOK] 2.2.2.2
sen...@domain.tld to: recipi...@domain.tld message ok [Subject] -

[Assp-test] Build 14121 high CPU / stuck threads

[Colin Waring]

Primary mailserver just crashed out twice and restarted ASSP:

 

2014-05-02 09:51:10 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_6 in 0.851 seconds - got (ok)

2014-05-02 09:51:10 [Worker_6] 1.1.1.1 IP 1.1.1.1 matches acceptAllMail -
with 1.1.1.1/32

2014-05-02 09:51:10 [Worker_6] Connected: session:9D3C0E50 1.1.1.1:37891 
195.88.101.110:25  127.0.0.1:125

2014-05-02 09:51:11 [Main_Thread] Info: Main_Thread got connection request

2014-05-02 09:51:11 m1-20662-07189 [Worker_6] 2.2.2.2 spo...@atpi.com to:
recipi...@domain.tld [Plugin] calling plugin ASSP_AFC

2014-05-02 09:51:16 [Worker_6] ClamAv Down

2014-05-02 09:51:16 [Worker_1] Info: Name Server 194.168.4.123:
ResponseTime = 16 ms for sourceforge.net

2014-05-02 09:51:33 [Worker_6] ClamAv Up

2014-05-02 09:51:41 [Main_Thread] Warning: got unexpected signal ALRM in
Main_Thread: package - main, file - sub main::ThreadYield, line - 2!

2014-05-02 09:51:42 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)

2014-05-02 09:51:42 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)

2014-05-02 09:51:42 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)

2014-05-02 09:51:42 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)

2014-05-02 09:51:42 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)

2014-05-02 09:51:42 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)

2014-05-02 09:51:42 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)

 

2014-05-02 10:24:53 [Main_Thread] Warning: got unexpected signal ALRM in
Main_Thread: package - main, file - sub main::ThreadYield, line - 15!

2014-05-02 10:25:17 [Worker_1] Info: Name Server 194.168.4.123:
ResponseTime = 30 ms for sourceforge.net

2014-05-02 10:25:43 [Worker_1] Info: synchronizing all BerkeleyDB hashes
to disk

2014-05-02 10:25:43 [Worker_1] Info: compacting all BerkeleyDB hashes on
disk

2014-05-02 10:25:45 [Main_Thread] Warning: Main_Thread is unable to transfer
connection to any worker - try again!

2014-05-02 10:25:48 [Worker_1] SSLfailedCache: cleaning cache finished:
IP's before=2, deleted=0

2014-05-02 10:25:48 [Worker_1] LocalFrequency: cleaning cache finished:
addresses's before=4, deleted=3

2014-05-02 10:25:48 [Worker_1] SubjectFrequency: cleaning cache
finished: subjects before=44, deleted=28

2014-05-02 10:26:16 [Main_Thread] Info: Loop in Worker_6 was not active for
206 seconds

2014-05-02 10:26:16 [Main_Thread] Info: Worker_6 : last sigoff in main, sub
main::SPFok, 7, main::SPFok_Run, 1, , ,  at 14-2-4 10:2251 1399022571.24198
- 78

2014-05-02 10:26:16 [Main_Thread] Info: Worker_6 : last sigon in main, sub
main::SMTPTraffic, 13, main::sigonTry, 1, , ,  at 14-2-4 10:2250
1399022570.72899 - 13

2014-05-02 10:26:16 [Main_Thread] Info: Worker_6 : last action was : SPF2

2014-05-02 10:26:16 [Main_Thread] Warning: try to terminate
inactive/stucking Worker_6

2014-05-02 10:26:16 [Main_Thread] Warning: Main_Thread is unable to transfer
connection to any worker - try again!

2014-05-02 10:26:16 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)

 

I logged on and noticed that the load was over 7, perl process from top:

 

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND

27003 root  20   0  609m 516m 5568 S  385 17.1  50:41.47 perl

 

I've been watching the worker status page. What happens is one by one the
workers go to PTROK (stuck). Immediately before I have seen the workers say
both SPF2 (stuck) and MXsomething (stuck).

 

I've had to roll back to 14097 unfortunately.

 

All the best,

Colin Waring.

--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu/SauceLabs
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] SSLFailed Cache

[Colin Waring]

Hi there,

 

Back in March last year I had an issue rear its head. The thread concerning
it was titled TLS fail cache. Unfortunately it has come back.

 

This issue is that a particular server seems to occasionally have a hiccup
when negotiating the SSL connection and as a result gets itself blocked from
SSL. This server is configured to only use SSL so that emails are sent out
encrypted leaving it unable to send mail.

 

My request back then was to have settings to control how entries get into
the cache as follows:

 

1)  A setting that defines the number of failures required before an IP
gets added to SSLFailed

2)  The option to exclude certain IPs from being added to the SSLFailed
cache altogether

 

I am aware that I can add the IP to acceptallmail so that the server will
need to fail twice before being blocked but I'd rather not bypass all the
checks for it. I could really do with option 2) being available so that I
don't have to keep going in and removing the entry from the cache.

 

All the best,

Colin Waring.

 

--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.  Get 
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu/SauceLabs
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] hmmdb is empty

[Colin Waring]

Hi there,

I think I've cracked this one. I may have changed some settings relating to bdb 
after one of my earlier questions. The result being that the mysql settings 
were over-ridden and assp was using local files instead of the database.

All the best,
Colin Waring

On 24 Apr 2014 22:29, Colin Waring co...@lanternhosting.co.uk wrote:

 Howdy, 



 I'm getting spam through the system. Something is wrong with HMM. 



 The rebuild went as follows: 



 2014-04-23 23:07:46 Start populating Hidden Markov Model. HMM-check is 
 disabled for this time! 

 2014-04-23 23:07:53 start populating Hidden Markov Model with 2,416,250 
 records! 

 2014-04-23 23:08:46 Finished populating Hidden Markov Model with 2,416,250 
 records! 

 2014-04-23 23:08:46 Finished populating Hidden Markov Model. HMM-check is 
 now enabled again! 



 This has been in the logs all day: 



 2014-04-24 00:00:05 m1-93972-11095 [Worker_1] [TLS-in] [TLS-out] 
 209.85.160.199 sen...@domain.tld to: recipi...@domain.tld HMM is not 
 available - hmmdb is empty 



 There is definitely data in the db and ASSP shows as connected. 



 mysql select count(*) from hmmdb; 

 +--+ 

 | count(*) | 

 +--+ 

 |  1245072 | 

 +--+ 

 1 row in set (12.91 sec) 



 Any ideas what might be causing this or how to debug? 



 All the best, 

 Colin Waring. 

 --
  
 Start Your Social Network Today - Download eXo Platform 
 Build your Enterprise Intranet with eXo Platform Software 
 Java Based Open Source Intranet - Social, Extensible, Cloud Ready 
 Get Started Now And Turn Your Intranet Into A Collaboration Platform 
 http://p.sf.net/sfu/ExoPlatform 
 ___ 
 Assp-test mailing list 
 Assp-test@lists.sourceforge.net 
 https://lists.sourceforge.net/lists/listinfo/assp-test 
--
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] hmmdb is empty

[Colin Waring]

Howdy,

 

I'm getting spam through the system. Something is wrong with HMM.

 

The rebuild went as follows:

 

2014-04-23 23:07:46 Start populating Hidden Markov Model. HMM-check is
disabled for this time!

2014-04-23 23:07:53 start populating Hidden Markov Model with 2,416,250
records!

2014-04-23 23:08:46 Finished populating Hidden Markov Model with 2,416,250
records!

2014-04-23 23:08:46 Finished populating Hidden Markov Model. HMM-check is
now enabled again!

 

This has been in the logs all day:

 

2014-04-24 00:00:05 m1-93972-11095 [Worker_1] [TLS-in] [TLS-out]
209.85.160.199 sen...@domain.tld to: recipi...@domain.tld HMM is not
available - hmmdb is empty

 

There is definitely data in the db and ASSP shows as connected.

 

mysql select count(*) from hmmdb;

+--+

| count(*) |

+--+

|  1245072 |

+--+

1 row in set (12.91 sec)

 

Any ideas what might be causing this or how to debug?

 

All the best,

Colin Waring.

--
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] enhancedOriginIPDetect

[Colin Waring]

Has this feature changed at all lately?

 

I've had to turn it off as emails are getting blocked where the sender
operates an internal network on private subnets. The first header could be
from 192.168.0.x and therefore this hits checks for bogons etc. I've never
had anyone complain about this before and I know people have had these kinds
of emails not blocked.

 

Cheers,

Colin.

 

 

--
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP fails with no error message when tmpDB folder full

[Colin Waring]

Even more on this.

My first attempt to upgrade a machine to 14.04 resulted in broken Perl - not
good.

In the meantime I remembered that Ubuntu has PAE enabled so added the extra
memory to the system. I doubled the tmpDB to 2GB and last night ASSP still
died during the rebuild. It didn't crash the system this time and monitoring
started it right back up again, clearing out the tmpDB as it went.

I'm concerned that for some reason the rebuild process is going runaway on
disk usage in tmpDB as I can't see why it would need double the memory it
has been running on for years.

All the best,
Colin Waring.

-Original Message-
From: Colin Waring [mailto:co...@lanternhosting.co.uk] 
Sent: 19 April 2014 10:41
To: 'ASSP development mailing list'
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full

A bit more on this,

Ubuntu 12.04 LTS died with a kernel error during last night's rebuild.

The system logged the following problems with perl 9 times at 07:39, then
waited 2 minutes before logging it a final time before becoming unresponsive
and required a hard reset. All 10 messages are identical bar the first line
where the number beginning with 2 changes  perlD 2bd3 0
X 1 0x

I am going to start testing upgrading 12.04 LTS to 14.04 LTS to get the
newer perl. Hopefully that will help somewhat.

I was thinking about allocating more space to the ramdisk but then
remembered that you recommended to run on an x86 system. An x86 system can't
make use of more than 3.6GB so if I allocate more memory to the ramdisk then
I am severly limiting that which is available to the system.

Are the recommendations still the same?

Sample log of the kernel lockup:

Apr 18 00:09:39 mail2 kernel: [746880.520185] perlD 2bd3
0 20566  1 0x
Apr 18 00:09:39 mail2 kernel: [746880.520192]  e9ed9e20 00200286 bb11
2bd3 2bd3 c0910fe0 c0a37e00 c0a37e00 Apr 18 00:09:39 mail2 kernel:
[746880.520200]  a3383ba9 0002a70b ebbe4e00
c175d8d0 d21c32c0  001a e9a100c0 Apr 18 00:09:39 mail2 kernel:
[746880.520208]  ffec  e9ed9df8 c06aba9d b7585730 
c175d8d0 c16b8000 Apr 18 00:09:39 mail2 kernel: [746880.520225] Call Trace:
Apr 18 00:09:39 mail2 kernel: [746880.520238]  [c06aba9d] ?
_raw_spin_lock_irqsave+0x2d/0x40
Apr 18 00:09:39 mail2 kernel: [746880.520246]  [c0158e5c] ?
mm_release+0xdc/0xf0
Apr 18 00:09:39 mail2 kernel: [746880.520250]  [c06a9ea5]
schedule+0x35/0x50
Apr 18 00:09:39 mail2 kernel: [746880.520254]  [c015eb7d]
exit_mm+0x6d/0x100
Apr 18 00:09:39 mail2 kernel: [746880.520258]  [c015ed49]
do_exit+0x139/0x3c0
Apr 18 00:09:39 mail2 kernel: [746880.520263]  [c016bd97] ?
recalc_sigpending+0x17/0x40
Apr 18 00:09:39 mail2 kernel: [746880.520267]  [c016bf11] ?
dequeue_signal+0x31/0x190
Apr 18 00:09:39 mail2 kernel: [746880.520271]  [c015f128]
do_group_exit+0x38/0xa0
Apr 18 00:09:39 mail2 kernel: [746880.520276]  [c016e1d6]
get_signal_to_deliver+0x1b6/0x3e0
Apr 18 00:09:39 mail2 kernel: [746880.520283]  [c011197f]
do_signal+0x3f/0xd0
Apr 18 00:09:39 mail2 kernel: [746880.520289]  [c0109809] ?
xen_clocksource_read+0x19/0x20
Apr 18 00:09:39 mail2 kernel: [746880.520293]  [c01848db] ?
ktime_get_ts+0xeb/0x120
Apr 18 00:09:39 mail2 kernel: [746880.520300]  [c0257514] ?
poll_select_set_timeout+0x64/0x80
Apr 18 00:09:39 mail2 kernel: [746880.520304]  [c025829a] ?
sys_poll+0x5a/0xd0
Apr 18 00:09:39 mail2 kernel: [746880.520308]  [c0111c25]
do_notify_resume+0x75/0x90
Apr 18 00:09:39 mail2 kernel: [746880.520313]  [c06abd10]
work_notifysig+0x13/0x1b

-Original Message-
From: Colin Waring [mailto:co...@lanternhosting.co.uk]
Sent: 16 April 2014 09:21
To: 'ASSP development mailing list'
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full

Hi Thomas,

ASSP died again overnight due to tmpDB being full. It looks like
BDBMaxCacheSize doesn't prevent ASSP from dying but allows it to clear the
folder and start up again once it does.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 11 April 2014 08:12
To: ASSP development mailing list
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full

Has anything changed recently that would increase the tmpDB requirements?

Thins could happen - it depends on the config and the count of files and
words in the corpus.

How ever 1GB for tmpDB is also too less for my system.

There are some improvements for BDB cache calculation in the latest
versions. This cache settings are useless for systems that uses a RAM-drive
for tmpDB - I'll cange this.

Thomas





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  11.04.2014 08:51
Betreff:Re: [Assp-test] ASSP fails with no error message when 
tmpDB   folder  full



Hi Thomas,

I didn't have to wait long, it turns out two runs

Re: [Assp-test] ASSP fails with no error message when tmpDB folder full

[Colin Waring]

A bit more on this,

Ubuntu 12.04 LTS died with a kernel error during last night's rebuild.

The system logged the following problems with perl 9 times at 07:39, then
waited 2 minutes before logging it a final time before becoming unresponsive
and required a hard reset. All 10 messages are identical bar the first line
where the number beginning with 2 changes  perlD 2bd3 0
X 1 0x

I am going to start testing upgrading 12.04 LTS to 14.04 LTS to get the
newer perl. Hopefully that will help somewhat.

I was thinking about allocating more space to the ramdisk but then
remembered that you recommended to run on an x86 system. An x86 system can't
make use of more than 3.6GB so if I allocate more memory to the ramdisk then
I am severly limiting that which is available to the system.

Are the recommendations still the same?

Sample log of the kernel lockup:

Apr 18 00:09:39 mail2 kernel: [746880.520185] perlD 2bd3
0 20566  1 0x
Apr 18 00:09:39 mail2 kernel: [746880.520192]  e9ed9e20 00200286 bb11
2bd3 2bd3 c0910fe0 c0a37e00 c0a37e00
Apr 18 00:09:39 mail2 kernel: [746880.520200]  a3383ba9 0002a70b ebbe4e00
c175d8d0 d21c32c0  001a e9a100c0
Apr 18 00:09:39 mail2 kernel: [746880.520208]  ffec  e9ed9df8
c06aba9d b7585730  c175d8d0 c16b8000
Apr 18 00:09:39 mail2 kernel: [746880.520225] Call Trace:
Apr 18 00:09:39 mail2 kernel: [746880.520238]  [c06aba9d] ?
_raw_spin_lock_irqsave+0x2d/0x40
Apr 18 00:09:39 mail2 kernel: [746880.520246]  [c0158e5c] ?
mm_release+0xdc/0xf0
Apr 18 00:09:39 mail2 kernel: [746880.520250]  [c06a9ea5]
schedule+0x35/0x50
Apr 18 00:09:39 mail2 kernel: [746880.520254]  [c015eb7d]
exit_mm+0x6d/0x100
Apr 18 00:09:39 mail2 kernel: [746880.520258]  [c015ed49]
do_exit+0x139/0x3c0
Apr 18 00:09:39 mail2 kernel: [746880.520263]  [c016bd97] ?
recalc_sigpending+0x17/0x40
Apr 18 00:09:39 mail2 kernel: [746880.520267]  [c016bf11] ?
dequeue_signal+0x31/0x190
Apr 18 00:09:39 mail2 kernel: [746880.520271]  [c015f128]
do_group_exit+0x38/0xa0
Apr 18 00:09:39 mail2 kernel: [746880.520276]  [c016e1d6]
get_signal_to_deliver+0x1b6/0x3e0
Apr 18 00:09:39 mail2 kernel: [746880.520283]  [c011197f]
do_signal+0x3f/0xd0
Apr 18 00:09:39 mail2 kernel: [746880.520289]  [c0109809] ?
xen_clocksource_read+0x19/0x20
Apr 18 00:09:39 mail2 kernel: [746880.520293]  [c01848db] ?
ktime_get_ts+0xeb/0x120
Apr 18 00:09:39 mail2 kernel: [746880.520300]  [c0257514] ?
poll_select_set_timeout+0x64/0x80
Apr 18 00:09:39 mail2 kernel: [746880.520304]  [c025829a] ?
sys_poll+0x5a/0xd0
Apr 18 00:09:39 mail2 kernel: [746880.520308]  [c0111c25]
do_notify_resume+0x75/0x90
Apr 18 00:09:39 mail2 kernel: [746880.520313]  [c06abd10]
work_notifysig+0x13/0x1b

-Original Message-
From: Colin Waring [mailto:co...@lanternhosting.co.uk] 
Sent: 16 April 2014 09:21
To: 'ASSP development mailing list'
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full

Hi Thomas,

ASSP died again overnight due to tmpDB being full. It looks like
BDBMaxCacheSize doesn't prevent ASSP from dying but allows it to clear the
folder and start up again once it does.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 11 April 2014 08:12
To: ASSP development mailing list
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full

Has anything changed recently that would increase the tmpDB requirements?

Thins could happen - it depends on the config and the count of files and
words in the corpus.

How ever 1GB for tmpDB is also too less for my system.

There are some improvements for BDB cache calculation in the latest
versions. This cache settings are useless for systems that uses a RAM-drive
for tmpDB - I'll cange this.

Thomas





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  11.04.2014 08:51
Betreff:Re: [Assp-test] ASSP fails with no error message when 
tmpDB   folder  full



Hi Thomas,

I didn't have to wait long, it turns out two runs of rebuildspamdb are
enough to fill a 1GB tempdb folder now.

I have added the entry and ASSP starts back up without coredumping. tmpDB
does remain 100% full though

Has anything changed recently that would increase the tmpDB requirements?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 09 April 2014 09:23
To: ASSP development mailing list
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full

add the following line to 'lib/CorrectASSPcfg.pm'

$main::BDBMaxCacheSize = 0;

and restart assp. Tell me if it works or not.

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  09.04.2014 09:56
Betreff:[Assp-test] ASSP fails with no error message when

Re: [Assp-test] 'Resend' button not working on iOS block reports

[Colin Waring]

What mail app are you using?

Android has a bug where the default mail app will put the whole mailto URL
into the to field and fail to separate the subject and message out to where
they should be. Apparently this also affects the gmail app on IOS and
possibly other mail apps.

The only work around is to copy and paste the link into a web browser where
it will be parsed correctly. This bug is not in ASSP however.

All the best,
Colin Waring.

-Original Message-
From: James Brown [mailto:jlbr...@bordo.com.au] 
Sent: 17 April 2014 08:21
To: ASSP development mailing list
Subject: [Assp-test] 'Resend' button not working on iOS block reports

Neither the left or right 'Resend' buttons work for us when reading the
block report on iOS.

Have had this problem for a long time.

Any solutions?

ASSP version 2.4.2(14097), OS X 10.7.5, Perl 5.18.2

Happened on earlier versions of Perl and ASSP.

(users have had to use webmail as a work-around).

Thanks,

James.

--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP fails with no error message when tmpDB folder full

[Colin Waring]

Hi Thomas,

ASSP died again overnight due to tmpDB being full. It looks like
BDBMaxCacheSize doesn't prevent ASSP from dying but allows it to clear the
folder and start up again once it does.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 11 April 2014 08:12
To: ASSP development mailing list
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full

Has anything changed recently that would increase the tmpDB requirements?

Thins could happen - it depends on the config and the count of files and
words in the corpus.

How ever 1GB for tmpDB is also too less for my system.

There are some improvements for BDB cache calculation in the latest
versions. This cache settings are useless for systems that uses a RAM-drive
for tmpDB - I'll cange this.

Thomas





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  11.04.2014 08:51
Betreff:Re: [Assp-test] ASSP fails with no error message when 
tmpDB   folder  full



Hi Thomas,

I didn't have to wait long, it turns out two runs of rebuildspamdb are
enough to fill a 1GB tempdb folder now.

I have added the entry and ASSP starts back up without coredumping. tmpDB
does remain 100% full though

Has anything changed recently that would increase the tmpDB requirements?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 09 April 2014 09:23
To: ASSP development mailing list
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB 
folder
full

add the following line to 'lib/CorrectASSPcfg.pm'

$main::BDBMaxCacheSize = 0;

and restart assp. Tell me if it works or not.

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net, 
Datum:  09.04.2014 09:56
Betreff:[Assp-test] ASSP fails with no error message when tmpDB 
folder full



Hi Folks,

 

At the weekend one of my mailservers died overnight. I had a quick check
over and saw that it was coredumping without any errors. I decided to 
leave
it till the morning as the other servers could handle things. By morning 
my
monitoring scripts had restarted it.

 

This morning I got up to the same issue, except the problem didn't go away
itself.

 

ASSP would core dump during startup without outputting any messages. If I
enabled debugging, not debug file was created.

 

I had to resort to strace to see that it was getting an error with space 
on
the tmpDB folder which was indeed completely full. There was over a 
gigabyte
of data contained in there, all relating to rebuildspamdb. Some was from
last night's run but some was from two nights prios.

 

I'm presuming that there is some code in the startup that clears up 
leftover
rebuildspamdb data as I emptied the folder but did not remove it. After
starting ASSP the rebuild folder disappeared.

 

I suspect this code needs to be called much earlier in the startup 
process.
I'll include the strace failure incase it gives an idea of where in the
process it needs to be moved to.

 

All the best,

Colin Waring.

 

 

stat64(/usr/local/assp/tmpDB, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=80,
...}) = 0

stat64(/usr/local/assp/tmpDB/_cachecheck, {st_mode=S_IFDIR|0755,
st_size=80, ...}) = 0

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.001, 
{st_mode=S_IFREG|0644,
st_size=0, ...}) = 0

unlink(/usr/local/assp/tmpDB/_cachecheck/__db.001) = 0

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.002, 0x8c07064) = -1 
ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.003, 0x8c07064) = -1 
ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.004, 0x8c07064) = -1 
ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt,
{st_mode=S_IFREG|0644, st_size=98, ...}) = 0

unlink(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt) = 

0

open(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt,
O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 4

_llseek(4, 0, [0], SEEK_END)= 0

ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfaccc38) = -1 ENOTTY
(Inappropriate ioctl for device)

_llseek(4, 0, [0], SEEK_CUR)= 0

fstat64(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

fcntl64(4, F_SETFD, FD_CLOEXEC) = 0

time(NULL)  = 1397029108

stat64(/etc/localtime, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0

time(NULL)  = 1397029108

open(/sys/devices/system/cpu/online, O_RDONLY|O_CLOEXEC) = 5

read(5, 0\n, 8192)= 2

close(5)= 0

write(4, 2014-04-09 08:38:28\nBDB cachesiz..., 50) = 50

fcntl64(4, F_GETFL) = 0x8401 (flags
O_WRONLY|O_APPEND|O_LARGEFILE)

fstat64(4, {st_mode=S_IFREG|0644, st_size

Re: [Assp-test] ASSP fails with no error message when tmpDB folder full

[Colin Waring]

Hi Thomas,

I didn't have to wait long, it turns out two runs of rebuildspamdb are
enough to fill a 1GB tempdb folder now.

I have added the entry and ASSP starts back up without coredumping. tmpDB
does remain 100% full though

Has anything changed recently that would increase the tmpDB requirements?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 09 April 2014 09:23
To: ASSP development mailing list
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full

add the following line to 'lib/CorrectASSPcfg.pm'

$main::BDBMaxCacheSize = 0;

and restart assp. Tell me if it works or not.

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net, 
Datum:  09.04.2014 09:56
Betreff:[Assp-test] ASSP fails with no error message when tmpDB 
folder full



Hi Folks,

 

At the weekend one of my mailservers died overnight. I had a quick check
over and saw that it was coredumping without any errors. I decided to 
leave
it till the morning as the other servers could handle things. By morning 
my
monitoring scripts had restarted it.

 

This morning I got up to the same issue, except the problem didn't go away
itself.

 

ASSP would core dump during startup without outputting any messages. If I
enabled debugging, not debug file was created.

 

I had to resort to strace to see that it was getting an error with space 
on
the tmpDB folder which was indeed completely full. There was over a 
gigabyte
of data contained in there, all relating to rebuildspamdb. Some was from
last night's run but some was from two nights prios.

 

I'm presuming that there is some code in the startup that clears up 
leftover
rebuildspamdb data as I emptied the folder but did not remove it. After
starting ASSP the rebuild folder disappeared.

 

I suspect this code needs to be called much earlier in the startup 
process.
I'll include the strace failure incase it gives an idea of where in the
process it needs to be moved to.

 

All the best,

Colin Waring.

 

 

stat64(/usr/local/assp/tmpDB, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=80,
...}) = 0

stat64(/usr/local/assp/tmpDB/_cachecheck, {st_mode=S_IFDIR|0755,
st_size=80, ...}) = 0

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.001, 
{st_mode=S_IFREG|0644,
st_size=0, ...}) = 0

unlink(/usr/local/assp/tmpDB/_cachecheck/__db.001) = 0

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.002, 0x8c07064) = -1 
ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.003, 0x8c07064) = -1 
ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.004, 0x8c07064) = -1 
ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt,
{st_mode=S_IFREG|0644, st_size=98, ...}) = 0

unlink(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt) = 
0

open(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt,
O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 4

_llseek(4, 0, [0], SEEK_END)= 0

ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfaccc38) = -1 ENOTTY
(Inappropriate ioctl for device)

_llseek(4, 0, [0], SEEK_CUR)= 0

fstat64(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

fcntl64(4, F_SETFD, FD_CLOEXEC) = 0

time(NULL)  = 1397029108

stat64(/etc/localtime, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0

time(NULL)  = 1397029108

open(/sys/devices/system/cpu/online, O_RDONLY|O_CLOEXEC) = 5

read(5, 0\n, 8192)= 2

close(5)= 0

write(4, 2014-04-09 08:38:28\nBDB cachesiz..., 50) = 50

fcntl64(4, F_GETFL) = 0x8401 (flags
O_WRONLY|O_APPEND|O_LARGEFILE)

fstat64(4, {st_mode=S_IFREG|0644, st_size=50, ...}) = 0

mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
=
0xb6ee

_llseek(4, 0, [50], SEEK_CUR)   = 0

open(/usr/local/assp/tmpDB/_cachecheck/DB_CONFIG, O_RDONLY|O_LARGEFILE) 
=
-1 ENOENT (No such file or directory)

stat64(/var/tmp, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0

open(/usr/local/assp/tmpDB/_cachecheck/__db.001,
O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 5

fcntl64(5, F_GETFD) = 0

fcntl64(5, F_SETFD, FD_CLOEXEC) = 0

open(/usr/local/assp/tmpDB/_cachecheck/__db.001,
O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 8

fcntl64(8, F_GETFD) = 0

fcntl64(8, F_SETFD, FD_CLOEXEC) = 0

_llseek(8, 16384, [16384], SEEK_SET)= 0

write(8,
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0..., 
8192)
= -1 ENOSPC (No space left on device)

write(4, write: 0xbc3caf0, 8192: No space..., 48) = 48

mmap2(NULL, 24576, PROT_READ|PROT_WRITE, MAP_SHARED, 8, 0) = 0xb66bd000

close(8)= 0

--- SIGBUS (Bus error) @ 0 (0) ---

+++ killed by SIGBUS (core

[Assp-test] Slow processing log files during rebuild

[Colin Waring]

Hi all.

Since clearing the tempdb folder rebuilds have slowed to a crawl.

The affected part is griplist and bounce reports as per logs:

2014-04-09 23:40:33 building new GripList records and bounce report
2014-04-09 23:40:33 processing Logfile /usr/local/assp/maillog.txt
2014-04-10 07:00:15 processing Logfile /usr/local/assp/14-04-08.maillog.txt
2014-04-10 16:40:57 processing Logfile /usr/local/assp/14-04-07.maillog.txt
2014-04-10 16:41:02 processing Logfile /usr/local/assp/14-04-06.maillog.txt
2014-04-10 17:30:54 processing Logfile /usr/local/assp/14-04-05.maillog.txt
2014-04-10 17:30:56 processing Logfile /usr/local/assp/14-03-09.maillog.txt

Any ideas what could cause that and how to debug further/resolve?

All the best,
Colin Waring
--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test  Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP fails with no error message when tmpDB folder full

[Colin Waring]

Apparently I spoke too soon, ASSP cleared out the folder after starting fully

All the best,
Colin Waring

On 11 Apr 2014 07:50, Colin Waring co...@lanternhosting.co.uk wrote:

 Hi Thomas, 

 I didn't have to wait long, it turns out two runs of rebuildspamdb are 
 enough to fill a 1GB tempdb folder now. 

 I have added the entry and ASSP starts back up without coredumping. tmpDB 
 does remain 100% full though 

 Has anything changed recently that would increase the tmpDB requirements? 

 All the best, 
 Colin Waring. 

 -Original Message- 
 From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
 Sent: 09 April 2014 09:23 
 To: ASSP development mailing list 
 Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder 
 full 

 add the following line to 'lib/CorrectASSPcfg.pm' 

 $main::BDBMaxCacheSize = 0; 

 and restart assp. Tell me if it works or not. 

 Thomas 



 Von:    Colin Waring co...@lanternhosting.co.uk 
 An: 'ASSP development mailing list' 
 assp-test@lists.sourceforge.net, 
 Datum:  09.04.2014 09:56 
 Betreff:    [Assp-test] ASSP fails with no error message when tmpDB 
 folder full 



 Hi Folks, 



 At the weekend one of my mailservers died overnight. I had a quick check 
 over and saw that it was coredumping without any errors. I decided to 
 leave 
 it till the morning as the other servers could handle things. By morning 
 my 
 monitoring scripts had restarted it. 



 This morning I got up to the same issue, except the problem didn't go away 
 itself. 



 ASSP would core dump during startup without outputting any messages. If I 
 enabled debugging, not debug file was created. 



 I had to resort to strace to see that it was getting an error with space 
 on 
 the tmpDB folder which was indeed completely full. There was over a 
 gigabyte 
 of data contained in there, all relating to rebuildspamdb. Some was from 
 last night's run but some was from two nights prios. 



 I'm presuming that there is some code in the startup that clears up 
 leftover 
 rebuildspamdb data as I emptied the folder but did not remove it. After 
 starting ASSP the rebuild folder disappeared. 



 I suspect this code needs to be called much earlier in the startup 
 process. 
 I'll include the strace failure incase it gives an idea of where in the 
 process it needs to be moved to. 



 All the best, 

 Colin Waring. 





 stat64(/usr/local/assp/tmpDB, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=80, 
 ...}) = 0 

 stat64(/usr/local/assp/tmpDB/_cachecheck, {st_mode=S_IFDIR|0755, 
 st_size=80, ...}) = 0 

 lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.001, 
 {st_mode=S_IFREG|0644, 
 st_size=0, ...}) = 0 

 unlink(/usr/local/assp/tmpDB/_cachecheck/__db.001) = 0 

 lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.002, 0x8c07064) = -1 
 ENOENT 
 (No such file or directory) 

 lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.003, 0x8c07064) = -1 
 ENOENT 
 (No such file or directory) 

 lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.004, 0x8c07064) = -1 
 ENOENT 
 (No such file or directory) 

 lstat64(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt, 
 {st_mode=S_IFREG|0644, st_size=98, ...}) = 0 

 unlink(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt) = 
 0 

 open(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt, 
 O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 4 

 _llseek(4, 0, [0], SEEK_END)    = 0 

 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfaccc38) = -1 ENOTTY 
 (Inappropriate ioctl for device) 

 _llseek(4, 0, [0], SEEK_CUR)    = 0 

 fstat64(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 

 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0 

 time(NULL)  = 1397029108 

 stat64(/etc/localtime, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0 

 time(NULL)  = 1397029108 

 open(/sys/devices/system/cpu/online, O_RDONLY|O_CLOEXEC) = 5 

 read(5, 0\n, 8192)    = 2 

 close(5)    = 0 

 write(4, 2014-04-09 08:38:28\nBDB cachesiz..., 50) = 50 

 fcntl64(4, F_GETFL) = 0x8401 (flags 
 O_WRONLY|O_APPEND|O_LARGEFILE) 

 fstat64(4, {st_mode=S_IFREG|0644, st_size=50, ...}) = 0 

 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
 = 
 0xb6ee 

 _llseek(4, 0, [50], SEEK_CUR)   = 0 

 open(/usr/local/assp/tmpDB/_cachecheck/DB_CONFIG, O_RDONLY|O_LARGEFILE) 
 = 
 -1 ENOENT (No such file or directory) 

 stat64(/var/tmp, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0 

 open(/usr/local/assp/tmpDB/_cachecheck/__db.001, 
 O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 5 

 fcntl64(5, F_GETFD) = 0 

 fcntl64(5, F_SETFD, FD_CLOEXEC) = 0 

 open(/usr/local/assp/tmpDB/_cachecheck/__db.001, 
 O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 8 

 fcntl64(8, F_GETFD) = 0 

 fcntl64(8, F_SETFD, FD_CLOEXEC) = 0 

 _llseek(8, 16384, [16384], SEEK_SET

[Assp-test] ASSP fails with no error message when tmpDB folder full

[Colin Waring]

Hi Folks,

 

At the weekend one of my mailservers died overnight. I had a quick check
over and saw that it was coredumping without any errors. I decided to leave
it till the morning as the other servers could handle things. By morning my
monitoring scripts had restarted it.

 

This morning I got up to the same issue, except the problem didn't go away
itself.

 

ASSP would core dump during startup without outputting any messages. If I
enabled debugging, not debug file was created.

 

I had to resort to strace to see that it was getting an error with space on
the tmpDB folder which was indeed completely full. There was over a gigabyte
of data contained in there, all relating to rebuildspamdb. Some was from
last night's run but some was from two nights prios.

 

I'm presuming that there is some code in the startup that clears up leftover
rebuildspamdb data as I emptied the folder but did not remove it. After
starting ASSP the rebuild folder disappeared.

 

I suspect this code needs to be called much earlier in the startup process.
I'll include the strace failure incase it gives an idea of where in the
process it needs to be moved to.

 

All the best,

Colin Waring.

 

 

stat64(/usr/local/assp/tmpDB, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=80,
...}) = 0

stat64(/usr/local/assp/tmpDB/_cachecheck, {st_mode=S_IFDIR|0755,
st_size=80, ...}) = 0

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.001, {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0

unlink(/usr/local/assp/tmpDB/_cachecheck/__db.001) = 0

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.002, 0x8c07064) = -1 ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.003, 0x8c07064) = -1 ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.004, 0x8c07064) = -1 ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt,
{st_mode=S_IFREG|0644, st_size=98, ...}) = 0

unlink(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt) = 0

open(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt,
O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 4

_llseek(4, 0, [0], SEEK_END)= 0

ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfaccc38) = -1 ENOTTY
(Inappropriate ioctl for device)

_llseek(4, 0, [0], SEEK_CUR)= 0

fstat64(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

fcntl64(4, F_SETFD, FD_CLOEXEC) = 0

time(NULL)  = 1397029108

stat64(/etc/localtime, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0

time(NULL)  = 1397029108

open(/sys/devices/system/cpu/online, O_RDONLY|O_CLOEXEC) = 5

read(5, 0\n, 8192)= 2

close(5)= 0

write(4, 2014-04-09 08:38:28\nBDB cachesiz..., 50) = 50

fcntl64(4, F_GETFL) = 0x8401 (flags
O_WRONLY|O_APPEND|O_LARGEFILE)

fstat64(4, {st_mode=S_IFREG|0644, st_size=50, ...}) = 0

mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0xb6ee

_llseek(4, 0, [50], SEEK_CUR)   = 0

open(/usr/local/assp/tmpDB/_cachecheck/DB_CONFIG, O_RDONLY|O_LARGEFILE) =
-1 ENOENT (No such file or directory)

stat64(/var/tmp, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0

open(/usr/local/assp/tmpDB/_cachecheck/__db.001,
O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 5

fcntl64(5, F_GETFD) = 0

fcntl64(5, F_SETFD, FD_CLOEXEC) = 0

open(/usr/local/assp/tmpDB/_cachecheck/__db.001,
O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 8

fcntl64(8, F_GETFD) = 0

fcntl64(8, F_SETFD, FD_CLOEXEC) = 0

_llseek(8, 16384, [16384], SEEK_SET)= 0

write(8,
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0..., 8192)
= -1 ENOSPC (No space left on device)

write(4, write: 0xbc3caf0, 8192: No space..., 48) = 48

mmap2(NULL, 24576, PROT_READ|PROT_WRITE, MAP_SHARED, 8, 0) = 0xb66bd000

close(8)= 0

--- SIGBUS (Bus error) @ 0 (0) ---

+++ killed by SIGBUS (core dumped) +++

--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test  Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP fails with no error message when tmpDB folder full

[Colin Waring]

Hi Thomas,

Thanks for the reply - unfortunately as I cleared out the tmpdb folder the
problem has gone away and I can't do any further testing. The folder is
currently around 45% used of 1GB so if the folder fills again I can look at
making the suggested change.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 09 April 2014 09:23
To: ASSP development mailing list
Subject: Re: [Assp-test] ASSP fails with no error message when tmpDB folder
full

add the following line to 'lib/CorrectASSPcfg.pm'

$main::BDBMaxCacheSize = 0;

and restart assp. Tell me if it works or not.

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net, 
Datum:  09.04.2014 09:56
Betreff:[Assp-test] ASSP fails with no error message when tmpDB 
folder full



Hi Folks,

 

At the weekend one of my mailservers died overnight. I had a quick check
over and saw that it was coredumping without any errors. I decided to 
leave
it till the morning as the other servers could handle things. By morning 
my
monitoring scripts had restarted it.

 

This morning I got up to the same issue, except the problem didn't go away
itself.

 

ASSP would core dump during startup without outputting any messages. If I
enabled debugging, not debug file was created.

 

I had to resort to strace to see that it was getting an error with space 
on
the tmpDB folder which was indeed completely full. There was over a 
gigabyte
of data contained in there, all relating to rebuildspamdb. Some was from
last night's run but some was from two nights prios.

 

I'm presuming that there is some code in the startup that clears up 
leftover
rebuildspamdb data as I emptied the folder but did not remove it. After
starting ASSP the rebuild folder disappeared.

 

I suspect this code needs to be called much earlier in the startup 
process.
I'll include the strace failure incase it gives an idea of where in the
process it needs to be moved to.

 

All the best,

Colin Waring.

 

 

stat64(/usr/local/assp/tmpDB, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=80,
...}) = 0

stat64(/usr/local/assp/tmpDB/_cachecheck, {st_mode=S_IFDIR|0755,
st_size=80, ...}) = 0

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.001, 
{st_mode=S_IFREG|0644,
st_size=0, ...}) = 0

unlink(/usr/local/assp/tmpDB/_cachecheck/__db.001) = 0

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.002, 0x8c07064) = -1 
ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.003, 0x8c07064) = -1 
ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/__db.004, 0x8c07064) = -1 
ENOENT
(No such file or directory)

lstat64(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt,
{st_mode=S_IFREG|0644, st_size=98, ...}) = 0

unlink(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt) = 
0

open(/usr/local/assp/tmpDB/_cachecheck/BDB-cachesize-test-error.txt,
O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 4

_llseek(4, 0, [0], SEEK_END)= 0

ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfaccc38) = -1 ENOTTY
(Inappropriate ioctl for device)

_llseek(4, 0, [0], SEEK_CUR)= 0

fstat64(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

fcntl64(4, F_SETFD, FD_CLOEXEC) = 0

time(NULL)  = 1397029108

stat64(/etc/localtime, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0

time(NULL)  = 1397029108

open(/sys/devices/system/cpu/online, O_RDONLY|O_CLOEXEC) = 5

read(5, 0\n, 8192)= 2

close(5)= 0

write(4, 2014-04-09 08:38:28\nBDB cachesiz..., 50) = 50

fcntl64(4, F_GETFL) = 0x8401 (flags
O_WRONLY|O_APPEND|O_LARGEFILE)

fstat64(4, {st_mode=S_IFREG|0644, st_size=50, ...}) = 0

mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
=
0xb6ee

_llseek(4, 0, [50], SEEK_CUR)   = 0

open(/usr/local/assp/tmpDB/_cachecheck/DB_CONFIG, O_RDONLY|O_LARGEFILE) 
=
-1 ENOENT (No such file or directory)

stat64(/var/tmp, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0

open(/usr/local/assp/tmpDB/_cachecheck/__db.001,
O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 5

fcntl64(5, F_GETFD) = 0

fcntl64(5, F_SETFD, FD_CLOEXEC) = 0

open(/usr/local/assp/tmpDB/_cachecheck/__db.001,
O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 8

fcntl64(8, F_GETFD) = 0

fcntl64(8, F_SETFD, FD_CLOEXEC) = 0

_llseek(8, 16384, [16384], SEEK_SET)= 0

write(8,
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0..., 
8192)
= -1 ENOSPC (No space left on device)

write(4, write: 0xbc3caf0, 8192: No space..., 48) = 48

mmap2(NULL, 24576, PROT_READ|PROT_WRITE, MAP_SHARED, 8, 0) = 0xb66bd000

close(8)= 0

--- SIGBUS (Bus error) @ 0 (0) ---

+++ killed by SIGBUS (core dumped

[Assp-test] Internal Caches

[Colin Waring]

Hi There,

 

Can we add a DKIMCache link on the internal caches section? It currently
resides as a button in the DKIMCacheInterval section only.

 

All the best,

Colin Waring.

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] dev versions

[Colin Waring]

http://assp.cvs.sourceforge.net/viewvc/assp/assp2/

This will give you the latest dev version and you can see all of the other
files that may have been altered.

All the best,
Colin Waring.

-Original Message-
From: K Post [mailto:nntp.p...@gmail.com] 
Sent: 26 March 2014 15:09
To: ASSP development mailing list
Subject: [Assp-test] dev versions

Looks like I'm running build 14081 of 2.4.1, but the updates on this
mailing list talk about 2.4.2 (dev right?).

I've looked at the v2 folder in http://sourceforge.net/projects/assp/files,
but I only see a 2.4.1 version there.  Questions:

1) Where do I manually download dev/2.4.2 versions from?
   a) if there a historical section too where an older build could be
downloaded to revert if necessary?

2) How often are prod versions updated?  I wouldn't think that frequently.
 Does it just happen that I installed right after a prod update?

3) Is there a changelog showing how the dev version compares to prod?

thanks

--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] BombHeaderRe

[Colin Waring]

Hi There,

 

Simple question - the default files/bombheaderre.txt contains two entries on
sourceforge. One of these lines has a number of Months in it. What is this
check designed to catch as I am seeing false positives?

 

What I am seeing is the regex matching regular Date: headers such as:

 

Date: Mon, 24 Mar 2014 11:37:55 +

 

This doesn't look like it happened until I updated to the most recent
version this morning.

 

All the best,

Colin Waring.

--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Is whiteListedDomains cached anywhere?

[Colin Waring]

Hi again.

 

I'm having a bad day of it. We've spent months trying to convince someone to
switch over to our spam filtering and they're already starting to demand we
change back :/

 

First off it was my fault because we had to whitelist them previously so
that they could email us. After removing the entry from the whitelist, ASSP
is still letting spam through as follows:

 

2014-03-24 14:54:40 [Worker_1]
discounted-drugstor...@plus.pl,u...@domain.tld matches @domain.tld in
whiteListedDomains

2014-03-24 14:54:43 [Worker_1] Error: the local address 'u...@domain.tld'
matches a definition in 'whiteListedDomains' - please remove this entry

2014-03-24 14:54:44 m1-72878-10018 [Worker_1] [TLS-out] [MessageOK]
46.169.117.212 discounted-drugstor...@plus.pl to: u...@domain.tld message
ok - (whiteListedDomains '@domain.tld') - [The Highest Grade Drugs And EXTRA
LOW Price] -
/usr/local/assp/store/notspam/The_Highest_Grade_Drugs_And_EXTRA_LOW_Price--1
367765.eml

 

The domain was removed from the whitelist over an hour ago. If I run the
email through the mail analyser is detects it as spam as expected. I have
restarted ASSP and grepped all of the files in the ASSP directory to make
sure there are no other errant entries.

 

I also note that the message ID is missing from two of the log lines.

 

How can I force ASSP to recognise that the domain is not in
whiteListedDomains anymore?

 

All the best,

Colin Waring.

--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Is whiteListedDomains cached anywhere?

[Colin Waring]

Hi,

There seems to be something wrong with the latest version, I've had to drop
back to 2.4.2(14072).

Since sending this message I noticed that the matches were completely
incorrect - it was matching the none-existant whiteListedDomains entry on
the recipient rather than the send address.

After dropping back to 14072 the log entries containing whiteListedDomains
entries have stopped completely and I can see spam being blocked again.

All the best,
Colin Waring.

-Original Message-
From: Colin Waring [mailto:co...@lanternhosting.co.uk] 
Sent: 24 March 2014 15:37
To: 'ASSP development mailing list'
Subject: [Assp-test] Is whiteListedDomains cached anywhere?

Hi again.

 

I'm having a bad day of it. We've spent months trying to convince someone to
switch over to our spam filtering and they're already starting to demand we
change back :/

 

First off it was my fault because we had to whitelist them previously so
that they could email us. After removing the entry from the whitelist, ASSP
is still letting spam through as follows:

 

2014-03-24 14:54:40 [Worker_1]
discounted-drugstor...@plus.pl,u...@domain.tld matches @domain.tld in
whiteListedDomains

2014-03-24 14:54:43 [Worker_1] Error: the local address 'u...@domain.tld'
matches a definition in 'whiteListedDomains' - please remove this entry

2014-03-24 14:54:44 m1-72878-10018 [Worker_1] [TLS-out] [MessageOK]
46.169.117.212 discounted-drugstor...@plus.pl to: u...@domain.tld message
ok - (whiteListedDomains '@domain.tld') - [The Highest Grade Drugs And EXTRA
LOW Price] -
/usr/local/assp/store/notspam/The_Highest_Grade_Drugs_And_EXTRA_LOW_Price--1
367765.eml

 

The domain was removed from the whitelist over an hour ago. If I run the
email through the mail analyser is detects it as spam as expected. I have
restarted ASSP and grepped all of the files in the ASSP directory to make
sure there are no other errant entries.

 

I also note that the message ID is missing from two of the log lines.

 

How can I force ASSP to recognise that the domain is not in
whiteListedDomains anymore?

 

All the best,

Colin Waring.


--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] DKIM spam

[Colin Waring]

Hi Peter,

Thanks for the email, configuring it that way makes far more sense than
turning it off. Last week was rather busy so I didn't give it the thought it
deserved past turning it off.

I've ended up doing the same for SPF because we've been getting a fair
amount of that through as well.

All the best,
Colin Waring.

-Original Message-
From: Peter Hinman [mailto:peter.hin...@myib.com] 
Sent: 19 March 2014 16:30
To: assp-test@lists.sourceforge.net
Subject: Re: [Assp-test] DKIM spam

I don't give much value to a DKIM pass, but I do score on a DKIM fail.  
DKIM still has it's place as a way to identify fraudulent use of a domain.
There isn't much that can be done about hacked domains :(

Peter Hinman
International Bridge / ParcelPool.com

On 3/14/2014 8:49 AM, Colin Waring wrote:
 Thanks for the reply, it is however somewhat off the mark.

 These messages don't come from authenticated sources or even trusted 
 sources
 - they are simply remote mail servers that have a valid DKIM record 
 thus causing them to score below the threshold.

 It me, it looks like a smart spammer/botnet that is using throwaway 
 domains with DKIM records set up. The problem is that anyone can set 
 up DKIM, though up until now spammers haven't bothered going to the 
 extra effort of doing so. If spammers are now deploying DKIM for their 
 messages then DKIM can no long be relied on as an indicator of spam/ham.

 This is why I asked if anyone else was seeing the same increase in 
 DKIM signed spam.

 All the best,
 Colin Waring.

 -Original Message-
 From: Grayhat [mailto:gray...@gmx.net]
 Sent: 14 March 2014 14:18
 To: assp-test@lists.sourceforge.net
 Subject: Re: [Assp-test] DKIM spam

 :: On Fri, 14 Mar 2014 13:51:37 -
 :: 
 sig.91501147d0.01cf3f8c$85a7ed20$90f7c760$@lanternhosting.co.uk
 :: Colin Waring co...@lanternhosting.co.uk wrote:

 I was wondering if anyone else was seeing an increase in spam 
 messages that come with a valid DKIM signature? It has gotten to the 
 point where I have had to set DoDKIM to disabled because so much 
 rubbish is coming through and I can't think of many circumstances 
 where DKIM is actually used extensively.
 I don't think it's a DKIM issue (or an SPF one or whatever); see, the 
 number of bots trying to bruteforce credentials (either over SMTP or
 POP3/IMAP) dramatically raised (and I'm not counting the malware which 
 steals them from victim's machines) and once those credentials are 
 upped to some botnet controller, the bots will just start pumping a 
 lot of junk through a server using the stolen credentials and DKIM or 
 SPF won't be able to do much; bottom line, ensure to check for bounces 
 and keep an eye on your servers; as for bounces; if someone here is 
 running on win and using the IIS SMTP as the outbound mail router, it 
 may (will !) be a good idea to configure it to also send a copy of NDR 
 emails to some mailbox you manage (say ndr...@example.com) so that 
 you'll be able to see the bounces and take action (ok, this is a raw 
 and straight approach but as a first step it's better than nothing)



 --
 --
 --
 Learn Graph Databases - Download FREE O'Reilly Book Graph Databases 
 is the definitive new guide to graph databases and their applications. 
 Written by three acclaimed leaders in the field, this first edition is now
available.
 Download your free book today!
 http://p.sf.net/sfu/13534_NeoTech
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


 --
  Learn Graph Databases - Download FREE O'Reilly Book Graph 
 Databases is the definitive new guide to graph databases and their 
 applications. Written by three acclaimed leaders in the field, this 
 first edition is now available. Download your free book today!
 http://p.sf.net/sfu/13534_NeoTech
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test



--
Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the
definitive new guide to graph databases and their applications. Written by
three acclaimed leaders in the field, this first edition is now available.
Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders

[Assp-test] DKIM spam

[Colin Waring]

Hi there,

 

I was wondering if anyone else was seeing an increase in spam messages that
come with a valid DKIM signature? It has gotten to the point where I have
had to set DoDKIM to disabled because so much rubbish is coming through and
I can't think of many circumstances where DKIM is actually used extensively.

 

All the best,

Colin Waring.

--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] DKIM spam

[Colin Waring]

Thanks for the reply, it is however somewhat off the mark. 

These messages don't come from authenticated sources or even trusted sources
- they are simply remote mail servers that have a valid DKIM record thus
causing them to score below the threshold.

It me, it looks like a smart spammer/botnet that is using throwaway domains
with DKIM records set up. The problem is that anyone can set up DKIM, though
up until now spammers haven't bothered going to the extra effort of doing
so. If spammers are now deploying DKIM for their messages then DKIM can no
long be relied on as an indicator of spam/ham.

This is why I asked if anyone else was seeing the same increase in DKIM
signed spam.

All the best,
Colin Waring.

-Original Message-
From: Grayhat [mailto:gray...@gmx.net] 
Sent: 14 March 2014 14:18
To: assp-test@lists.sourceforge.net
Subject: Re: [Assp-test] DKIM spam

:: On Fri, 14 Mar 2014 13:51:37 -
:: sig.91501147d0.01cf3f8c$85a7ed20$90f7c760$@lanternhosting.co.uk
:: Colin Waring co...@lanternhosting.co.uk wrote:

 I was wondering if anyone else was seeing an increase in spam messages 
 that come with a valid DKIM signature? It has gotten to the point 
 where I have had to set DoDKIM to disabled because so much rubbish is 
 coming through and I can't think of many circumstances where DKIM is 
 actually used extensively.

I don't think it's a DKIM issue (or an SPF one or whatever); see, the number
of bots trying to bruteforce credentials (either over SMTP or
POP3/IMAP) dramatically raised (and I'm not counting the malware which
steals them from victim's machines) and once those credentials are upped to
some botnet controller, the bots will just start pumping a lot of junk
through a server using the stolen credentials and DKIM or SPF won't be able
to do much; bottom line, ensure to check for bounces and keep an eye on your
servers; as for bounces; if someone here is running on win and using the IIS
SMTP as the outbound mail router, it may (will !) be a good idea to
configure it to also send a copy of NDR emails to some mailbox you manage
(say ndr...@example.com) so that you'll be able to see the bounces and take
action (ok, this is a raw and straight approach but as a first step it's
better than nothing)




--
Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the
definitive new guide to graph databases and their applications. Written by
three acclaimed leaders in the field, this first edition is now available.
Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Repeated unable to detect any running worker for a new connection errors

[Colin Waring]

I've started getting this too about three versions ago but haven't had the
time to look into it.

Worker status is:

2   10646 s HMMOK_Run (stuck)
3   13307 s HMMOK_Run (stuck)

My main mailserver has restarted 4 times overnight - I had debugging turned
on overnight too. I have to go out to clients today but can pull the logs
later.

All the best,
Colin Waring.

-Original Message-
From: James Brown [mailto:jlbr...@bordo.com.au] 
Sent: 07 February 2014 00:03
To: assp-test@lists.sourceforge.net
Subject: [Assp-test] Repeated unable to detect any running worker for a new
connection errors

I've noticed that I've been getting this a lot recently:

Feb-07-14 10:47:49 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)
Feb-07-14 10:47:49 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)
Feb-07-14 10:47:49 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)
Feb-07-14 10:47:50 [Main_Thread] Info: unable to detect any running worker
for a new connection - wait (max 30 seconds)
Feb-07-14 10:47:50 [Main_Thread] Info: ConnectionTransferTimeOut (30
seconds) is now reached
Feb-07-14 10:47:50 [Main_Thread] Warning: Main_Thread is unable to transfer
connection to any worker - try again!
Feb-07-14 10:47:50 [Main_Thread] Error: Main_Thread is unable to transfer
connection to any worker within 120 seconds - restart ASSP!
Feb-07-14 10:47:50 [Main_Thread] Initializing shutdown sequence

Have always been on the latest version - now at 2.3.4(14037) after the above
restart.

Running on Mac 10.7.5.

Perl is 5.012003.

Any ideas?

Thanks,

James.

--
Managing the Performance of Cloud-Based Applications Take advantage of what
the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] rebuildspamdb always hangs at certain position

[Colin Waring]

Hi Thomas,

Perl v5.14.2 - this is what is bundled with Ubuntu and running multiple
versions of Perl is a headache/upgrading the system Perl is a big no.

WordStem is 1.24

The only other change I made around the same time was to enable the OCR
plugin but none of the threads say anything about that. 

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 07 February 2014 09:47
To: ASSP development mailing list
Subject: Re: [Assp-test] rebuildspamdb always hangs at certain position

Is there anyone else having a problem with a stucking rebuild process or
hanging workers  (on HMM or Bayes) running ASSP_WordStem 1.24 and Perl
5.16.3 or later?

Thomas





Von:Michael a...@agrodur.com
An: assp-test assp-test@lists.sourceforge.net, 
Datum:  07.02.2014 10:18
Betreff:[Assp-test] rebuildspamdb always hangs at certain position



Good Morning!

Since version 14034 with the new version of wordstem the rebuildspamdb task
always hangs at a certain position in /errors/notspam and does not complete
the job.

Any hints how to locate the file which causes the problem? Then I could try
again after deletion.

After downgrading to 14033 and the prior wordstem.pm rebuildspamdb works
fine again.

Regards
Michael



--
Managing the Performance of Cloud-Based Applications Take advantage of what
the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***




--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP_WordStem.pm 1.25

[Colin Waring]

Hi Thomas,

Thanks for the quick work - I will get it up and running now.

Oddly I had several restarts due to hanging workers last night but since 9am
this morning not a single one. I'm a bit baffled on that one..

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 07 February 2014 16:39
To: ASSP development mailing list
Subject: [Assp-test] ASSP_WordStem.pm 1.25

Hi all,

ASSP_WordStem.pm version 1.25 is released on SF-CVS.
It will fix the hanging Workers/rebuildspamdb for Perl version  5.16. I
don't expect to see this issue on Perl 5.16 and higher.

Responses are wanted!

Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***




--
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.3.4 build 14032

[Colin Waring]

Hi Thomas,

I can see the entry for BayesAfterHMM appearing in the sync config but not
in assp.cfg and there doesn't appear to be any option in the GUI for it.

Where should it appear?

Thanks for the quick work though!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 01 February 2014 11:21
To: ASSP development mailing list
Subject: [Assp-test] fixes in assp 2.3.4 build 14032

Hi all,

fixed in assp 2.3.4 build 14032:

- some bomb regular expression were not working if 'DoTransliterate' was set
to on

- the DoIPinHelo check has show wrong results in the analyzer

- reduces the temporary memory usage for bomb , HMM and Bayesian checks

- some of the 'bulls' in the analyzer had a wrong color in some cases

changed:

- updated file 'ASSP-MIB'

added:

'BayesAfterHMM','Do Bayesian depends on HMM results'
 This value is ignored if DoHMM is not enabled or set to monitor. The
Bayesian check will only run,  if the spam/ham probability of the HMM check
is in a given value range or the HMM check has given too few results.
 Leave this blank to run the Bayesian check every time, independend from any
HMM result (default).
 To set this value, define a probability value range like 0.3-0.7 (eg.).'


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***




--
WatchGuard Dimension instantly turns raw network data into actionable 
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.3.4 build 14032

[Colin Waring]

Ignore this!

My local sourceforge mirror obviously hadn't updated and one of my servers
didn't pull the latest version. It looks to have now!

-Original Message-
From: Colin Waring [mailto:co...@lanternhosting.co.uk] 
Sent: 01 February 2014 12:22
To: 'ASSP development mailing list'
Subject: Re: [Assp-test] fixes in assp 2.3.4 build 14032

Hi Thomas,

I can see the entry for BayesAfterHMM appearing in the sync config but not
in assp.cfg and there doesn't appear to be any option in the GUI for it.

Where should it appear?

Thanks for the quick work though!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 01 February 2014 11:21
To: ASSP development mailing list
Subject: [Assp-test] fixes in assp 2.3.4 build 14032

Hi all,

fixed in assp 2.3.4 build 14032:

- some bomb regular expression were not working if 'DoTransliterate' was set
to on

- the DoIPinHelo check has show wrong results in the analyzer

- reduces the temporary memory usage for bomb , HMM and Bayesian checks

- some of the 'bulls' in the analyzer had a wrong color in some cases

changed:

- updated file 'ASSP-MIB'

added:

'BayesAfterHMM','Do Bayesian depends on HMM results'
 This value is ignored if DoHMM is not enabled or set to monitor. The
Bayesian check will only run,  if the spam/ham probability of the HMM check
is in a given value range or the HMM check has given too few results.
 Leave this blank to run the Bayesian check every time, independend from any
HMM result (default).
 To set this value, define a probability value range like 0.3-0.7 (eg.).'


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***





--
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import a
virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
WatchGuard Dimension instantly turns raw network data into actionable 
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Bayes mistake

[Colin Waring]

Hi Thomas,

Turns out another one got through the spam filtering yesterday evening.
Again same message content.

We have it the way it is because HMM misses smaller messages, we can't put
either one to a higher weight otherwise we end up with more false positives.
I'd love to turn off Bayes and just use HMM but it isn't worth it for the
complaints on the short messages spam that gets through.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 31 January 2014 07:04
To: ASSP development mailing list
Subject: Re: [Assp-test] Bayes mistake

Two reasons:

I hadn't reported the previous one as a false negative yet.

1) Another one has reported the same or similar mail. ASSP V2 recalculates
the Bayes and HMM database on the fly if a mail is reported

2) A rebuild was done.

Is there any way to figure out why Bayes made a boob on the first one?

No - all checks are done on the current DB's - no chance to go back in the
past. But I think, after eliminating pairs of very low (ham) and very high
(spam) values, there was at least one very low value left.

If you use both HMM and Bayes - set the scoring so, that your trust on HMM
is higher. Bayes is fine but less exact - for this reason HMM was
implemented.

Thomas




Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  30.01.2014 21:15
Betreff:[Assp-test] Bayes mistake



Hi there,

 

I'm wondering what's the best way to troubleshoot a Bayes mistake. We get
tonnes of fake bank security alert emails and nearly all of them got
blocked.

 

Imagine my surprise to see one in my own inbox this morning from
barcl...@email.barclays.co.uk mailto:barcl...@email.barclays.co.uk 

 

So I checked the logs. What I found was more surprising. The exact same
message with the exact same content (I compared the .eml files and only 
the
headers were different) hit my server later on and was blocked by Bayes. I
hadn't reported the previous one as a false negative yet.

 

Is there any way to figure out why Bayes made a boob on the first one?

 

Cheers,

Colin.

 

2014-01-30 09:41:52 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 barcl...@email.barclays.co.uk to: m...@mydomain.tld HMM 
Check
[scoring] - Prob: 1.0 = spam

2014-01-30 09:41:52 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 barcl...@email.barclays.co.uk to: m...@mydomain.tld
Message-Score: added 20 for HMM Probability: 1., total score for this
message is now 35

2014-01-30 09:41:53 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 barcl...@email.barclays.co.uk to: m...@mydomain.tld 
Bayesian
Check [scoring] - Prob: 0.10750 = ham

 

 

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld HMM Check [scoring] -
Prob: 1.0 = spam

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld Message-Score: added 
20
for HMM Probability: 1., total score for this message is now 40

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld Bayesian Check 
[scoring]
- Prob: 0.99597 = spam

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld Message-Score: added 
30
for Bayesian Probability: 0.99597, total score for this message is now 70

 

 


--
WatchGuard Dimension instantly turns raw network data into actionable 
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
WatchGuard Dimension instantly turns raw network data into actionable 
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk

Re: [Assp-test] Bayes mistake

[Colin Waring]

Hi Thomas,

This is actually a separate email. The previous one that I was talking about
seems to be ok now. This mistake is a different message altogether.

I was wondering if we could have the option to have HMM operate first and
Bayes only run if HMM has too few results to score the message. This would
make setting the scoring much easier as HMM can be used to outright block
and Bayes against any that HMM can't handle.

Do you think that would improve the efficiency of the system? It would also
reduce the number of checks run per message and therefore load on the system
as well.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 31 January 2014 13:26
To: ASSP development mailing list
Subject: Re: [Assp-test] Bayes mistake

Collin,

back to start - I think this was the short mail with the UTF-8 BOM and the
single link.

put the following in bombDataRe (in a single line)

^\s*[\S\x80-\xFF]{0,3}\s*(?:\/?(?:html\s*|head\s*|meta[^]*))+\s*\\s*body
\s*\s*(?:ht|f)tps?:\/\/[\w\.\/\-\?\\=]+\s*\/\s*body[^]*\s*\s*\/html\s*
[
\t\f]*\s*.{0,10}$

switch off 'DoTransliterate' otherwise the regex will not match

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  31.01.2014 11:03
Betreff:Re: [Assp-test] Bayes mistake



Hi Thomas,

Turns out another one got through the spam filtering yesterday evening.
Again same message content.

We have it the way it is because HMM misses smaller messages, we can't put
either one to a higher weight otherwise we end up with more false 
positives.
I'd love to turn off Bayes and just use HMM but it isn't worth it for the
complaints on the short messages spam that gets through.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 31 January 2014 07:04
To: ASSP development mailing list
Subject: Re: [Assp-test] Bayes mistake

Two reasons:

I hadn't reported the previous one as a false negative yet.

1) Another one has reported the same or similar mail. ASSP V2 recalculates
the Bayes and HMM database on the fly if a mail is reported

2) A rebuild was done.

Is there any way to figure out why Bayes made a boob on the first one?

No - all checks are done on the current DB's - no chance to go back in the
past. But I think, after eliminating pairs of very low (ham) and very high
(spam) values, there was at least one very low value left.

If you use both HMM and Bayes - set the scoring so, that your trust on HMM
is higher. Bayes is fine but less exact - for this reason HMM was
implemented.

Thomas




Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  30.01.2014 21:15
Betreff:[Assp-test] Bayes mistake



Hi there,

 

I'm wondering what's the best way to troubleshoot a Bayes mistake. We get
tonnes of fake bank security alert emails and nearly all of them got
blocked.

 

Imagine my surprise to see one in my own inbox this morning from
barcl...@email.barclays.co.uk mailto:barcl...@email.barclays.co.uk 

 

So I checked the logs. What I found was more surprising. The exact same
message with the exact same content (I compared the .eml files and only 
the
headers were different) hit my server later on and was blocked by Bayes. I
hadn't reported the previous one as a false negative yet.

 

Is there any way to figure out why Bayes made a boob on the first one?

 

Cheers,

Colin.

 

2014-01-30 09:41:52 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 barcl...@email.barclays.co.uk to: m...@mydomain.tld HMM 
Check
[scoring] - Prob: 1.0 = spam

2014-01-30 09:41:52 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 barcl...@email.barclays.co.uk to: m...@mydomain.tld
Message-Score: added 20 for HMM Probability: 1., total score for this
message is now 35

2014-01-30 09:41:53 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 barcl...@email.barclays.co.uk to: m...@mydomain.tld 
Bayesian
Check [scoring] - Prob: 0.10750 = ham

 

 

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld HMM Check [scoring] -
Prob: 1.0 = spam

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld Message-Score: added 
20
for HMM Probability: 1., total score for this message is now 40

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld Bayesian Check 
[scoring]
- Prob: 0.99597 = spam

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld Message-Score: added 
30
for Bayesian Probability: 0.99597, total score for this message is now 70

[Assp-test] Bayes mistake

[Colin Waring]

Hi there,

 

I'm wondering what's the best way to troubleshoot a Bayes mistake. We get
tonnes of fake bank security alert emails and nearly all of them got
blocked.

 

Imagine my surprise to see one in my own inbox this morning from
barcl...@email.barclays.co.uk mailto:barcl...@email.barclays.co.uk 

 

So I checked the logs. What I found was more surprising. The exact same
message with the exact same content (I compared the .eml files and only the
headers were different) hit my server later on and was blocked by Bayes. I
hadn't reported the previous one as a false negative yet.

 

Is there any way to figure out why Bayes made a boob on the first one?

 

Cheers,

Colin.

 

2014-01-30 09:41:52 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 barcl...@email.barclays.co.uk to: m...@mydomain.tld HMM Check
[scoring] - Prob: 1.0 = spam

2014-01-30 09:41:52 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 barcl...@email.barclays.co.uk to: m...@mydomain.tld
Message-Score: added 20 for HMM Probability: 1., total score for this
message is now 35

2014-01-30 09:41:53 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 barcl...@email.barclays.co.uk to: m...@mydomain.tld Bayesian
Check [scoring] - Prob: 0.10750 = ham

 

 

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld HMM Check [scoring] -
Prob: 1.0 = spam

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld Message-Score: added 20
for HMM Probability: 1., total score for this message is now 40

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld Bayesian Check [scoring]
- Prob: 0.99597 = spam

2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
barcl...@email.barclays.co.uk to: m...@mydomain.tld Message-Score: added 30
for Bayesian Probability: 0.99597, total score for this message is now 70

 

 

--
WatchGuard Dimension instantly turns raw network data into actionable 
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Forged spam from btinternet.com

[Colin Waring]

Hi Folks,

 

I'm trying to figure out a way to block some messages so I'll start with an
example.

 

The envelope-from is lucilefo...@orange.fr and the message comes from
out.smtpout.orange.fr

The From is:

VAL HUGHES val.hughes...@btinternet.com

Reply-To is:

val.hughes...@btinternet.com

 

The message scores 25 through a DNSBL neutral response in psbl.surriel.com.
HMM has too few results and bayes gives a probability of 0.06687.

 

The message itself contains a content-type of text/html and the only content
is opening html tags, the text of a link (not even an html link) and the
closing html tags.

 

Other examples come from various ISPs around the world that are similarly
configured to let their users specify their own from addresses.

 

I'm not sure how to block these as btinternet.com publishes no SPF or DKIM
and there are so many of them that the corpus will be poisoned towards them.

 

The subject always contains the part of the from header between the quotes,
in this case VAL HUGHES - this may be the only thing that I can use to block
it.

 

Any suggestions? Message below:

 

X-Assp-Version: 2.3.4(14025) on mail.smtphost.co.uk

X-Assp-Server-TLS: yes

X-Assp-Delay: delayed for 7m 34s; 29 Jan 2014 14:22:20 +

X-Assp-Message-Score: 1 (193.252.22 in griplist (0.77))

X-Assp-Received-SPF: none ip=193.252.22.214 mailfrom=lucilefo...@orange.fr

helo=out.smtpout.orange.fr

X-Original-Authentication-Results: mail.smtphost.co.uk; spf=none

X-Assp-Message-Score: 25 (DNSBL: neutral, 193.252.22.214 listed in

psbl.surriel.com)

X-Assp-IP-Score: 25 (DNSBL: neutral, 193.252.22.214 listed in

psbl.surriel.com)

X-Assp-DNSBL: neutral, 193.252.22.214 listed in
(psbl.surriel.com-127.0.0.2; )

X-Assp-ID: mail.smtphost.co.uk m1-05340-04746

X-Assp-Session: 9937DC00

Received: from out.smtpout.orange.fr ([193.252.22.214]
helo=out.smtpout.orange.fr)

by mail.smtphost.co.uk with SMTP (2.3.4); 29 Jan 2014 14:22:20 +

Received: from mycomputer ([85.173.157.78])

by mwinf5d69 with ME

id KqEF1n00B1hmGmD03qEMi3; Wed, 29 Jan 2014 15:14:46 +0100

Message-ID: 3e86af4410ab1d2c7d833603b2709...@mwinf5d69.me-wanadoo.net

From: VAL HUGHES val.hughes...@btinternet.com

To: Judy Beadle judybea...@lyonsholidayparks.co.uk,

lloyd imrie lloyd.im...@as.elliottuk.com,

driversclub driversc...@shell.co.uk,

email em...@customer.tld, DebbieB debb...@parasene.com,

halkyn hal...@iwantago.co.uk, halkyn hal...@wantago.co.uk,

Joelaands joelaa...@outlook.com

Subject: VAL HUGHES

Date: Tue, 29 Jan 2014 03:14:21 +0100

MIME-Version: 1.0

X-mailer: Microsoft Office Outlook, Build 11.0.5510

Reply-To: val.hughes...@btinternet.com

Content-type: Multipart/mixed; boundary=4EA5B719_4BB697AF_boundary

Content-Description: Multipart message

 

--4EA5B719_4BB697AF_boundary

Content-type: text/html; charset=UTF-8

Content-Transfer-Encoding: Quoted-printable

Content-Disposition: inline

Content-Description: HTML text

 

=EF=BB=BFhtmlheadmeta http-equiv=3Dcontent-type content: text/html;=

charset=

=3DUTF-8/headbodyhttp://sohbetdostlari.com/vjt/khwsdgunwyyqn.ihzbug/bo
dy=

/html

--4EA5B719_4BB697AF_boundary--

--
WatchGuard Dimension instantly turns raw network data into actionable 
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Forged spam from btinternet.com

[Colin Waring]

Hi Doug,

Unfortunately BT use Yahoo! Mail as well as their own infrastructure for
sending mail. I have no way of knowing where emails will come from in order
to create an SPF record. Good suggestion though!

All the best,
Colin Waring.


-Original Message-
From: Doug Lytle [mailto:supp...@drdos.info] 
Sent: 29 January 2014 16:19
To: ASSP development mailing list
Subject: Re: [Assp-test] Forged spam from btinternet.com

 I'm not sure how to block these as btinternet.com publishes no SPF

You can use spf override and create them a SPF record in ASSP, then use SPF
strict.

I believe there are examples in the ASSP GUI.

Doug


--
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import a
virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
WatchGuard Dimension instantly turns raw network data into actionable 
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: Forged spam from btinternet.com

[Colin Waring]

BT is the national incumbent telecoms provider in the UK and as a result a
lot of legitimate mail will come from this domain.

It would be really helpful if they implemented SPF themselves or something
similar but BT aren't exactly known for their helpfulness

I'll try the addition to bombDataRe as I can see this in the other examples
of the spam as well. Hopefully that will do it.

Thanks for the suggestions.
Colin

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 29 January 2014 16:44
To: ASSP development mailing list
Subject: [Assp-test] Antwort: Forged spam from btinternet.com

If you don't expect to get legit mails from @btinternet.com , block this
with (for example) 'bombHeaderRe'

\w+\@btinternet\.com

if 'DoBombHeaderRe' is in scoring mode and the default score is too less to
block via PenaltyBox, use

\w+\@btinternet\.com=100


There is one really strange unique thing in this mail - the encoded (here
Quoted-printable) UTF-8 BOM at the begin of the text -  =EF=BB=BF.

'bombDataRe' would find this with:

^\s*\xEF\xBB\xBF=100


or set the weights so, that adding the 25 (DNSBL: neutral) is high enough
for both to block at the PenaltyBox

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  29.01.2014 16:57
Betreff:[Assp-test] Forged spam from btinternet.com



Hi Folks,

 

I'm trying to figure out a way to block some messages so I'll start with 
an
example.

 

The envelope-from is lucilefo...@orange.fr and the message comes from
out.smtpout.orange.fr

The From is:

VAL HUGHES val.hughes...@btinternet.com

Reply-To is:

val.hughes...@btinternet.com

 

The message scores 25 through a DNSBL neutral response in 
psbl.surriel.com.
HMM has too few results and bayes gives a probability of 0.06687.

 

The message itself contains a content-type of text/html and the only 
content
is opening html tags, the text of a link (not even an html link) and the
closing html tags.

 

Other examples come from various ISPs around the world that are similarly
configured to let their users specify their own from addresses.

 

I'm not sure how to block these as btinternet.com publishes no SPF or DKIM
and there are so many of them that the corpus will be poisoned towards 
them.

 

The subject always contains the part of the from header between the 
quotes,
in this case VAL HUGHES - this may be the only thing that I can use to 
block
it.

 

Any suggestions? Message below:

 

X-Assp-Version: 2.3.4(14025) on mail.smtphost.co.uk

X-Assp-Server-TLS: yes

X-Assp-Delay: delayed for 7m 34s; 29 Jan 2014 14:22:20 +

X-Assp-Message-Score: 1 (193.252.22 in griplist (0.77))

X-Assp-Received-SPF: none ip=193.252.22.214 mailfrom=lucilefo...@orange.fr

helo=out.smtpout.orange.fr

X-Original-Authentication-Results: mail.smtphost.co.uk; spf=none

X-Assp-Message-Score: 25 (DNSBL: neutral, 193.252.22.214 listed in

psbl.surriel.com)

X-Assp-IP-Score: 25 (DNSBL: neutral, 193.252.22.214 listed in

psbl.surriel.com)

X-Assp-DNSBL: neutral, 193.252.22.214 listed in
(psbl.surriel.com-127.0.0.2; )

X-Assp-ID: mail.smtphost.co.uk m1-05340-04746

X-Assp-Session: 9937DC00

Received: from out.smtpout.orange.fr ([193.252.22.214]
helo=out.smtpout.orange.fr)

by mail.smtphost.co.uk with SMTP (2.3.4); 29 Jan 2014 14:22:20 
+

Received: from mycomputer ([85.173.157.78])

by mwinf5d69 with ME

id KqEF1n00B1hmGmD03qEMi3; Wed, 29 Jan 2014 15:14:46 +0100

Message-ID: 3e86af4410ab1d2c7d833603b2709...@mwinf5d69.me-wanadoo.net

From: VAL HUGHES val.hughes...@btinternet.com

To: Judy Beadle judybea...@lyonsholidayparks.co.uk,

lloyd imrie lloyd.im...@as.elliottuk.com,

driversclub driversc...@shell.co.uk,

email em...@customer.tld, DebbieB debb...@parasene.com,

halkyn hal...@iwantago.co.uk, halkyn hal...@wantago.co.uk,

Joelaands joelaa...@outlook.com

Subject: VAL HUGHES

Date: Tue, 29 Jan 2014 03:14:21 +0100

MIME-Version: 1.0

X-mailer: Microsoft Office Outlook, Build 11.0.5510

Reply-To: val.hughes...@btinternet.com

Content-type: Multipart/mixed; boundary=4EA5B719_4BB697AF_boundary

Content-Description: Multipart message

 

--4EA5B719_4BB697AF_boundary

Content-type: text/html; charset=UTF-8

Content-Transfer-Encoding: Quoted-printable

Content-Disposition: inline

Content-Description: HTML text

 

=EF=BB=BFhtmlheadmeta http-equiv=3Dcontent-type content: 
text/html;=

charset=

=3DUTF-8/headbodyhttp://sohbetdostlari.com/vjt/khwsdgunwyyqn.ihzbug
/bo
dy=

/html

--4EA5B719_4BB697AF_boundary--


--
WatchGuard Dimension instantly turns raw network data into actionable 
security intelligence. It gives you real-time visual feedback on key
security issues and trends.  Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http

Re: [Assp-test] Problems with clamav and filescanner.

[Colin Waring]

Any reason you're using filescan for clam?

This is an extract from my server build document for Ubuntu 12.04 LTS - note
that a couple of Perl modules fail to build when using the ASSP mod_inst.pl
so you have to get them installed manually. Below assumes you are starting
with a completely fresh perl install so does some config you may not need.
Also verify with the clamav-unofficial-sigs project that 3.7.2 has not been
replaced by a newer version and remember to check for updates occasionally.

apt-get install rsync clamav clamd clamav-daemon #For some reason I have
clamd and -daemon, I think one is deprecated but you would need to check
that
perl -MCPAN -e shell
install Bundle::LWP
install YAML
o conf init connect_to_internet_ok urllist
o conf init urllist
o conf urllist push http://mirror.bytemark.co.uk/CPAN/
o conf urllist push http://mirror.ox.ac.uk/sites/www.cpan.org/
o conf commit
install CPAN
reload cpan
install File::Scan::ClamAV
cd /root
wget
http://downloads.sourceforge.net/project/unofficial-sigs/clamav-unofficial-s
igs-3.7.2.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Funofficial-sigs
%2Fts=1382875090use_mirror=heanet
tar -xzvf clamav-unofficial-sigs-3.7.2.tar.gz
cd clamav-unofficial-sigs-3.7.2

You'll then need to put the script in a permenant location, set up the conf,
logrotate and cron files where they should be and you'll get a lot of handy
extra definitions

Back in ASSP, simply check UseAvClamd. Make sure the other clam settings are
ok and consider using the ASSP_AFC plugin.

I've never touched filescan settings and the stats report plenty of viruses
detected.

All the best,
Colin Waring.

-Original Message-
From: Pontus Hellgren [mailto:pon...@scandinavianhosting.se] 
Sent: 05 December 2013 11:14
To: 'ASSP development mailing list'
Subject: [Assp-test] Problems with clamav and filescanner.

Hi There!

I run ASSP version 2.3.4(13335) on Ubuntu 12,4 LTS and have installed clamd
for antivirus scanning.
For some reason it does not catch anything running it with ASSP.
If I run tests with eicar.com (txt, zip, 2xzip) files via commadline
(clamscan) picks it out, but if I mail them by ASSP there is no catch.
So, I installed AVG commandline scanner and made a script to use it as a
filescanner and ran some testruns, no catch via filescan in ASSP.
(logs say files where OK but they should not have been)
Running the AVG commadline scanner by hand worked every time, but not in
ASSP.
So, I rewrote the script to use clamscan as a filescanner, made same tests,
still no catch.

Somehow, I belive, ASSP write the .eml files, in the  virusscan directory,
in a fashion(I don't kon how), so it makes clamd, clamscan/filescanners
unable to catch the eicar.com(txt, zip, 2xzip) and then probably all other
viruses.

Here is the script I used for filescanning in ASSP:
#!/bin/bash
# Filename: avg.sh
# Createor: Pontus Hellgren 20131201
# Use: Script using avgscan for filescanning in ASSP

logfile=/var/log/assp-avg.log # Set a logfile

if [ -z $1 ] # Check for input filename
then
echo OK # NO filename input, no need to confuse ASSP
echo No file to scan, sending [OK]  $logfile # Log it to logfile
exit # No need to continue
fi

scanfilename=$1 # Set filename to scan

if [ -f $scanfilename ] # If file exist do scan
then
result=`avgscan -a $scanfilename /dev/null 21; echo $?` # Scan with AVG
and save result, discard other output
if [ $result -eq 0  ] # If 0 then no virus found, reply OK else reply VIRUS
then
echo OK
echo $scanfilename [OK]  $logfile # Log ok to logfile
echo  $logfile
elif [[ $result -eq 4 || $result -eq 5 ]] # If 4 or 5, Virus or potentially
unwanted program detected
then
echo VIRUS
echo $scanfilename [VIRUS]  $logfile # Log virus to logfile
echo  $logfile
else
echo OK # Return of any other number equals to us an error, don't confuse
ASSP
echo $scanfilename [ERROR]  $logfile # Log error to logfile
echo  $logfile
fi
fi

In ASSP I set the following:
FileScanDir: /usr/share/assp/virusscan/
FileScanCMD: /usr/share/assp/virusscan/avg.sh FILENAME
FileScanBad:VIRUS
FileScanGood:OK
FileScanRespRe:

Any ideeas to what goes wrong?

To use script for clamscan, change avgscan -a to clamscan, change the
ELIF to only 1 for VIRUS.
Clamscan only reports 2 for any error else but OK or VIRUS.

Regards,
Pontus






--
Sponsored by Intel(R) XDK 
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!
http://pubads.g.doubleclick.net/gampad/clk?id=111408631iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
Sponsored by Intel(R) XDK 
Develop, test and display web and hybrid apps with a single code base.
Download it for free now!
http://pubads.g.doubleclick.net/gampad/clk?id=111408631iu=/4140

Re: [Assp-test] Problems with clamav and filescanner.

[Colin Waring]

Have you any logs that show ASSP calling the virus scanner and what it
returns?

Have you got AVG or clamscan configured to log what they do when they are
called?

Those kind of logs will probably tell you what is going on and if there are
any obvious issues.

-Original Message-
From: Pontus Hellgren [mailto:pon...@scandinavianhosting.se] 
Sent: 05 December 2013 13:29
To: 'ASSP development mailing list'
Subject: Re: [Assp-test] Problems with clamav and filescanner.

Hi again!

As stated in the mail, filescan mode was only used to verify that/or not
clamscan or any other commandline scanner had the same failing result. (as
was the case) I am using File::Scan::ClamAV but since it does not report any
viruses anymore I had to test different ways to establish if/or not clamAV
was a problem. (seems not to be)

I will compare your install below to mine and see if there is anything that
differs(at a glance, not much!) Thanks for fast reply.
I'll be back!

Regards,
Pontus

-Original Message-
From: Colin Waring [mailto:co...@lanternhosting.co.uk]
Sent: den 5 december 2013 13:55
To: 'ASSP development mailing list'
Subject: Re: [Assp-test] Problems with clamav and filescanner.

Any reason you're using filescan for clam?

This is an extract from my server build document for Ubuntu 12.04 LTS - note
that a couple of Perl modules fail to build when using the ASSP mod_inst.pl
so you have to get them installed manually. Below assumes you are starting
with a completely fresh perl install so does some config you may not need.
Also verify with the clamav-unofficial-sigs project that 3.7.2 has not been
replaced by a newer version and remember to check for updates occasionally.

apt-get install rsync clamav clamd clamav-daemon #For some reason I have
clamd and -daemon, I think one is deprecated but you would need to check
that perl -MCPAN -e shell install Bundle::LWP install YAML o conf init
connect_to_internet_ok urllist o conf init urllist o conf urllist push
http://mirror.bytemark.co.uk/CPAN/
o conf urllist push http://mirror.ox.ac.uk/sites/www.cpan.org/
o conf commit
install CPAN
reload cpan
install File::Scan::ClamAV
cd /root
wget
http://downloads.sourceforge.net/project/unofficial-sigs/clamav-unofficial-s
igs-3.7.2.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Funofficial-sigs
%2Fts=1382875090use_mirror=heanet
tar -xzvf clamav-unofficial-sigs-3.7.2.tar.gz
cd clamav-unofficial-sigs-3.7.2

You'll then need to put the script in a permenant location, set up the conf,
logrotate and cron files where they should be and you'll get a lot of handy
extra definitions

Back in ASSP, simply check UseAvClamd. Make sure the other clam settings are
ok and consider using the ASSP_AFC plugin.

I've never touched filescan settings and the stats report plenty of viruses
detected.

All the best,
Colin Waring.

-Original Message-
From: Pontus Hellgren [mailto:pon...@scandinavianhosting.se]
Sent: 05 December 2013 11:14
To: 'ASSP development mailing list'
Subject: [Assp-test] Problems with clamav and filescanner.

Hi There!

I run ASSP version 2.3.4(13335) on Ubuntu 12,4 LTS and have installed clamd
for antivirus scanning.
For some reason it does not catch anything running it with ASSP.
If I run tests with eicar.com (txt, zip, 2xzip) files via commadline
(clamscan) picks it out, but if I mail them by ASSP there is no catch.
So, I installed AVG commandline scanner and made a script to use it as a
filescanner and ran some testruns, no catch via filescan in ASSP.
(logs say files where OK but they should not have been) Running the AVG
commadline scanner by hand worked every time, but not in ASSP.
So, I rewrote the script to use clamscan as a filescanner, made same tests,
still no catch.

Somehow, I belive, ASSP write the .eml files, in the  virusscan directory,
in a fashion(I don't kon how), so it makes clamd, clamscan/filescanners
unable to catch the eicar.com(txt, zip, 2xzip) and then probably all other
viruses.

Here is the script I used for filescanning in ASSP:
#!/bin/bash
# Filename: avg.sh
# Createor: Pontus Hellgren 20131201
# Use: Script using avgscan for filescanning in ASSP

logfile=/var/log/assp-avg.log # Set a logfile

if [ -z $1 ] # Check for input filename then echo OK # NO filename
input, no need to confuse ASSP echo No file to scan, sending [OK] 
$logfile # Log it to logfile exit # No need to continue fi

scanfilename=$1 # Set filename to scan

if [ -f $scanfilename ] # If file exist do scan then result=`avgscan -a
$scanfilename /dev/null 21; echo $?` # Scan with AVG and save result,
discard other output if [ $result -eq 0  ] # If 0 then no virus found, reply
OK else reply VIRUS then echo OK
echo $scanfilename [OK]  $logfile # Log ok to logfile echo  $logfile
elif [[ $result -eq 4 || $result -eq 5 ]] # If 4 or 5, Virus or potentially
unwanted program detected then echo VIRUS
echo $scanfilename [VIRUS]  $logfile # Log virus to logfile echo 
$logfile else echo OK # Return of any other number equals to us

Re: [Assp-test] how to report spam with MS Outlook (assp 2.3.3 13276)

[Colin Waring]

I'm not sure which options you mean by email interface as there are a lot.


EmailInterfaceOk is ticket, EmailReportDestination is left empty and the
various addresses are specified LHS only without the @ or domain.

smtpDestination and smptDestinationSSL and smtpAuthServer are all
127.0.0.1:125 which is my Exim MTA.

localDomains is a file file:files/localdomains.txt, domains are listed one
per line.

I'd suggest trying enabling auth, it isn't too difficult to get the
credentials set up and at least it rules that out as the issue.

All the best,
Colin Waring.

-Original Message-
From: aquilinux [mailto:aquili...@gmail.com] 
Sent: 02 December 2013 17:57
To: ASSP development mailing list
Subject: Re: [Assp-test] how to report spam with MS Outlook (assp 2.3.3
13276)

Please, post your relevant config entries for email interface, smtp
destinations and local domains. I don't use auth since i trust everything
coming from my org Exchange.

Thanks again.
Il 02/dic/2013 18:31 Colin Waring co...@lanternhosting.co.uk ha scritto:

 Sorry I can't guess why but can show my config.

 The domain I use is set in localdomains.txt so that is unlikely to be 
 the issue.

 The only thing I don't see in your logs is authentication which is in 
 mine..

 -Original Message-
 From: aquilinux [mailto:aquili...@gmail.com]
 Sent: 02 December 2013 16:18
 To: ASSP development mailing list
 Subject: Re: [Assp-test] how to report spam with MS Outlook (assp 
 2.3.3
 13276)

 Hi Colin, thanks for your feedback.
 Strange things are happening...
 i set a custom domain for *EmailBlockReportDomain* ie. @antispam.corp 
 and added that domain to *localDomains* as i wanted to use the same 
 for email interface.

 then:

 - if i send a mail through email interface to mail-wh...@antispam.corp 
 (my
 *EmailWhitelistAdd*) the address gets whitelisted (i receive a report 
 saying thanks bla bla.  address some@addr added to whitelist)
 - if i send a mail through email interface to mail-s...@antispam.corp 
 (my
 *EmailSpam*) i run into the issue i posted before (no address found in 
 body or header and report saying thanks bla bla., nothing more). in 
 this case spam report is not working.

 but

 - if i send a mail through email interface to 
 mail-s...@assp-nospam.org (the assp default domain) all is working as 
 your logs.

 can you guess why *EmailBlockReportDomain* is working for
 *EmailWhitelistAdd* (and basicly for any other email interface 
 address) and not for *EmailSpam* ?

 Regards.


 On Mon, Dec 2, 2013 at 12:35 PM, Colin Waring
 co...@lanternhosting.co.ukwrote:

  I have .eml in MaillogExt and everything seems to work fine. I get 
  the response email with the subject Thank you for reporting this 
  message as spam. _Hello_this_is_lantern_factory_from_Tina
 
  Email response content:
 
  china_lante...@126.com: added to the personal blacklist of 
  m...@domain.co.uk
  2013120215140629603...@china-lanterns.com: added to the personal 
  blacklist of m...@domain.co.uk
  shgy-t...@china-lanterns.com: added to the personal blacklist of 
  m...@domain.co.uk
  shyh-t...@hotmail.com: added to the personal blacklist of 
  m...@domain.co.uk
  shyh_t...@163.com: added to the personal blacklist of 
  m...@domain.co.uk
  -t...@china-lanterns.com: added to the personal blacklist of 
  m...@domain.co.uk
 
  This shows it is correctly pulling the information from the 
  attachment rather than my report email.
 
  Example logs below. Two things stand out to me - yours say no 
  attachment found and do report-header before report-body. Mine don't 
  log anything about attachments and it processes the body before the 
  header. I'm not sure of where to look next other than make sure 
  you're on the latest version and have run the most recent perl 
  module installer/updated all perl modules. I run Perl v5.14.2 on Ubuntu.
 
  All the best,
  Colin Waring.
 
  2013-12-02 11:26:05 [Worker_2] Connected: session:7A2E05D8
  1.1.1.1:57874 
  195.88.101.110:25  127.0.0.1:125
  2013-12-02 11:26:05 [Worker_2] 1.1.1.1 [SMTP Reply] 220 
  mail.smtphost.co.uk ESMTP Exim 4.76 Mon, 02 Dec 2013 11:26:05 +
  2013-12-02 11:26:06 [Worker_2] 1.1.1.1 [SMTP Reply] 250 HELP
  2013-12-02 11:26:06 [Worker_2] 1.1.1.1 info: got STARTTLS request 
  from
  1.1.1.1
  2013-12-02 11:26:06 [Worker_2] 1.1.1.1 [SMTP Reply] 220 TLS go ahead
  2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP 
  Reply]
  250 HELP
  2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 info:
  authentication - login is used
  2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP 
  Reply]
  334 
  2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP 
  Reply]
  334 
  2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP 
  Reply]
  235 Authentication succeeded
  2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out]
  1.1.1.1 m...@domain.co.uk [SMTP Reply] 250 OK
  2013-12-02 11:26:06 m1-83566-00228

Re: [Assp-test] how to report spam with MS Outlook (assp 2.3.3 13276)

[Colin Waring]

I have .eml in MaillogExt and everything seems to work fine. I get the
response email with the subject Thank you for reporting this message as
spam. _Hello_this_is_lantern_factory_from_Tina

Email response content:

china_lante...@126.com: added to the personal blacklist of m...@domain.co.uk
2013120215140629603...@china-lanterns.com: added to the personal blacklist
of m...@domain.co.uk
shgy-t...@china-lanterns.com: added to the personal blacklist of
m...@domain.co.uk
shyh-t...@hotmail.com: added to the personal blacklist of m...@domain.co.uk
shyh_t...@163.com: added to the personal blacklist of m...@domain.co.uk
-t...@china-lanterns.com: added to the personal blacklist of m...@domain.co.uk

This shows it is correctly pulling the information from the attachment
rather than my report email.

Example logs below. Two things stand out to me - yours say no attachment
found and do report-header before report-body. Mine don't log anything about
attachments and it processes the body before the header. I'm not sure of
where to look next other than make sure you're on the latest version and
have run the most recent perl module installer/updated all perl modules. I
run Perl v5.14.2 on Ubuntu.

All the best,
Colin Waring.

2013-12-02 11:26:05 [Worker_2] Connected: session:7A2E05D8 1.1.1.1:57874 
195.88.101.110:25  127.0.0.1:125
2013-12-02 11:26:05 [Worker_2] 1.1.1.1 [SMTP Reply] 220 mail.smtphost.co.uk
ESMTP Exim 4.76 Mon, 02 Dec 2013 11:26:05 +
2013-12-02 11:26:06 [Worker_2] 1.1.1.1 [SMTP Reply] 250 HELP
2013-12-02 11:26:06 [Worker_2] 1.1.1.1 info: got STARTTLS request from
1.1.1.1
2013-12-02 11:26:06 [Worker_2] 1.1.1.1 [SMTP Reply] 220 TLS go ahead
2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP Reply] 250
HELP
2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 info:
authentication - login is used
2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP Reply] 334

2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP Reply] 334

2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP Reply] 235
Authentication succeeded
2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk [SMTP Reply] 250 OK
2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk email: combined spam  whitelist report
2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk email: combined spam  noprocessing report
2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk [SMTP Reply] 250 OK
2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk [SMTP Reply] 354 OK Send spam body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address china_lante...@126.com in mail
body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address china_lante...@126.com in mail
body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address china_lante...@126.com in mail
body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address china_lante...@126.com in mail
body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address myaddre...@mydomain.co.uk in
mail body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address myaddre...@mydomain.co.uk in
mail body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address myaddre...@mydomain.co.uk in
mail body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address
2013120215140629603...@china-lanterns.com in mail body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address myaddre...@mydomain.co.uk  in
mail body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address china_lante...@126.com in mail
body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address shgy-t...@china-lanterns.com in
mail body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address shgy-t...@china-lanterns.com in
mail body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address shyh-t...@hotmail.com in mail
body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1
m...@domain.co.uk report-body: found address shyh-t...@hotmail.com in mail
body
2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out

Re: [Assp-test] how to report spam with MS Outlook (assp 2.3.3 13276)

[Colin Waring]

Sorry I can't guess why but can show my config.

The domain I use is set in localdomains.txt so that is unlikely to be the
issue.

The only thing I don't see in your logs is authentication which is in mine..

-Original Message-
From: aquilinux [mailto:aquili...@gmail.com] 
Sent: 02 December 2013 16:18
To: ASSP development mailing list
Subject: Re: [Assp-test] how to report spam with MS Outlook (assp 2.3.3
13276)

Hi Colin, thanks for your feedback.
Strange things are happening...
i set a custom domain for *EmailBlockReportDomain* ie. @antispam.corp and
added that domain to *localDomains* as i wanted to use the same for email
interface.

then:

- if i send a mail through email interface to mail-wh...@antispam.corp (my
*EmailWhitelistAdd*) the address gets whitelisted (i receive a report saying
thanks bla bla.  address some@addr added to whitelist)
- if i send a mail through email interface to mail-s...@antispam.corp (my
*EmailSpam*) i run into the issue i posted before (no address found in body
or header and report saying thanks bla bla., nothing more). in this case
spam report is not working.

but

- if i send a mail through email interface to mail-s...@assp-nospam.org(the
assp default domain) all is working as your logs.

can you guess why *EmailBlockReportDomain* is working for
*EmailWhitelistAdd* (and basicly for any other email interface address) and
not for *EmailSpam* ?

Regards.


On Mon, Dec 2, 2013 at 12:35 PM, Colin Waring
co...@lanternhosting.co.ukwrote:

 I have .eml in MaillogExt and everything seems to work fine. I get the 
 response email with the subject Thank you for reporting this message 
 as spam. _Hello_this_is_lantern_factory_from_Tina

 Email response content:

 china_lante...@126.com: added to the personal blacklist of 
 m...@domain.co.uk
 2013120215140629603...@china-lanterns.com: added to the personal 
 blacklist of m...@domain.co.uk
 shgy-t...@china-lanterns.com: added to the personal blacklist of 
 m...@domain.co.uk
 shyh-t...@hotmail.com: added to the personal blacklist of 
 m...@domain.co.uk
 shyh_t...@163.com: added to the personal blacklist of m...@domain.co.uk
 -t...@china-lanterns.com: added to the personal blacklist of 
 m...@domain.co.uk

 This shows it is correctly pulling the information from the attachment 
 rather than my report email.

 Example logs below. Two things stand out to me - yours say no 
 attachment found and do report-header before report-body. Mine don't 
 log anything about attachments and it processes the body before the 
 header. I'm not sure of where to look next other than make sure you're 
 on the latest version and have run the most recent perl module 
 installer/updated all perl modules. I run Perl v5.14.2 on Ubuntu.

 All the best,
 Colin Waring.

 2013-12-02 11:26:05 [Worker_2] Connected: session:7A2E05D8 
 1.1.1.1:57874 
 195.88.101.110:25  127.0.0.1:125
 2013-12-02 11:26:05 [Worker_2] 1.1.1.1 [SMTP Reply] 220 
 mail.smtphost.co.uk ESMTP Exim 4.76 Mon, 02 Dec 2013 11:26:05 +
 2013-12-02 11:26:06 [Worker_2] 1.1.1.1 [SMTP Reply] 250 HELP
 2013-12-02 11:26:06 [Worker_2] 1.1.1.1 info: got STARTTLS request from
 1.1.1.1
 2013-12-02 11:26:06 [Worker_2] 1.1.1.1 [SMTP Reply] 220 TLS go ahead
 2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP Reply] 
 250 HELP
 2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 info:
 authentication - login is used
 2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP Reply] 
 334 
 2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP Reply] 
 334 
 2013-12-02 11:26:06 [Worker_2] [TLS-in] [TLS-out] 1.1.1.1 [SMTP Reply] 
 235 Authentication succeeded
 2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 
 1.1.1.1 m...@domain.co.uk [SMTP Reply] 250 OK
 2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 
 1.1.1.1 m...@domain.co.uk email: combined spam  whitelist report
 2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 
 1.1.1.1 m...@domain.co.uk email: combined spam  noprocessing report
 2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 
 1.1.1.1 m...@domain.co.uk [SMTP Reply] 250 OK
 2013-12-02 11:26:06 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 
 1.1.1.1 m...@domain.co.uk [SMTP Reply] 354 OK Send spam body
 2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 
 1.1.1.1 m...@domain.co.uk report-body: found address 
 china_lante...@126.com in mail body
 2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 
 1.1.1.1 m...@domain.co.uk report-body: found address 
 china_lante...@126.com in mail body
 2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 
 1.1.1.1 m...@domain.co.uk report-body: found address 
 china_lante...@126.com in mail body
 2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS-in] [TLS-out] 
 1.1.1.1 m...@domain.co.uk report-body: found address 
 china_lante...@126.com in mail body
 2013-12-02 11:26:07 m1-83566-00228 [Worker_2] [TLS

Re: [Assp-test] how to report spam with MS Outlook (assp 2.3.3 13276)

[Colin Waring]

Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out] 172.20.55.51
m...@me.tld email: combined spam  whitelist report
Nov-29-13 14:58:18 [Worker_1] Info: report message written to -
/opt/assp/errors/spam/I_New_CyberSec_Executive_Order_Impact_On_IT_ -
24257.rpt.msg

What part of this indicates that ASSP isn't detecting it properly? It is
logging that it is seeing a report and it is writing the file to the errors
corpus not the general corpus.

This is the bit that concerns me because it isn't able to see any addresses
in the report..

Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld report-header: no addresses found in header tags
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld report-body: no addresses found in header tags

All the best,
Colin Waring

-Original Message-
From: aquilinux [mailto:aquili...@gmail.com] 
Sent: 29 November 2013 14:04
To: ASSP development mailing list
Subject: [Assp-test] how to report spam with MS Outlook (assp 2.3.3 13276)

Hi all,
i'm testing spam report with MS Outlook 2010 but i cannot get attached files
to be detected.
i'm sending spam emails to email interface address (mail-spam@assp.local)
forwarding them as attachments, one per email.
my assp flow is:

outlook - exchange -- assp + postfix -- Internet

But looking at logs it seems that attachment is not detected and forwarding
message is treated as the spam message itself:

Nov-29-13 14:58:17 [Worker_1] 172.20.55.51 [SMTP Reply] 220 EAIT - Keep it
legit, or keep out
Nov-29-13 14:58:17 [Worker_1] 172.20.55.51 [SMTP Reply] 250 DSN
Nov-29-13 14:58:17 [Worker_1] 172.20.55.51 [SMTP Reply] 220 2.0.0 Ready to
start TLS
Nov-29-13 14:58:18 [Worker_1] [TLS-in] [TLS-out] 172.20.55.51 [SMTP Reply]
250 DSN
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld info : changed sender from m...@me.tld to
prvs=7045142390=m...@me.tld
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld info: found message size announcement: 109.75 kByte
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld [SMTP Reply] 250 2.1.0 Ok
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld email: combined spam  whitelist report
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld email: combined spam  noprocessing report
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld [SMTP Reply] 250 OK
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld [SMTP Reply] 354 OK Send spam body
Nov-29-13 14:58:18 [Worker_1] Spam-Report: process message from m...@me.tld
Nov-29-13 14:58:18 [Worker_1] Spam-Report: (no attachment) - processing raw
email
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld report-header: no addresses found in header tags
Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 m...@me.tld report-body: no addresses found in header tags
Nov-29-13 14:58:18 [Worker_1] Info: report message written to -
/opt/assp/errors/spam/I_New_CyberSec_Executive_Order_Impact_On_IT_ -
24257.rpt.msg
Nov-29-13 14:58:18 [Worker_1] Spam-Report: finished report-message from
m...@me.tld
Nov-29-13 14:58:18 m1-33498-37618 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 [SMTP Reply] 250 2.0.0 Ok
Nov-29-13 14:58:18 m1-33498-37618 [Worker_1] [TLS-in] [TLS-out]
172.20.55.51 [SMTP Reply] 221 2.0.0 Bye
Nov-29-13 14:58:18 [Worker_1] Info: report successful sent to m...@me.tld

this happens either if i set MaillogExt to .eml or .msg any suggestion?

thanks.

--
Madness, like small fish, runs in hosts, in vast numbers of instances.

Nessuno mi pettina bene come il vento.

--
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET,  PHP application. Start your 15-day FREE TRIAL of AppDynamics
Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349351iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
Rapidly troubleshoot problems before they affect your business. Most IT 
organizations don't have a clear picture of how application performance 
affects their revenue. With AppDynamics, you get 100% visibility into your 
Java,.NET,  PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349351iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp

Re: [Assp-test] how to report spam with MS Outlook (assp 2.3.3 13276)

[Colin Waring]

What have you got in maillogExt?

Outlook uses .msg normally, you also need to need to have the perl module
Email::Outlook::Message installed for that.

All the best,
Colin Waring

-Original Message-
From: aquilinux [mailto:aquili...@gmail.com] 
Sent: 30 November 2013 08:59
To: ASSP development mailing list
Subject: Re: [Assp-test] how to report spam with MS Outlook (assp 2.3.3
13276)

Hi Colin,
what puzzles me is:

Nov-29-13 14:58:18 [Worker_1] Spam-Report: (no attachment) - processing raw
email

I guess it means it is not detecting the attached e-mail, hence the message
saved for spamdb is the forward e-mail itself (which doesn't contain any
reference to the spam message).

Thanks.
 Il 30/nov/2013 09:41 Colin Waring co...@lanternhosting.co.uk ha
scritto:

 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51
 m...@me.tld email: combined spam  whitelist report
 Nov-29-13 14:58:18 [Worker_1] Info: report message written to - 
 /opt/assp/errors/spam/I_New_CyberSec_Executive_Order_Impact_On_IT_ - 
 24257.rpt.msg

 What part of this indicates that ASSP isn't detecting it properly? It 
 is logging that it is seeing a report and it is writing the file to 
 the errors corpus not the general corpus.

 This is the bit that concerns me because it isn't able to see any 
 addresses in the report..

 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld report-header: no addresses found in header 
 tags
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld report-body: no addresses found in header 
 tags

 All the best,
 Colin Waring

 -Original Message-
 From: aquilinux [mailto:aquili...@gmail.com]
 Sent: 29 November 2013 14:04
 To: ASSP development mailing list
 Subject: [Assp-test] how to report spam with MS Outlook (assp 2.3.3 
 13276)

 Hi all,
 i'm testing spam report with MS Outlook 2010 but i cannot get attached 
 files to be detected.
 i'm sending spam emails to email interface address 
 (mail-spam@assp.local) forwarding them as attachments, one per email.
 my assp flow is:

 outlook - exchange -- assp + postfix -- Internet

 But looking at logs it seems that attachment is not detected and 
 forwarding message is treated as the spam message itself:

 Nov-29-13 14:58:17 [Worker_1] 172.20.55.51 [SMTP Reply] 220 EAIT - 
 Keep it legit, or keep out
 Nov-29-13 14:58:17 [Worker_1] 172.20.55.51 [SMTP Reply] 250 DSN
 Nov-29-13 14:58:17 [Worker_1] 172.20.55.51 [SMTP Reply] 220 2.0.0 
 Ready to start TLS
 Nov-29-13 14:58:18 [Worker_1] [TLS-in] [TLS-out] 172.20.55.51 [SMTP 
 Reply]
 250 DSN
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld info : changed sender from m...@me.tld to 
 prvs=7045142390=m...@me.tld
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld info: found message size announcement: 109.75 
 kByte
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld [SMTP Reply] 250 2.1.0 Ok
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld email: combined spam  whitelist report
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld email: combined spam  noprocessing report
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld [SMTP Reply] 250 OK
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld [SMTP Reply] 354 OK Send spam body
 Nov-29-13 14:58:18 [Worker_1] Spam-Report: process message from 
 m...@me.tld
 Nov-29-13 14:58:18 [Worker_1] Spam-Report: (no attachment) - 
 processing raw email
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld report-header: no addresses found in header 
 tags
 Nov-29-13 14:58:18 m1-33498-01104 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 m...@me.tld report-body: no addresses found in header 
 tags
 Nov-29-13 14:58:18 [Worker_1] Info: report message written to - 
 /opt/assp/errors/spam/I_New_CyberSec_Executive_Order_Impact_On_IT_ - 
 24257.rpt.msg
 Nov-29-13 14:58:18 [Worker_1] Spam-Report: finished report-message 
 from m...@me.tld
 Nov-29-13 14:58:18 m1-33498-37618 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 [SMTP Reply] 250 2.0.0 Ok
 Nov-29-13 14:58:18 m1-33498-37618 [Worker_1] [TLS-in] [TLS-out]
 172.20.55.51 [SMTP Reply] 221 2.0.0 Bye
 Nov-29-13 14:58:18 [Worker_1] Info: report successful sent to 
 m...@me.tld

 this happens either if i set MaillogExt to .eml or .msg any suggestion?

 thanks.

 --
 Madness, like small fish, runs in hosts, in vast numbers of instances.

 Nessuno mi pettina bene come il vento.

 --
 --
 --
 Rapidly troubleshoot problems before they affect your business. Most 
 IT organizations don't have a clear picture of how application 
 performance affects their revenue

[Assp-test] Using ASSP with Office 365

[Colin Waring]

Hi,

 

We are in a position where we have some clients using Office 365 who wish
for better spam filtering. Currently we are not able to offer this due to
the way that Office 365 works. We can set Office 365 and ASSP up to
communicate with each other but there is one big problem.

 

Office 365 does not allow you to specify authentication details for the
outbound SMTP connector. This means that we have no way to secure mail from
Office 365. If we configure ASSP to accept email from the Office 365
platform it will accept mail from the entire platform, not just our hosted
account. Without authentication, the email interface will not work either.

 

I cannot seem to find any information on a workaround. The best I can find
is a provide called SMTPLogic who have a guide on how to set up their
services with Office 365, unfortunately anything useful is done in the back
end of their system however it does show that Office 365 authentication can
somehow be achieved.

 

So, does anyone know of a way to do this with the existing options? Is it
something that would require extra development of ASSP to support?

 

All the best,

Colin Waring.

 

--
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most 
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] assp/postfix 5.2.2 command not recognized

[Colin Waring]

Hi Andras,

I have looked back through my emails and confirmed that I saw the exact same 
behaviour which I reported on the list on 30/09/2013. I'm not sure what sparked 
the issue off, whether it was an ASSP update, a perl or perl module update or 
an OS update. Either way, disabling OCR resolved the issue for me.

I thought Thomas released an updated module that address the issue but I 
haven't turned it back on as all the requirements for OCR are a pain to install 
on Ubuntu. I can't risk turning it back on to see if the problem is still there 
at the moment, we've had a few too many issues lately (not specifically ASSP, 
other issues like datacentre outages etc).

All the best,
Colin Waring.

-Original Message-
From: Virag Andras [mailto:snowfl...@snowflake.hu] 
Sent: 05 November 2013 11:55
To: ASSP development mailing list
Subject: Re: [Assp-test] assp/postfix 5.2.2 command not recognized


 Hi,

 I can't say I understand the last passage, but the rest makes sense.
 Still, why would the clients from several hosts would start sending 
 unrecognized commands for 3 weeks now (after the assp upgrade),
 when they never did it in the past, I'm still perplexed.
 What I did after Colins suggestion, I did a check on the file versions 
 and it was consistent, but a couple of perl modules were outdated.
 I updated them, installed the latest assp, disabled OCR (somehow in 
 most of the faulty mails I could check, had image parts).
 So far so good, I don't see the error. Wish me luck.
 Thanks guys

 Andras


 On Tue, 5 Nov 2013 07:11:28 +0100, Thomas Eckardt 
 thomas.ecka...@thockar.com wrote:
Nov  4 10:45:47 mail.server/smtpd[18910]: match_string: 
 gIl2eY22EOu8Yx6
 ~? CONNECT
Nov  4 10:45:47 mail.server/smtpd[18910]: match_string: 
 gIl2eY22EOu8Yx6
 ~? GET
Nov  4 10:45:47 mail.server/smtpd[18910]: match_string: 
 gIl2eY22EOu8Yx6
 ~? POST

 This is HTTP stuff. ASSP never does this in SMTP workers - so the 
 commands
 are submitted from the client or server.
 Because assp is a proxy, these commands are processed transparent.

 The mail is delivered because assp saw
Nov  4 10:45:47 mail.server/smtpd[18910]: 
 localhost.localdomain[127.0.0.1]: 250 2.0.0 Ok: queued as 1C08E5303D9
 after that a QUIT, RSET, NOOP, or MAIL FROM is expected. How ever, 
 postfix
 answers with 502, which is right - assp counts the protocol mistakes 
 and
 drops the connection with
 Date 84504-01966 [Worker_2] [MaxErrors]

 There is nothing you can do - except if the connected host is one of 
 your
 locals.

 One possible cause for this behavior would be, if assp/Perl/system
 scrambles connection (socket) data between the threads. In this case,
 there must be a HTTP request at exact the same time via the GUI.

 Thomas




 Von:Andras Virag snowfl...@snowflake.hu
 An: 'ASSP development mailing list'
 assp-test@lists.sourceforge.net,
 Datum:  04.11.2013 21:13
 Betreff:Re: [Assp-test] assp/postfix 5.2.2 command not 
 recognized



 No, I really loved it. :) Especially that I was/am a support engineer
 myself, and Thomas was so right! I really was dumb.

 So:
 This is perl 5, version 14, subversion 1 (v5.14.1) built for
 x86_64-linux-thread-multi
 and ASSP version 2.3.4(13303)
 Yes, the kernel is old, but it's not that easy to upgrade now


 And, the error message from assp:
 Date 84504-01966 [Worker_2] IP from@email to: to@email [SMTP Error] 
 502
 5.5.2 Error: command not recognized
 Date 84504-01966 [Worker_2] [MaxErrors] 91.239.66.136
 ad...@vz11967.dahost.pl to: to@email max errors (MaxErrors=5) 
 exceeded
 -- dropping connection - after reply: 502 5.5.2 Error: command not
 recognized from 127.0.0.1

 Postfix:
 Nov  4 10:45:46 mail.server/smtpd [18910]: input attribute name: 
 (end)
 Nov  4 10:45:46 mail.server/smtpd [18910]: send attr flags = 178
 Nov  4 10:45:46 mail.server/smtpd [18910]: 1C08E5303D9:
 client=localhost.localdomain[127.0.0.1]
 Nov  4 10:45:46 mail.server/smtpd [18910]: 
 localhost.localdomain[127.0.0.1]: 250 2.1.5 Ok
 Nov  4 10:45:46 mail.server/smtpd [18910]: watchdog_pat: 
 0x7f7a3f5abcd0
 Nov  4 10:45:46 mail.server/smtpd [18910]: 
 localhost.localdomain[127.0.0.1]: DATA
 Nov  4 10:45:46 mail.server/smtpd [18910]: match_string: DATA ~? NOOP
 Nov  4 10:45:46 mail.server/smtpd [18910]: match_list_match: DATA: no
 match
 Nov  4 10:45:46 mail.server/smtpd [18910]: 
 localhost.localdomain[127.0.0.1]: 354 End data with CRLF.CRLF
 Nov  4 10:45:47 mail.server/smtpd [18910]: public/cleanup socket: 
 wanted
 attribute: status
 Nov  4 10:45:47 mail.server/smtpd [18910]: input attribute name: 
 status
 Nov  4 10:45:47 mail.server/smtpd [18910]: input attribute value: 0
 Nov  4 10:45:47 mail.server/smtpd [18910]: public/cleanup socket: 
 wanted
 attribute: reason
 Nov  4 10:45:47 mail.server/smtpd [18910]: input attribute name: 
 reason
 Nov  4 10:45:47 mail.server/smtpd [18910]: input attribute value: 
 (end)
 Nov  4 10:45:47 mail.server/smtpd [18910]: public/cleanup socket: 
 wanted
 attribute

Re: [Assp-test] assp/postfix 5.2.2 command not recognized

[Colin Waring]

I dropped mine getting it out of the box unfortunately. My best guess is
that they may not be running the latest version. We did have this issue
earlier in the year but it was fixed a while back.


-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 04 November 2013 16:09
To: ASSP development mailing list
Subject: Re: [Assp-test] assp/postfix 5.2.2 command not recognized

Hu ... I'm a clairvoyant, I take my crystal ball and I'll tell you what
I can see.

postfix - seems to be a linux system - ...  too much fog in the
ball, can't see the distro I try to see some more, but it is hard today 
it's not my day ... what that ? showing something blurred .. 
assp - I can see something with Perl - a sh. fog again - wait a
moment ... no chance, can't see the Perl or assp version

Oh, I give up!

Is there anyone here who has a new crystal ball for sale? My good old one
seems to be damaged, it is every time too foggy - I'm unable to find any
usefull information in such posts anymore.

Thomas 





Von:Virag Andras snowfl...@snowflake.hu
An: assp-test@lists.sourceforge.net, 
Datum:  04.11.2013 16:10
Betreff:[Assp-test] assp/postfix 5.2.2 command not recognized




 Hi,

 I am experiencing a 502 5.5.2 problem recently from some connections  for
a couple of versions now.

 The error message is:
 smtpserver.ip.address failed after I sent the message.
 Remote host said: 502 5.5.2 Error: command not recognized

 What I see in the logs, that after the Message is OK and delivered, it
triggers something on postfix which answers the previous error message
after warning 5 times.
 Sometimes the message gets thru to the recipient even though the sender
receives this bounce.

 Can you advise where and what to check? I had no such problems before.

 Andras




--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***




--
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: local but not valid recipient removed from mail header

[Colin Waring]

Have a look at the archives of the list.

I raised the same issue though it is interesting to see someone else with it
- it means that some update has either revealed a flaw in our configurations
or put together a combination of circumstances that make it not work

Make sure you are running the latest version and unset DoHeaderAddrCheck

All the best,
Colin Waring.

-Original Message-
From: Grille [mailto:c.vielha...@me.com] 
Sent: 27 October 2013 20:27
To: assp-test@lists.sourceforge.net
Subject: Re: [Assp-test] Antwort: local but not valid recipient removed
from mail header

Hi,

since two weeks i receive a lot of forwarded messages from our empolyees
where the TO-line in Header is corrupted.

Mail addresses from not-local recipients are listed, but local recipients
are removed by assp.

I think it's because ASSP, because i have not updated any packages of perl
end the rest of the server since mid august 13 and asap updates itself.

So i looked up mails in not spam folder, but the messages seems to be okay
there, before they get the X-ASSP header tags.


I looked up the maillot.txt and saw same message as subject of this post.
Oct-27-13 20:36:40 m1-02600-01263 [Worker_2] [TLS-out] 157.55.1.152
jane@outlook.com to: jane-...@domain.com TO: - local but not valid
recipient jane-...@domain.com removed from mail he


So mails in MailClient look like:   TO: f...@bar.com, ' , jane Doe 



I analyze all mails i received by our employees and figured out, all mails
have the TNEF Header Tag.
/I tried sending myself and a lot of other recipient a mail from iCloud.com
and a mail from outlook.com with the following result.
mail from iCloud - delivered to all recipients - *NOT CORRUPTED*
TO-Header-Tag mail from outlook - delivered to all recipients - *CORRUPTED
*TO-Header-Tag /


*I could fix it by disabling DoHeaderAddrCheck*


Is there a way to enable it again and solve the problem anyway ?
Because HeaderChecks are nice to have :-)

Regards,
Chris




--
View this message in context:
http://anti-spam-smtp-proxy-server.996265.n3.nabble.com/local-but-not-valid-
recipient-removed-from-mail-header-tp37621p37681.html
Sent from the assp-test mailing list archive at Nabble.com.


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from the latest Intel processors and coprocessors. See abstracts and
register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] free spamming domain wanted

[Colin Waring]

Hi Thomas,

I have influ.co.uk that is an old domain I moved away from because of
spammers. The only reason I haven't let it expire is nostalgia and I use it
for my spam corpus. I can point the MX to you, not sure what volumes of mail
go to it these days as I haven't checked the stats for a long time.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 26 October 2013 07:05
To: ASSP development mailing list
Subject: Re: [Assp-test] free spamming domain wanted

Thank you Peter, but what I'm looking for is a still existing but unused
(MX) domain (or subdomain) with a high amount of spam.

Thomas





Von:Peter Hinman peter.hin...@myib.com
An: ASSP development mailing list assp-test@lists.sourceforge.net, 
Datum:  25.10.2013 19:21
Betreff:Re: [Assp-test] free spamming domain wanted



Would a sub-domain work for you? Could  you send to something like
s...@onlyspam.tld.com?

How long would you need this?

Peter Hinman
International Bridge / ParcelPool.com

On 10/24/2013 9:18 AM, Thomas Eckardt wrote:
 Hi all,

 I need to increase the spam-workload on my prod system to make some 
 strange tests. I need a free (unused ) domain (only the MX), which 
 receives always SPAM.

 Thank you.

 Thomas



 DISCLAIMER:
 ***
 This email and any files transmitted with it may be confidential,
legally
 privileged and protected in law and are intended solely for the use of
the

 individual to whom it is addressed.
 This email was multiple times scanned for viruses. There should be no 
 known virus in this email!
 ***




 

--
 October Webinars: Code for Performance Free Intel webinars can help 
 you accelerate application performance.
 Explore tips for MPI, OpenMP, advanced profiling, and more. Get the 
 most
from
 the latest Intel processors and coprocessors. See abstracts and 
 register

 
http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk



 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from the latest Intel processors and coprocessors. See abstracts and
register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***




--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135991iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: Re: Antwort: Re: fixes in assp 2.3.4 build 13294

[Colin Waring]

Thanks for the reply Thomas,

I have that set to DB: and the table in the database is completely empty so
that does not look to be the cause.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 25 October 2013 06:09
To: ASSP development mailing list
Subject: [Assp-test] Antwort: Re: Antwort: Re: fixes in assp 2.3.4 build
13294

Collin,

check your 'ldaplistdb' - I saw case, foreign domain were listed there -
remove them!
I don't know how this could happen - I'm still working on this problem.

Thomas 





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  24.10.2013 20:08
Betreff:Re: [Assp-test] Antwort: Re:  fixes in assp 2.3.4 build 
13294



Maybe, however it does not explain why I haven't had any issues at all 
with
it until recently.

All I have set under Recipients/Local Domains is removeForeignBCC,
DoRFC822 (both), localDomains, VRFYforceRCPTTO (MTA IP:port), DisableVRFY
and some of the max dupes settings. 

As far as I can tell there is no reason why ASSP should need to know
anything but the domain names in order to achieve those checks. 

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 23 October 2013 08:39
To: ASSP development mailing list
Subject: [Assp-test] Antwort: Re: fixes in assp 2.3.4 build 13294

Most of my recipient validation is handled by the MTA which calls 
forward
to
check for valid addresses instead of managing its own list however
localdomains is used.

'Most of' - this explains why you've been running in to problems with this
feature. An header address check is only usefull, if assp is able to 
verify
ALL addresses!

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  22.10.2013 10:39
Betreff:Re: [Assp-test] fixes in assp 2.3.4 build 13294



Thanks,

Most of my recipient validation is handled by the MTA which calls forward 
to
check for valid addresses instead of managing its own list however
localdomains is used.

I have gone back through my logs, it is worth noting that I did not have 
any
problems with the check prior to installing ASSP version 2.3.4(13282). I
started that version of ASSP at 2013-10-09 18:19:27 and experienced the
first log event within an hour at 2013-10-09 19:00:52.

From memory, I think I ran mod_inst.pl at the same time so it is more 
likely
to be an updated perl module causing it than the code changes as I was
running 13276 for some time without the issue and the code changes were 
only
minor.

Any suggestions on how to track down which bit of the system is throwing 
the
check off?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 22 October 2013 06:18
To: ASSP development mailing list
Subject: Re: [Assp-test] fixes in assp 2.3.4 build 13294

Will ASSP no longer remove
foreign BCCs?

Assp will remove foreign BCCs if configured to do so.

DoHeaderAddrCheck = 0 will disable all other to,cc and bcc checks (not
local, not valid, spamtrap, recipient replacement).

It is not possible to see local but not valid if DoHeaderAddrCheck is 
set
to zero.

temporary fix pending investigation into the
root cause

This feature works like it is designed - if it does'nt work for you, 
disable
it. It is designed to be very very strict on any of the header addresses. 
It
is NOT a RFC mistake to submit foreign addresses in this header lines 
except
BCC. So this strictness could lead in to false positives.
I never had any issue with it.

I'm thinking about to set the default to zero, if it is integrated in to 
the
GUI.

Thomas





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  21.10.2013 12:32
Betreff:Re: [Assp-test] fixes in assp 2.3.4 build 13294



Hi Thomas,

I have put this in place and the errors I had are no longer being logged.

To give you an idea of the scale of the problem, I ran the following:

root@mail:/usr/local/assp# grep local but not valid maillog.txt |wc -l
2022

2022 lines with the error were detected between the logfile being rolled 
at
midnight and me updating at 11:10!

What checks get disabled by this new setting? Will ASSP no longer remove
foreign BCCs? If so does it reject or pass the messages with foreign
addresses and is this just a temporary fix pending investigation into the
root cause?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 21 October 2013 05:50
To: ASSP development mailing list
Subject: [Assp-test] fixes in assp 2.3.4 build 13294

Hi all,

fixed in assp 2.3.4 build 13294:

changed:

- it is now possible to disable the header address check for to,cc and bcc
addresses

our $DoHeaderAddrCheck = 1;  # (0

Re: [Assp-test] Antwort: Re: fixes in assp 2.3.4 build 13294

[Colin Waring]

Maybe, however it does not explain why I haven't had any issues at all with
it until recently.

All I have set under Recipients/Local Domains is removeForeignBCC,
DoRFC822 (both), localDomains, VRFYforceRCPTTO (MTA IP:port), DisableVRFY
and some of the max dupes settings. 

As far as I can tell there is no reason why ASSP should need to know
anything but the domain names in order to achieve those checks. 

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 23 October 2013 08:39
To: ASSP development mailing list
Subject: [Assp-test] Antwort: Re: fixes in assp 2.3.4 build 13294

Most of my recipient validation is handled by the MTA which calls 
forward
to
check for valid addresses instead of managing its own list however
localdomains is used.

'Most of' - this explains why you've been running in to problems with this
feature. An header address check is only usefull, if assp is able to verify
ALL addresses!

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  22.10.2013 10:39
Betreff:Re: [Assp-test] fixes in assp 2.3.4 build 13294



Thanks,

Most of my recipient validation is handled by the MTA which calls forward 
to
check for valid addresses instead of managing its own list however
localdomains is used.

I have gone back through my logs, it is worth noting that I did not have 
any
problems with the check prior to installing ASSP version 2.3.4(13282). I
started that version of ASSP at 2013-10-09 18:19:27 and experienced the
first log event within an hour at 2013-10-09 19:00:52.

From memory, I think I ran mod_inst.pl at the same time so it is more 
likely
to be an updated perl module causing it than the code changes as I was
running 13276 for some time without the issue and the code changes were 
only
minor.

Any suggestions on how to track down which bit of the system is throwing 
the
check off?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 22 October 2013 06:18
To: ASSP development mailing list
Subject: Re: [Assp-test] fixes in assp 2.3.4 build 13294

Will ASSP no longer remove
foreign BCCs?

Assp will remove foreign BCCs if configured to do so.

DoHeaderAddrCheck = 0 will disable all other to,cc and bcc checks (not
local, not valid, spamtrap, recipient replacement).

It is not possible to see local but not valid if DoHeaderAddrCheck is 
set
to zero.

temporary fix pending investigation into the
root cause

This feature works like it is designed - if it does'nt work for you, 
disable
it. It is designed to be very very strict on any of the header addresses. 
It
is NOT a RFC mistake to submit foreign addresses in this header lines 
except
BCC. So this strictness could lead in to false positives.
I never had any issue with it.

I'm thinking about to set the default to zero, if it is integrated in to 
the
GUI.

Thomas





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  21.10.2013 12:32
Betreff:Re: [Assp-test] fixes in assp 2.3.4 build 13294



Hi Thomas,

I have put this in place and the errors I had are no longer being logged.

To give you an idea of the scale of the problem, I ran the following:

root@mail:/usr/local/assp# grep local but not valid maillog.txt |wc -l
2022

2022 lines with the error were detected between the logfile being rolled 
at
midnight and me updating at 11:10!

What checks get disabled by this new setting? Will ASSP no longer remove
foreign BCCs? If so does it reject or pass the messages with foreign
addresses and is this just a temporary fix pending investigation into the
root cause?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 21 October 2013 05:50
To: ASSP development mailing list
Subject: [Assp-test] fixes in assp 2.3.4 build 13294

Hi all,

fixed in assp 2.3.4 build 13294:

changed:

- it is now possible to disable the header address check for to,cc and bcc
addresses

our $DoHeaderAddrCheck = 1;  # (0/1) do the header address 
check for TO,CC,BCC

The default value for this is '1' - to disable the check add the following
line to 'lib/CorrectASSPcfg.pm'
and restart assp.

$main::DoHeaderAddrCheck = 0;


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 



individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known
virus in this email!
***





--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application

Re: [Assp-test] fixes in assp 2.3.4 build 13294

[Colin Waring]

Thanks,

Most of my recipient validation is handled by the MTA which calls forward to
check for valid addresses instead of managing its own list however
localdomains is used.

I have gone back through my logs, it is worth noting that I did not have any
problems with the check prior to installing ASSP version 2.3.4(13282). I
started that version of ASSP at 2013-10-09 18:19:27 and experienced the
first log event within an hour at 2013-10-09 19:00:52.

From memory, I think I ran mod_inst.pl at the same time so it is more likely
to be an updated perl module causing it than the code changes as I was
running 13276 for some time without the issue and the code changes were only
minor.

Any suggestions on how to track down which bit of the system is throwing the
check off?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 22 October 2013 06:18
To: ASSP development mailing list
Subject: Re: [Assp-test] fixes in assp 2.3.4 build 13294

Will ASSP no longer remove
foreign BCCs?

Assp will remove foreign BCCs if configured to do so.

DoHeaderAddrCheck = 0 will disable all other to,cc and bcc checks (not
local, not valid, spamtrap, recipient replacement).

It is not possible to see local but not valid if DoHeaderAddrCheck is set
to zero.

temporary fix pending investigation into the
root cause

This feature works like it is designed - if it does'nt work for you, disable
it. It is designed to be very very strict on any of the header addresses. It
is NOT a RFC mistake to submit foreign addresses in this header lines except
BCC. So this strictness could lead in to false positives.
I never had any issue with it.

I'm thinking about to set the default to zero, if it is integrated in to the
GUI.

Thomas





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  21.10.2013 12:32
Betreff:Re: [Assp-test] fixes in assp 2.3.4 build 13294



Hi Thomas,

I have put this in place and the errors I had are no longer being logged.

To give you an idea of the scale of the problem, I ran the following:

root@mail:/usr/local/assp# grep local but not valid maillog.txt |wc -l
2022

2022 lines with the error were detected between the logfile being rolled 
at
midnight and me updating at 11:10!

What checks get disabled by this new setting? Will ASSP no longer remove
foreign BCCs? If so does it reject or pass the messages with foreign
addresses and is this just a temporary fix pending investigation into the
root cause?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 21 October 2013 05:50
To: ASSP development mailing list
Subject: [Assp-test] fixes in assp 2.3.4 build 13294

Hi all,

fixed in assp 2.3.4 build 13294:

changed:

- it is now possible to disable the header address check for to,cc and bcc
addresses

our $DoHeaderAddrCheck = 1;  # (0/1) do the header address 
check for TO,CC,BCC

The default value for this is '1' - to disable the check add the following
line to 'lib/CorrectASSPcfg.pm'
and restart assp.

$main::DoHeaderAddrCheck = 0;


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 


individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known
virus in this email!
***





--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most 
from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135031iu=/4140/ostg.clktrk

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk

Re: [Assp-test] fixes in assp 2.3.4 build 13294

[Colin Waring]

Hi Thomas,

I have put this in place and the errors I had are no longer being logged.

To give you an idea of the scale of the problem, I ran the following:

root@mail:/usr/local/assp# grep local but not valid maillog.txt |wc -l
2022

2022 lines with the error were detected between the logfile being rolled at
midnight and me updating at 11:10!

What checks get disabled by this new setting? Will ASSP no longer remove
foreign BCCs? If so does it reject or pass the messages with foreign
addresses and is this just a temporary fix pending investigation into the
root cause?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 21 October 2013 05:50
To: ASSP development mailing list
Subject: [Assp-test] fixes in assp 2.3.4 build 13294

Hi all,

fixed in assp 2.3.4 build 13294:

changed:

- it is now possible to disable the header address check for to,cc and bcc
addresses

our $DoHeaderAddrCheck = 1;  # (0/1) do the header address 
check for TO,CC,BCC

The default value for this is '1' - to disable the check add the following
line to 'lib/CorrectASSPcfg.pm'
and restart assp.

$main::DoHeaderAddrCheck = 0;


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***




--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135031iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: local but not valid recipient removed from mail header

[Colin Waring]

Hi Thomas,

Further to this, we are now seeing more reports of problems as a result of
this.

The messages get through but have headers missing so things like Outlook
rules to not work.

I am also getting reports from at least one other client so it is not sender
specific and not recipient specific.

Anything I can do to help with this one as it is quickly becoming a big
issue?

All the best,
Colin Waring.

-Original Message-
From: Colin Waring [mailto:co...@lanternhosting.co.uk] 
Sent: 17 October 2013 08:34
To: 'ASSP development mailing list'
Subject: Re: [Assp-test] Antwort: local but not valid recipient removed
from mail header

Hi Thomas,

I have confirmed with the sender that this is not the case. The first
message he sent had an internal CC address on it for his network. Subsequent
messages have also shown exactly the same error when all he did was click
forward and put in the valid recipient once.

He has also emailed me and I can confirm the exact same behaviour on
messages to me, except it only reports the error once.

Sorry to keep finding these oddities for you!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 17 October 2013 07:25
To: ASSP development mailing list
Subject: [Assp-test] Antwort: local but not valid recipient removed from
mail header

ASSP checks the addresses in the TO:, CC:, and BCC: header fields - in your
case TO: was defined two times with an local domain but no valid local
recipient.

Thomas





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  16.10.2013 18:43
Betreff:[Assp-test] local but not valid recipient removed from 
mail header



Howdy,

 

I've had someone having trouble emailing one of the accounts that we host
and the logs had something I have never seen before.

 

 

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld TO: - local but not valid
recipient recipi...@domain.tld removed from mail header

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld Message-Score: added 10
(irValencePB) for InvalidAddress, total score for this message is now 10

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld TO: - local but not valid
recipient recipi...@domain.tld removed from mail header

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld Message-Score: added 10
(irValencePB) for InvalidAddress, total score for this message is now 20

 

First thing, the scoring was done twice.

 

Most importantly, there was only one recipient on the email and it is a
perfectly valid recipient that has been sending/receiving emails throughout
the day. Emails from this particular sender always seem to be rejected.

 

If I put the recipient email address into the following link, it
successfully validates it:

 

http://mythic-beasts.com/~pdw/cgi-bin/emailvalidate

 

Any ideas what the problem may be?

 

All the best,

Colin Waring.


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from the latest Intel processors and coprocessors. See abstracts and
register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135031iu=/4140/ostg.clktrk

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***





--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135031iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling

Re: [Assp-test] local but not valid recipient removed from mail header

[Colin Waring]

Hi Thomas,

The invalid PTR seems to be a DNS issue. Although it looks up fine and has
matching forward/reverse DNS on my home connection, it does not have any
reverse DNS when my hosts nameservers or Google's are queried. I only
checked my connection last night which threw me a red herring.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 17 October 2013 07:16
To: ASSP development mailing list
Subject: Re: [Assp-test] local but not valid recipient removed from mail
header

Does the invalid PTR check simply check for existing and valid PTR

it checks the existance of a PTR and if one is found, it is checked against
the regular expressions.
The HELO is not used for the PTR check.

Thomas





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  16.10.2013 18:51
Betreff:Re: [Assp-test] local but not valid recipient removed 
from mail   header



Further to this, I was about to email the sender back saying they had a
network misconfiguration. The message was also having an invalid PTR yet
when I do an nslookup the PTR is set and has matching forward and reverse
DNS.

Does the invalid PTR check simply check for existing and valid PTR or does
it compare it to the HELO? In this case the HELO is set and has valid
matching forward DNS. The reverse DNS is for a hostname set up by their
datacentre that does not match the HELO.



-Original Message-
From: Colin Waring [mailto:co...@lanternhosting.co.uk] 
Sent: 16 October 2013 17:42
To: 'ASSP development mailing list'
Subject: [Assp-test] local but not valid recipient removed from mail
header

Howdy,

 

I've had someone having trouble emailing one of the accounts that we host
and the logs had something I have never seen before.

 

 

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld TO: - local but not valid
recipient recipi...@domain.tld removed from mail header

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld Message-Score: added 10
(irValencePB) for InvalidAddress, total score for this message is now 10

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld TO: - local but not valid
recipient recipi...@domain.tld removed from mail header

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld Message-Score: added 10
(irValencePB) for InvalidAddress, total score for this message is now 20

 

First thing, the scoring was done twice.

 

Most importantly, there was only one recipient on the email and it is a
perfectly valid recipient that has been sending/receiving emails 
throughout
the day. Emails from this particular sender always seem to be rejected.

 

If I put the recipient email address into the following link, it
successfully validates it:

 

http://mythic-beasts.com/~pdw/cgi-bin/emailvalidate

 

Any ideas what the problem may be?

 

All the best,

Colin Waring.


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from the latest Intel processors and coprocessors. See abstracts and
register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135031iu=/4140/ostg.clktrk

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test



--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most 
from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135031iu=/4140/ostg.clktrk

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from

Re: [Assp-test] Antwort: local but not valid recipient removed from mail header

[Colin Waring]

Hi Thomas,

I have confirmed with the sender that this is not the case. The first
message he sent had an internal CC address on it for his network. Subsequent
messages have also shown exactly the same error when all he did was click
forward and put in the valid recipient once.

He has also emailed me and I can confirm the exact same behaviour on
messages to me, except it only reports the error once.

Sorry to keep finding these oddities for you!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 17 October 2013 07:25
To: ASSP development mailing list
Subject: [Assp-test] Antwort: local but not valid recipient removed from
mail header

ASSP checks the addresses in the TO:, CC:, and BCC: header fields - in your
case TO: was defined two times with an local domain but no valid local
recipient.

Thomas





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net,
Datum:  16.10.2013 18:43
Betreff:[Assp-test] local but not valid recipient removed from 
mail header



Howdy,

 

I've had someone having trouble emailing one of the accounts that we host
and the logs had something I have never seen before.

 

 

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld TO: - local but not valid
recipient recipi...@domain.tld removed from mail header

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld Message-Score: added 10
(irValencePB) for InvalidAddress, total score for this message is now 10

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld TO: - local but not valid
recipient recipi...@domain.tld removed from mail header

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld Message-Score: added 10
(irValencePB) for InvalidAddress, total score for this message is now 20

 

First thing, the scoring was done twice.

 

Most importantly, there was only one recipient on the email and it is a
perfectly valid recipient that has been sending/receiving emails 
throughout
the day. Emails from this particular sender always seem to be rejected.

 

If I put the recipient email address into the following link, it
successfully validates it:

 

http://mythic-beasts.com/~pdw/cgi-bin/emailvalidate

 

Any ideas what the problem may be?

 

All the best,

Colin Waring.


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most 
from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135031iu=/4140/ostg.clktrk

___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135031iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] local but not valid recipient removed from mail header

[Colin Waring]

Howdy,

 

I've had someone having trouble emailing one of the accounts that we host
and the logs had something I have never seen before.

 

 

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld TO: - local but not valid
recipient recipi...@domain.tld removed from mail header

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld Message-Score: added 10
(irValencePB) for InvalidAddress, total score for this message is now 10

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld TO: - local but not valid
recipient recipi...@domain.tld removed from mail header

2013-10-16 16:57:29 m1-39048-10113 [Worker_1] [TLS-out] 1.1.1.1
sen...@domain.tld to: recipi...@domain.tld Message-Score: added 10
(irValencePB) for InvalidAddress, total score for this message is now 20

 

First thing, the scoring was done twice.

 

Most importantly, there was only one recipient on the email and it is a
perfectly valid recipient that has been sending/receiving emails throughout
the day. Emails from this particular sender always seem to be rejected.

 

If I put the recipient email address into the following link, it
successfully validates it:

 

http://mythic-beasts.com/~pdw/cgi-bin/emailvalidate

 

Any ideas what the problem may be?

 

All the best,

Colin Waring.

--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60135031iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.3.4 build 13284

[Colin Waring]

I updated (using a script that pulls from sourceforge) and my servers are
showing 13284 as the build number.

Is it worth re-downloading Steve and seeing if that corrects it?

-Original Message-
From: Steve Moffat [mailto:st...@optimum.bm] 
Sent: 11 October 2013 14:26
To: ASSP development mailing list
Subject: Re: [Assp-test] fixes in assp 2.3.4 build 13284

Thomas

The assp.pl file that is purported to be assp 2.3.4 build 13284 is actually
a copy of assp 2.3.4 build 13283

Steve

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: Friday, October 11, 2013 4:26 AM
To: ASSP development mailing list
Subject: [Assp-test] fixes in assp 2.3.4 build 13284

Hi all,

fixed in assp 2.3.4 build 13284:

- removeForeignBCC was working only for one mail per worker

- NpWlTimeOut was not used for whitelisted mails, only for noprocessing

- graphical stats were not shown in all browsers because of an HTML-tag typo



I've includes some enhanced debug code to get rid of the
SenderBase-Stck-Workers problem.

Thomas



DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***




--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60134071iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test


--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60134071iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] ForeignBCC

[Colin Waring]

Hi,

 

One of my accounts is seeing messages blocked, reason relay attempt blocked
for non local BCC: recipient

 

I have removeForeignBCC turned on. I thought that having this on would allow
the message through to the local recipient and strip out anything none local
rather than block the message.

 

Have I misunderstood something?

 

All the best,

Colin Waring.

--
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register 
http://pubads.g.doubleclick.net/gampad/clk?id=60134071iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test