[Assp-test] Seg Fault

[Colin Waring]

For the past couple of weeks ASSP has been dying around the same time, though 
not consistently every day.

The logs haven't shown anything - they just stop. I changed ASSP to not run as 
a daemon and this weekend have caught this error on the command line.

2018-08-18 08:30:07 [Worker_1] Warning: got unexpected signal SEGV in 
Worker_1: package - Net::SMTP, file - sub Net::SMTP::DESTROY_SSLNS, line - 
10!
Segmentation fault

Perl is v5.22.1
Net::SMTP is up to date with CPAN:

/usr/local/share/perl/5.22.1/Net/SMTP.pm
Installed: 3.11
CPAN:  3.11  up to date

The most recent log for Worker_1 is almost 30 minutes prior and related to 
AFC being called. The lines immediately before related to MaxAUTHErrors:

2018-08-18 08:30:03 m1-77402-01879 [Worker_5] [MaxAUTHErrors] 181.214.206.111 
too many (26) AUTH errors from 181.214.206.0
2018-08-18 08:30:03 m1-77402-01879 [Worker_5] 181.214.206.111 Message-Score: 
added 60 (autValencePB) for AUTHErrors, total score for this message is now 60
2018-08-18 08:30:03 m1-77402-01879 [Worker_5] 181.214.206.111 info: start 
damping on closing connection (12)

The only thing possibly consistent with the timing is that block reports are 
run at 8am, though they have been for a long time. I don't know of anything 
else recurring even close to that time.

Any suggestions?
All the best,
Colin.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Log entries for resend

[Colin Waring]

I've noticed the log entries are inaccurate and it logs as "successful sent" 
even if it is not successful.

This causes log searches to be inaccurate as searching for "resend" doesn't 
bring up the error.

2018-07-26 09:30:13 [Worker_1] Error: can't open requested file .eml in 
any collection folder
2018-07-26 09:30:16 [Worker_1] Info: successful sent file 
/usr/local/assp/resendmail/.eml to 1.1.1.1:1 (smtpDestination)

Can the second line be updated so that it states the send failed?

Also less important, the correct grammar should be "successfully sent".

Thanks,
Colin.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Can't use an undefined value as a subroutine reference at sub main::ThreadMaintMain2 line 63

[Colin Waring]

Thanks, that’s firing up now and I’ll see what happens next time I catch the 
error.

All the best,
Colin.

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 16 May 2018 07:18
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Can't use an undefined value as a subroutine reference 
at sub main::ThreadMaintMain2 line 63

Colin,

I'm unable to reproduce this behavior.

I've uploaded a modified version (18136) to the test folder in SVN.
This version will tell us what happens.

Thomas





Von:    "Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:15.05.2018 10:10
Betreff:[Assp-test] Can't use an undefined value as a subroutine 
reference at sub main::ThreadMaintMain2 line 63



Hi,



I’ve caught this today:



2018-05-15 08:03:31 [Main_Thread] Saving config

2018-05-15 08:03:31 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged

2018-05-15 08:03:31 [Main_Thread] Adminupdate: file 
'/usr/local/assp/files/blockreportuser.txt' for config 'BlockReportFile' was 
changed

2018-05-15 08:03:32 [Main_Thread] Info: added schedule : BlockReport - for : 
*@domain.tld=>*=>1<mailto:*@domain.tld=%3e*=%3e1>=> - at : 0 0,4,8,12,16,20 * * 
* - next run is at : 2018-05-15 12:00:00

2018-05-15 08:03:32 [Worker_1] Info: notification message queued to sent to 
monitoraddr...@ourdomain.tld<mailto:monitoraddr...@ourdomain.tld>

2018-05-15 08:03:32 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.

2018-05-15 08:03:32 [Main_Thread] SyncCFG: start synchronization of 
BlockReportFile

2018-05-15 08:03:32 [Worker_1] Info: auto restart died worker Worker_1

2018-05-15 08:03:32 [Worker_1] Info: cleaned command 'syncConfigSend' from 
commandqueue

2018-05-15 08:04:11 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!

2018-05-15 08:04:56 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!



There then seems to be no traffic until 08:05:00 (approx. 90s)



It is highly unlikely that BlockReportFile was changed at this time. The line 
from BlockReportFile that is quoted used to work but I can see it is now 
missing the “# next run” so I’m suspecting the 4 hour schedule is the issue 
here. I know it used to be right because I questioned whether the number of 
days could be less than 1 when it was initially set up.



Even more odd is that I don’t get this error every four hours – the last time 
it happened was on the 10th so there must be more to it than the entry in the 
file:



2018-05-10 08:02:46 [Main_Thread] Saving config

2018-05-10 08:02:46 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged

2018-05-10 08:02:46 [Main_Thread] Adminupdate: file 
'/usr/local/assp/files/blockreportuser.txt' for config 'BlockReportFile' was 
changed

2018-05-10 08:02:47 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.

2018-05-10 08:02:47 [Worker_1] Info: notification message queued to sent to 
support.dolphinict.co...@email.uk.autotask.net<mailto:support.dolphinict.co...@email.uk.autotask.net>

2018-05-10 08:02:47 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.

2018-05-10 08:02:47 [Main_Thread] Info: added schedule : BlockReport - for : 
*@domain.tld<mailto:*@domain.tld> =>*=>1=> - at : 0 0,4,8,12,16,20 * * * - next 
run is at : 2018-05-10 12:00:00

2018-05-10 08:02:47 [Worker_1] Info: auto restart died worker Worker_1

2018-05-10 08:02:47 [Main_Thread] SyncCFG: start synchronization of 
BlockReportFile

2018-05-10 08:03:04 [Worker_10001] SyncCFG: request to synchronize 
BlockReportFile

2018-05-10 08:03:13 [Worker_10001] SyncCFG: successfully sent config for 
BlockReportFile to 10.0.5.219:25

2018-05-10 08:03:46 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!



Traffic didn’t stop that time.

I’m not sure it’s a significant problem, but it’s an error nonetheless.

All the best,

Colin.



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>
https://lists.sourceforge.net/lists/list

[Assp-test] Can't use an undefined value as a subroutine reference at sub main::ThreadMaintMain2 line 63

[Colin Waring]

Hi,

I've caught this today:

2018-05-15 08:03:31 [Main_Thread] Saving config
2018-05-15 08:03:31 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged
2018-05-15 08:03:31 [Main_Thread] Adminupdate: file 
'/usr/local/assp/files/blockreportuser.txt' for config 'BlockReportFile' was 
changed
2018-05-15 08:03:32 [Main_Thread] Info: added schedule : BlockReport - for : 
*@domain.tld=>*=>1=> - at : 0 0,4,8,12,16,20 * * * - next run is at : 
2018-05-15 12:00:00
2018-05-15 08:03:32 [Worker_1] Info: notification message queued to sent to 
monitoraddr...@ourdomain.tld
2018-05-15 08:03:32 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.
2018-05-15 08:03:32 [Main_Thread] SyncCFG: start synchronization of 
BlockReportFile
2018-05-15 08:03:32 [Worker_1] Info: auto restart died worker Worker_1
2018-05-15 08:03:32 [Worker_1] Info: cleaned command 'syncConfigSend' from 
commandqueue
2018-05-15 08:04:11 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!
2018-05-15 08:04:56 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!

There then seems to be no traffic until 08:05:00 (approx. 90s)

It is highly unlikely that BlockReportFile was changed at this time. The line 
from BlockReportFile that is quoted used to work but I can see it is now 
missing the "# next run" so I'm suspecting the 4 hour schedule is the issue 
here. I know it used to be right because I questioned whether the number of 
days could be less than 1 when it was initially set up.

Even more odd is that I don't get this error every four hours - the last time 
it happened was on the 10th so there must be more to it than the entry in the 
file:

2018-05-10 08:02:46 [Main_Thread] Saving config
2018-05-10 08:02:46 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged
2018-05-10 08:02:46 [Main_Thread] Adminupdate: file 
'/usr/local/assp/files/blockreportuser.txt' for config 'BlockReportFile' was 
changed
2018-05-10 08:02:47 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.
2018-05-10 08:02:47 [Worker_1] Info: notification message queued to sent to 
support.dolphinict.co...@email.uk.autotask.net
2018-05-10 08:02:47 [Worker_1] Error: Worker_1: Can't use an undefined 
value as a subroutine reference at sub main::ThreadMaintMain2 line 63.
2018-05-10 08:02:47 [Main_Thread] Info: added schedule : BlockReport - for : 
*@domain.tld =>*=>1=> - at : 0 0,4,8,12,16,20 * * * - next run is at : 
2018-05-10 12:00:00
2018-05-10 08:02:47 [Worker_1] Info: auto restart died worker Worker_1
2018-05-10 08:02:47 [Main_Thread] SyncCFG: start synchronization of 
BlockReportFile
2018-05-10 08:03:04 [Worker_10001] SyncCFG: request to synchronize 
BlockReportFile
2018-05-10 08:03:13 [Worker_10001] SyncCFG: successfully sent config for 
BlockReportFile to 10.0.5.219:25
2018-05-10 08:03:46 [Main_Thread] Warning: Main_Thread is unable to transfer 
connection to any worker - try again!

Traffic didn't stop that time.
I'm not sure it's a significant problem, but it's an error nonetheless.
All the best,
Colin.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

So your domain is thockar.com therefore the forged domain would be 
thockar.com-1.me

My example domain was a .co.uk therefore the forged domain was 
example.co.uk-1.me

Whoever registered uk-1.me also registered com-1.me because the DNS records 
include CNAMEs that point to the uk-1.me

The only way I can see to catch this would be to have the sender/from/reply-to 
checked to see if the domain contains any line from the local domains file. If 
the entry appears anywhere other than at the end of the address then score. It 
would have to work only on the part after the @ because many mailing lists 
include sender addresses in the left hand side as a way of message tracking.

For my purposes, I cannot see any reason why any of my domains would appear in 
part in anyone else’s domains – however I can see cases where not everyone’s 
domains are unique enough so there would have to be an over-ride where specific 
domains could be excluded should that be necessary.

Hopefully I’ve managed to explain what’s in my head well enough?
All the best,
Colin.


From: Thomas Eckardt <thomas.ecka...@thockar.com>
Sent: 21 April 2018 10:20
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

Or would it be ''thockar.uk-1.me' ?

In either case - this is hard to catch. The bombHeaderRe may help, if there are 
only some local domains hosted.

Thomas





Von:"Thomas Eckardt" 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:21.04.2018 10:41
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains




Only to be clear - for my domain the domainname would be 'thockar.com.uk-1.me' 
- right?



Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:21.04.2018 09:51
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains



In this case, the actual domain on the reply-to header was uk-1.me – this 
exists and is registered. The domain has wildcard DNS so *.uk-1.me will return 
valid DNS records – both A and MX. I suspect that the domain has been 
registered for the express intention of sending these kinds of phishing emails 
so I’ve added *@*.uk-1.me<mailto:*@*.uk-1.me> to blackListedDomains but it 
wouldn’t take much for them to change domains.



As a result, the reply-to address of localdomain.co.uk-1.me appears valid to 
all checks. The only thing that could tell ASSP that this is a phishing address 
is that the hostname contains an entry from localdomains with a bit on the end.



It just so happens that this particular message also had multiple from headers 
– something that you have updated ASSP to be able to detect now. We will now 
catch any similar emails on that basis however it is still possible that such a 
phishing email would get past if it did not have multiple from headers.



As these kinds of emails tend to be targeted and manually crafted for high 
value amounts I would guess it won’t take long for a miscreant to figure that 
out with a few tests.



All the best,

Colin.



From: Thomas Eckardt 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
Sent: 21 April 2018 08:18
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



>None of the addresses are actually @domain.tld

I'm right ? The used domains never ends with a valid TLD - so the domains never 
exists? Or at least - they ends with a valid TLD, but domains not exists?

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:20.04.2018 21:49
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Yes there is so this particular message gets caught which is great.

There is no guarantee that all emails with the -1.me also have multiple from 
headers, also the -1.me can change but it is always -something.tld on the end.

All the best,

Colin.



From: Thomas Eckardt 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
Sent: 20 April 2018 17:54
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

In this case, the actual domain on the reply-to header was uk-1.me – this 
exists and is registered. The domain has wildcard DNS so *.uk-1.me will return 
valid DNS records – both A and MX. I suspect that the domain has been 
registered for the express intention of sending these kinds of phishing emails 
so I’ve added *@*.uk-1.me<mailto:*@*.uk-1.me> to blackListedDomains but it 
wouldn’t take much for them to change domains.

As a result, the reply-to address of localdomain.co.uk-1.me appears valid to 
all checks. The only thing that could tell ASSP that this is a phishing address 
is that the hostname contains an entry from localdomains with a bit on the end.

It just so happens that this particular message also had multiple from headers 
– something that you have updated ASSP to be able to detect now. We will now 
catch any similar emails on that basis however it is still possible that such a 
phishing email would get past if it did not have multiple from headers.

As these kinds of emails tend to be targeted and manually crafted for high 
value amounts I would guess it won’t take long for a miscreant to figure that 
out with a few tests.

All the best,
Colin.

From: Thomas Eckardt <thomas.ecka...@thockar.com>
Sent: 21 April 2018 08:18
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

>None of the addresses are actually @domain.tld

I'm right ? The used domains never ends with a valid TLD - so the domains never 
exists? Or at least - they ends with a valid TLD, but domains not exists?

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:20.04.2018 21:49
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains



Yes there is so this particular message gets caught which is great.

There is no guarantee that all emails with the -1.me also have multiple from 
headers, also the -1.me can change but it is always -something.tld on the end.

All the best,

Colin.



From: Thomas Eckardt 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
Sent: 20 April 2018 17:54
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



But there should be a scoring because of multiple Fom: and/or Sender: headers-

Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:20.04.2018 16:42
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Hi Thomas,



I’ve run the message through the analyser and although a great feature to have 
it is not going to catch these emails.



None of the addresses are actually @domain.tld



The Reply-to: is @domain.tld-1.me so the extra -1.me bypasses the spoofing 
check.



The DoNoFrom: option is catching the multiple from headers which is great.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 20 April 2018 15:24
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



Colin,

solved build 18107 the problem for you?


changed:
...
'DoNoSpoofing4From','Do NoSpoofing for from:'
'Do the NoSpoofing check also for header 'from:', 'sender:', 'reply-to:' and 
'errors-to:' addresses.

Thomas





Von:"cw" <colin.war...@gmail.com<mailto:colin.war...@gmail.com>>
An:"ASSP development mailing list" 
<Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>>
Datum:14.04.2018 09:47
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Hi Thomas,

Looks like a good feature. I'll have to double check the headers for this 
message. I think the domains in all three from headers actually exist but have 
no relation to the recipient.

As the smtp address & from headers are a legitimate but compromised account the 
only header that would fail a legitimate domain check would be the reply to 
header.

These are carefully crafted phishing emails that are targeted, I've seen them 
sent to many accounts departments pretending to be from company directors 
requesting bank payments of up to £10,000. Of course the accounts department 
goes straight to said direc

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

Yes there is so this particular message gets caught which is great.
There is no guarantee that all emails with the -1.me also have multiple from 
headers, also the -1.me can change but it is always -something.tld on the end.
All the best,
Colin.

From: Thomas Eckardt <thomas.ecka...@thockar.com>
Sent: 20 April 2018 17:54
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

But there should be a scoring because of multiple Fom: and/or Sender: headers-

Thomas





Von:    "Colin Waring" <co...@dolphinict.co.uk>
An:"ASSP development mailing list" <assp-test@lists.sourceforge.net>
Datum:20.04.2018 16:42
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains



Hi Thomas,



I’ve run the message through the analyser and although a great feature to have 
it is not going to catch these emails.



None of the addresses are actually @domain.tld



The Reply-to: is @domain.tld-1.me so the extra -1.me bypasses the spoofing 
check.



The DoNoFrom: option is catching the multiple from headers which is great.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 20 April 2018 15:24
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



Colin,

solved build 18107 the problem for you?


changed:
...
'DoNoSpoofing4From','Do NoSpoofing for from:'
 'Do the NoSpoofing check also for header 'from:', 'sender:', 'reply-to:' and 
'errors-to:' addresses.

Thomas





Von:"cw" <colin.war...@gmail.com<mailto:colin.war...@gmail.com>>
An:"ASSP development mailing list" 
<Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>>
Datum:14.04.2018 09:47
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Hi Thomas,

Looks like a good feature. I'll have to double check the headers for this 
message. I think the domains in all three from headers actually exist but have 
no relation to the recipient.

As the smtp address & from headers are a legitimate but compromised account the 
only header that would fail a legitimate domain check would be the reply to 
header.

These are carefully crafted phishing emails that are targeted, I've seen them 
sent to many accounts departments pretending to be from company directors 
requesting bank payments of up to £10,000. Of course the accounts department 
goes straight to said director who comes to us wanting to know why we aren't 
blocking them.

All the best,
Colin

On Sat, 14 Apr 2018, 08:26 Thomas Eckardt, 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>> wrote:
> I thought this would not be caught by nospoofing because that would only 
> match if the RHS ended in the entry from localdomains.

OK.

And what if the 'DoNoFrom' feature would work like this:

Check for Existing and Valid From: and Sender: Header Tag and Address (DoNoFrom)

If enabled, the MIME header is checked for valid From: and Sender: header tags.
This header check fails and faults are counted, if both headers (From: and 
Sender:) are missing - or if any of these headers contains not a valid email 
address - or if multiple of the same headers are found.
The scoring value nofromValencePB is added for each detected fault.


In your example:

Reply-to: Sender Name 
<n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>>

To: recipi...@recipientdomain.tld<mailto:recipi...@recipientdomain.tld>

From: Sender Name <f...@domain.tld<mailto:f...@domain.tld>>

From: Sender Name <f...@domain2.tld<mailto:f...@domain2.tld>>

From: Sender Name 
<actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld<mailto:actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld>>

'nofromValencePB' would be added two times - one time for each additionally 
From: header.




Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:13.04.2018 20:55
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains





Thank you for the reply Thomas,



Being able to include sender:, reply to: and errors-to: would be handy in my 
opinion



However, in this case the local domain was not in any of the from: fields 
whatsoever. By using 
n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>, this hits 
a stupid bug in Outlook where in some places it will only display 
n...@recipientdomain.

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

Hi Thomas,

I’ve run the message through the analyser and although a great feature to have 
it is not going to catch these emails.

None of the addresses are actually @domain.tld

The Reply-to: is @domain.tld-1.me so the extra -1.me bypasses the spoofing 
check.

The DoNoFrom: option is catching the multiple from headers which is great.

All the best,
Colin.

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 20 April 2018 15:24
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

Colin,

solved build 18107 the problem for you?


changed:
...
'DoNoSpoofing4From','Do NoSpoofing for from:'
  'Do the NoSpoofing check also for header 'from:', 'sender:', 'reply-to:' and 
'errors-to:' addresses.

Thomas





Von:"cw" <colin.war...@gmail.com<mailto:colin.war...@gmail.com>>
An:"ASSP development mailing list" 
<Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>>
Datum:14.04.2018 09:47
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains




Hi Thomas,

Looks like a good feature. I'll have to double check the headers for this 
message. I think the domains in all three from headers actually exist but have 
no relation to the recipient.

As the smtp address & from headers are a legitimate but compromised account the 
only header that would fail a legitimate domain check would be the reply to 
header.

These are carefully crafted phishing emails that are targeted, I've seen them 
sent to many accounts departments pretending to be from company directors 
requesting bank payments of up to £10,000. Of course the accounts department 
goes straight to said director who comes to us wanting to know why we aren't 
blocking them.

All the best,
Colin

On Sat, 14 Apr 2018, 08:26 Thomas Eckardt, 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>> wrote:
> I thought this would not be caught by nospoofing because that would only 
> match if the RHS ended in the entry from localdomains.

OK.

And what if the 'DoNoFrom' feature would work like this:

Check for Existing and Valid From: and Sender: Header Tag and Address (DoNoFrom)

If enabled, the MIME header is checked for valid From: and Sender: header tags.
This header check fails and faults are counted, if both headers (From: and 
Sender:) are missing - or if any of these headers contains not a valid email 
address - or if multiple of the same headers are found.
The scoring value nofromValencePB is added for each detected fault.


In your example:

Reply-to: Sender Name 
<n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>>

To: recipi...@recipientdomain.tld<mailto:recipi...@recipientdomain.tld>

From: Sender Name <f...@domain.tld<mailto:f...@domain.tld>>

From: Sender Name <f...@domain2.tld<mailto:f...@domain2.tld>>

From: Sender Name 
<actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld<mailto:actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld>>

'nofromValencePB' would be added two times - one time for each additionally 
From: header.




Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:13.04.2018 20:55
Betreff:Re: [Assp-test] Multiple From headers/regex based on 
localdomains



Thank you for the reply Thomas,



Being able to include sender:, reply to: and errors-to: would be handy in my 
opinion



However, in this case the local domain was not in any of the from: fields 
whatsoever. By using 
n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>, this hits 
a stupid bug in Outlook where in some places it will only display 
n...@recipientdomain.tld<mailto:n...@recipientdomain.tld>. The 
-1.me<http://1.me/> is completely fictional and varies from message to message. 
I thought this would not be caught by nospoofing because that would only match 
if the RHS ended in the entry from localdomains.



All the best,

Colin.



From: Thomas Eckardt 
<thomas.ecka...@thockar.com<mailto:thomas.ecka...@thockar.com>>
Sent: 13 April 2018 16:55
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains



Colin,

'DoNoSpoofing4From' should do it - but it is'nt. Only the first 'From:' address 
is currently checked and this will not prevent this attack.
But it is possible to include 'sender: , reply-to: and errors-to:' in to this 
check - which would catch this mails

Re: [Assp-test] RebuildSDB not running

[Colin Waring]

Hi John,

You need to capture the output of the perl module installer and find out why 
all those modules are failing to install. Fix that. Alternatively, try 
installing each module manually and see the errors.

I’m running Ubuntu 16 and the installer would install most of those if the 
system was set up right.

All the best,
Colin.

From: John Wolf <jwo...@gmail.com>
Sent: 13 April 2018 18:51
To: ASSP development mailing list <Assp-test@lists.sourceforge.net>
Subject: [Assp-test] RebuildSDB not running

Hello All,
A month or so ago I created a new virtual machine in Virtual box.  It is a 
virtual Ubuntu 16.04 server currently running ASSP version 2.6.1  *Fortress*  
build 17355 .  I don't know if I missed something during the install, it seems 
to be working ok except the rebuildsdb process aborts out.  The log shows the 
following:
Server Name:

sfilter



ASSP host UUID:

d96937e9-2ed4-11e8-8f8d-e4fd4c4d0c10



Server OS:

linux



Server IP:

127.0.1.1



used DNS Servers:

192.168.xxx.3 192.168.xxx.254

Local DNS Servers<http://192.168.50.4:5/#UseLocalDNS> in use



defined DNS Servers:

192.168..3 192.168.xxx.254



DNS Servers query time:

min: 0.000 , avg: 0.031 , max: 0.226



Perl Version:

5.022001

Perl.org<http://www.perl.org/get.html>

assp-process-memory:

current: 1407 MB

min: 1075 MB

max: 1407 MB

Spamdb version:

used:

2_14315_UAX#29_UAX#15_WordStem2.02

required:

2_14315_UAX#15

HMMdb version:

used:

n/a

required:

2_14315_UAX#15

code integrity signature:

expected:

D052878A93FA57BC3AAF9774BF1407E1845BD98E

current:

D052878A93FA57BC3AAF9774BF1407E1845BD98E

ASSP Version:

2.6.1(17355)

show current local change log

show last available change 
log<http://downloads.sourceforge.net/project/assp/ASSP%20V2%20multithreading/changelog.txt>


release<http://sourceforge.net/project/showfiles.php?group_id=69172>

beta<http://sourceforge.net/p/assp/svn/HEAD/tree/assp2/trunk/>





 Apr-13-18 11:58:25 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:25 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:25 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:25 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:28 m1-35100-03802 [Worker_2] 66.220.155.145 
<notification+pvv13...@facebookmail.com<mailto:notification%2bpvv13...@facebookmail.com>>
 to: adw...@wselectronics.com<mailto:adw...@wselectronics.com> info: 
PB-IP-Score for '66.220.155.0' is 0, added 10 in this session

 Apr-13-18 11:58:30 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:30 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:30 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:30 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:35 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:35 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:35 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:35 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:40 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:40 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:40 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:40 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:45 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:45 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:45 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:45 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-18 11:58:50 [Worker_10001] Start rebuildAddCorrections

 Apr-13-18 11:58:50 [Worker_10001] Error: Can't locate object method "priority" 
via package "threads" at sub ASSP::Priority::new line 11.

 Apr-13-18 11:58:50 [Worker_10001] Info: RebuildSpamdb Scheduler stopped

 Apr-13-18 11:58:50 [Worker_10001] Info: starting RebuildSpamdb Scheduler with 
'00 01 * * *' - next RebuildSpamdb is scheduled for Apr-14-18 01:00:00

 Apr-13-

Re: [Assp-test] Multiple From headers/regex based on localdomains

[Colin Waring]

Thank you for the reply Thomas,

Being able to include sender:, reply to: and errors-to: would be handy in my 
opinion

However, in this case the local domain was not in any of the from: fields 
whatsoever. By using 
n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>, this hits 
a stupid bug in Outlook where in some places it will only display 
n...@recipientdomain.tld<mailto:n...@recipientdomain.tld>. The -1.me is 
completely fictional and varies from message to message. I thought this would 
not be caught by nospoofing because that would only match if the RHS ended in 
the entry from localdomains.

All the best,
Colin.

From: Thomas Eckardt <thomas.ecka...@thockar.com>
Sent: 13 April 2018 16:55
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Multiple From headers/regex based on localdomains

Colin,

'DoNoSpoofing4From' should do it - but it is'nt. Only the first 'From:' address 
is currently checked and this will not prevent this attack.
But it is possible to include 'sender: , reply-to: and errors-to:' in to this 
check - which would catch this mails.

What do you think?

Thomas





Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:13.04.2018 17:17
Betreff:[Assp-test] Multiple From headers/regex based on localdomains



Hi,



I’ve a couple of fun ones at the moment. Basically I’m getting reports of 
phishing emails that get past everything.



The headers are like this:



Reply-to: Sender Name 
<n...@recipientdomain.tld-1.me<mailto:n...@recipientdomain.tld-1.me>>

To: recipi...@recipientdomain.tld<mailto:recipi...@recipientdomain.tld>

From: Sender Name <f...@domain.tld<mailto:f...@domain.tld>>

From: Sender Name <f...@domain2.tld<mailto:f...@domain2.tld>>

From: Sender Name 
<actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld<mailto:actualsmtpfromaddr...@legitimatebutcompromiseddomain.tld>>



These bypass no spoofing as none of the from/SMTP header domains are actually 
the recipient domain. Annoyingly, Outlook chooses the Reply-to address to 
display so it appears almost legitimate.



I’m aware that the RFCs allow multiple from headers, though I can’t see of any 
legitimate reason for this so I was considering blocking or increasing spam 
score based on this – is this possible with ASSP at the moment or not?



The second thing I was looking at doing was coming up with a regex. 
Essentially, all recipient domains are in localdomains.txt so I’d want a regex 
that would take all lines from localdomains. If the reply to or smtp from 
address is a line from localdomains with anything else after it, then bin it. I 
accept that there may in some extremely obscure cases be a clash with a 
legitimate domain but do not believe that to be likely. I’ll have a look next 
week as to if I can figure out a way to do it but if there’s something obvious 
that you could let me know that’d be great.



All the best,

Colin.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net<mailto:Assp-test@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/assp-test





DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known 
virus in this email!
***
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

So,

As suspected the rebuild debug shows nothing useful at this stage.

I can however now tell where the content of hmmdb is coming from – it is being 
populated whenever someone reports a message through the email interface.

The only files I have in tmpDB currently are:

-rw-r--r-- 1 root root 118557988 Jan  8 06:46 rbtmp.hamHMM.chains
-rw-r--r-- 1 root root  94224002 Jan  8 06:46 rbtmp.hamHMM.totals
-rw-r--r-- 1 root root 240631755 Jan  8 06:47 rbtmp.spamHMM.chains
-rw-r--r-- 1 root root 185125930 Jan  8 06:47 rbtmp.spamHMM.totals


So I’m missing rbtmp.hamHMM and rbtmp.spamHMM

I had a look at the code and saw that the populate part runs the database 
import routine against the hash HMMresObj yet the only place the hash is 
populated is:

$HMMresObj=tie %HMMres,'BerkeleyDB::Hash',
 (-Filename => "$DBDir/rb_HMMres.bdb" ,
  -Flags => DB_CREATE,
  -Env => $BDBEnv);

So, how does the database get populated if BDB is off?

That’s about as far as I can get at the moment I think..

Incidentally I have noticed that spamdb.helo.rb.tmp gets created in the assp 
working directory not tmpDB – I’m not sure whether it is supposed to be there?

All the best,
Colin.


From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 07 January 2018 22:43
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Rebuild has completed:

mysql> select * from hmmdb;
+--++-+
| pkey | pvalue | pfrozen |
+--++-+
| ***COUNT***  | 3  |   0 |
| ***DB-VERSION*** | 2_14315_UAX#29_UAX#15_WordStem2.02 |   0 |
| ***bayesnorm***  | 0.54300466416  |   0 |
+--++-+
3 rows in set (0.00 sec)

So nothing in mysql. ASSP status is all green and I can see the above data by 
using the edit list button next to hmmdb.

Could DBCacheMaxAge have anything to do with this? It was set to 10.

I’m re-running rebuild with the debug file created and will have to check in 
the morning.



From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 07 January 2018 21:08
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Hi Thomas,

I’ve checked and RebuildTestMode is not set.

mysql> select count(*) from hmmdb;
+--+
| count(*) |
+--+
|  5194934 |
+--+
1 row in set (3.35 sec)

The count hasn’t changed overnight so it is definitely not updating.

So I’ve dropped hmmdb, spamdb and spamdbhelo. Run a full update on all the 
servers including perl modules and then restarted everything. Tables recreated 
and now a rebuild is running to hopefully set them up afresh.

Fingers crossed that solves it and hopefully no other tables are affected.

All the best,
Colin.
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 19:06
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Colin, did you set RebuildTestMode  For me, it looks like.

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Remove these tables - they were possibly created many many years ago. I can't 
remember.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 19:29
Betreff:Re: [Assp-test] Meltdown/Spectre



Hi Thomas,



Maybe I’m misunderstanding what populating is? Is populating when the temporary 
db generated by the rebuild are loaded into the mysql server?



I was therefore looking at the mysql server to confirm if any new data was 
being put in it.



Is there any debugging I can turn up to get more information on what is 
happening at that point? I’m not sure if rebuilddebug.txt would give more 
information, I imagine it’d certainly slow down other parts of the rebuild.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 17:34
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



>2018-01-06 22:00:00 Maxbytes: 20,000
ok nearly two hours - that's long - takes on my system ~ 30 min
>2018-01-06 23:51:13 start popul

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

Rebuild has completed:

mysql> select * from hmmdb;
+--++-+
| pkey | pvalue | pfrozen |
+--++-+
| ***COUNT***  | 3  |   0 |
| ***DB-VERSION*** | 2_14315_UAX#29_UAX#15_WordStem2.02 |   0 |
| ***bayesnorm***  | 0.54300466416  |   0 |
+--++-+
3 rows in set (0.00 sec)

So nothing in mysql. ASSP status is all green and I can see the above data by 
using the edit list button next to hmmdb.

Could DBCacheMaxAge have anything to do with this? It was set to 10.

I’m re-running rebuild with the debug file created and will have to check in 
the morning.



From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 07 January 2018 21:08
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Hi Thomas,

I’ve checked and RebuildTestMode is not set.

mysql> select count(*) from hmmdb;
+--+
| count(*) |
+--+
|  5194934 |
+--+
1 row in set (3.35 sec)

The count hasn’t changed overnight so it is definitely not updating.

So I’ve dropped hmmdb, spamdb and spamdbhelo. Run a full update on all the 
servers including perl modules and then restarted everything. Tables recreated 
and now a rebuild is running to hopefully set them up afresh.

Fingers crossed that solves it and hopefully no other tables are affected.

All the best,
Colin.
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 19:06
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Colin, did you set RebuildTestMode  For me, it looks like.

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Remove these tables - they were possibly created many many years ago. I can't 
remember.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 19:29
Betreff:Re: [Assp-test] Meltdown/Spectre



Hi Thomas,



Maybe I’m misunderstanding what populating is? Is populating when the temporary 
db generated by the rebuild are loaded into the mysql server?



I was therefore looking at the mysql server to confirm if any new data was 
being put in it.



Is there any debugging I can turn up to get more information on what is 
happening at that point? I’m not sure if rebuilddebug.txt would give more 
information, I imagine it’d certainly slow down other parts of the rebuild.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 17:34
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



>2018-01-06 22:00:00 Maxbytes: 20,000
ok nearly two hours - that's long - takes on my system ~ 30 min
>2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
>check is now disabled!

>2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
>Bayesian check is now enabled!
there is something wrong - 5 seconds duration with a hardcoded delay of 5 
seconds for 2.5 million records

>2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 
>records!

>2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
>records!
same here, 5.4 million records in less than a second - this is impossible

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Where do you get these MySQL tables/records from ? There is no option (and also 
NO CODE) in assp to tie the temporary HMM tables to mysql. And even if this 
would be possible - mysql is too slow to build the HMM. There are only two 
options in assp to hold the temp HMM tables, BerkeleyDB and memory.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 17:51
Betreff:Re: [Assp-test] Meltdown/Spectre





So a report in from last nights’ rebuild

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

Hi Thomas,

I’ve checked and RebuildTestMode is not set.

mysql> select count(*) from hmmdb;
+--+
| count(*) |
+--+
|  5194934 |
+--+
1 row in set (3.35 sec)

The count hasn’t changed overnight so it is definitely not updating.

So I’ve dropped hmmdb, spamdb and spamdbhelo. Run a full update on all the 
servers including perl modules and then restarted everything. Tables recreated 
and now a rebuild is running to hopefully set them up afresh.

Fingers crossed that solves it and hopefully no other tables are affected.

All the best,
Colin.
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 19:06
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

Colin, did you set RebuildTestMode  For me, it looks like.

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Remove these tables - they were possibly created many many years ago. I can't 
remember.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 19:29
Betreff:Re: [Assp-test] Meltdown/Spectre



Hi Thomas,



Maybe I’m misunderstanding what populating is? Is populating when the temporary 
db generated by the rebuild are loaded into the mysql server?



I was therefore looking at the mysql server to confirm if any new data was 
being put in it.



Is there any debugging I can turn up to get more information on what is 
happening at that point? I’m not sure if rebuilddebug.txt would give more 
information, I imagine it’d certainly slow down other parts of the rebuild.



All the best,

Colin.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 17:34
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



>2018-01-06 22:00:00 Maxbytes: 20,000
ok nearly two hours - that's long - takes on my system ~ 30 min
>2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
>check is now disabled!

>2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
>Bayesian check is now enabled!
there is something wrong - 5 seconds duration with a hardcoded delay of 5 
seconds for 2.5 million records

>2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 
>records!

>2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
>records!
same here, 5.4 million records in less than a second - this is impossible

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Where do you get these MySQL tables/records from ? There is no option (and also 
NO CODE) in assp to tie the temporary HMM tables to mysql. And even if this 
would be possible - mysql is too slow to build the HMM. There are only two 
options in assp to hold the temp HMM tables, BerkeleyDB and memory.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 17:51
Betreff:Re: [Assp-test] Meltdown/Spectre





So a report in from last nights’ rebuild.

Logs are:



2018-01-06 22:00:00 Maxbytes: 20,000
2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
check is now disabled!

2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
Bayesian check is now enabled!
2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 records!

2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
records!
2018-01-06 23:52:22 Total processing time: 6,742 second(s)
2018-01-06 23:52:22 Total processing data: 975.63 Mbyte



So that’s about 20 minutes quicker with nearly double the data processed. 
Marginally more Spamdb records and a reduction of HMM records by 2 million.



Still about half the speed of yours though.

All the best,

Colin.



From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 06 January 2018 20:48
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



I’ll try upping Maxbytes  to 2

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

Hi Thomas,

Maybe I’m misunderstanding what populating is? Is populating when the temporary 
db generated by the rebuild are loaded into the mysql server?

I was therefore looking at the mysql server to confirm if any new data was 
being put in it.

Is there any debugging I can turn up to get more information on what is 
happening at that point? I’m not sure if rebuilddebug.txt would give more 
information, I imagine it’d certainly slow down other parts of the rebuild.

All the best,
Colin.

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 07 January 2018 17:34
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

>2018-01-06 22:00:00 Maxbytes: 20,000
ok nearly two hours - that's long - takes on my system ~ 30 min
>2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
>check is now disabled!

>2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
>Bayesian check is now enabled!
there is something wrong - 5 seconds duration with a hardcoded delay of 5 
seconds for 2.5 million records

>2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 
>records!

>2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
>records!
same here, 5.4 million records in less than a second - this is impossible

mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |

Where do you get these MySQL tables/records from ? There is no option (and also 
NO CODE) in assp to tie the temporary HMM tables to mysql. And even if this 
would be possible - mysql is too slow to build the HMM. There are only two 
options in assp to hold the temp HMM tables, BerkeleyDB and memory.

Thomas




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:07.01.2018 17:51
Betreff:Re: [Assp-test] Meltdown/Spectre



So a report in from last nights’ rebuild.

Logs are:



2018-01-06 22:00:00 Maxbytes: 20,000
2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
check is now disabled!

2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
Bayesian check is now enabled!
2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 records!

2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
records!
2018-01-06 23:52:22 Total processing time: 6,742 second(s)
2018-01-06 23:52:22 Total processing data: 975.63 Mbyte



So that’s about 20 minutes quicker with nearly double the data processed. 
Marginally more Spamdb records and a reduction of HMM records by 2 million.



Still about half the speed of yours though.

All the best,

Colin.



From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 06 January 2018 20:48
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



I’ll try upping Maxbytes  to 2 and see what happens. I’ve also turned off 
usedb4rebuild to see what happens in relation to your other message.



As far as hmmdb goes, I checked everything over and can’t see anything wrong 
although the numbers don’t add up to the ones in the log. The db entries don’t 
have dates against them so I’m not sure how I would check to see if they are 
recent.



-rw-r--r-- 1 root root 0 Jan  5 22:00 BDB-error.txt

-rw-r--r-- 1 root root434175 Jan  5 22:00 __db.001

-rw-r--r-- 1 root root   3325951 Jan  5 22:00 __db.002

-rw-r--r-- 1 root root  65544191 Jan  5 22:13 __db.003

-rw-r--r-- 1 root root663552 Jan  6 00:12 rb_Helo.bdb

-rw-r--r-- 1 root root 334389248 Jan  6 00:08 rb_spam.bdb

-rw-r--r-- 1 root root 332099584 Jan  6 00:13 rbtmp.hamHMM.bdb

-rw-r--r-- 1 root root 168296448 Jan  6 00:13 rbtmp.hamHMM.totals.bdb

-rw-r--r-- 1 root root 339763200 Jan  6 00:13 rbtmp.spamHMM.bdb

-rw-r--r-- 1 root root 335945728 Jan  6 00:13 rbtmp.spamHMM.totals.bdb

-rw-r--r-- 1 root root 12288 Jan  5 23:21 trashlist.bdb



mysql> select count(*) from hmmdb;

|  5194934 |



mysql> mysql> select count(*) from hmmham;

|  1248444 |



mysql> select count(*) from hmmhamtot;

|  1123064 |



mysql> select count(*) from hmmspam;

|  1654660 |



mysql> select count(*) from hmmspamtot;

|  1495532 |





From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 06 January 2018 06:54
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



> 

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

So a report in from last nights’ rebuild.

Logs are:



2018-01-06 22:00:00 Maxbytes: 20,000
2018-01-06 23:51:13 start populating Spamdb with 2,514,865 records - Bayesian 
check is now disabled!

2018-01-06 23:51:18 Finished populating Spamdb with 2,514,865 records - 
Bayesian check is now enabled!
2018-01-06 23:52:22 start populating Hidden Markov Model with 5,418,395 records!

2018-01-06 23:52:22 Finished populating Hidden Markov Model with 5,418,395 
records!
2018-01-06 23:52:22 Total processing time: 6,742 second(s)
2018-01-06 23:52:22 Total processing data: 975.63 Mbyte


So that’s about 20 minutes quicker with nearly double the data processed. 
Marginally more Spamdb records and a reduction of HMM records by 2 million.

Still about half the speed of yours though.
All the best,
Colin.

From: Colin Waring [mailto:co...@dolphinict.co.uk]
Sent: 06 January 2018 20:48
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

I’ll try upping Maxbytes  to 2 and see what happens. I’ve also turned off 
usedb4rebuild to see what happens in relation to your other message.

As far as hmmdb goes, I checked everything over and can’t see anything wrong 
although the numbers don’t add up to the ones in the log. The db entries don’t 
have dates against them so I’m not sure how I would check to see if they are 
recent.

-rw-r--r-- 1 root root 0 Jan  5 22:00 BDB-error.txt
-rw-r--r-- 1 root root434175 Jan  5 22:00 __db.001
-rw-r--r-- 1 root root   3325951 Jan  5 22:00 __db.002
-rw-r--r-- 1 root root  65544191 Jan  5 22:13 __db.003
-rw-r--r-- 1 root root663552 Jan  6 00:12 rb_Helo.bdb
-rw-r--r-- 1 root root 334389248 Jan  6 00:08 rb_spam.bdb
-rw-r--r-- 1 root root 332099584 Jan  6 00:13 rbtmp.hamHMM.bdb
-rw-r--r-- 1 root root 168296448 Jan  6 00:13 rbtmp.hamHMM.totals.bdb
-rw-r--r-- 1 root root 339763200 Jan  6 00:13 rbtmp.spamHMM.bdb
-rw-r--r-- 1 root root 335945728 Jan  6 00:13 rbtmp.spamHMM.totals.bdb
-rw-r--r-- 1 root root 12288 Jan  5 23:21 trashlist.bdb

mysql> select count(*) from hmmdb;
|  5194934 |

mysql> mysql> select count(*) from hmmham;
|  1248444 |

mysql> select count(*) from hmmhamtot;
|  1123064 |

mysql> select count(*) from hmmspam;
|  1654660 |

mysql> select count(*) from hmmspamtot;
|  1495532 |


From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 06 January 2018 06:54
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

> I’m wondering why I have so many more records when Maxbytes is less and the 
> total data is less.

This is caused by HTML mails - mostly SPAM mails.

You may have a look in to some spam mails with a size of 20.000 and more bytes. 
You'll find some, which are starting with alot of HTML header stuff (CSS and 
script and so on). Most times this content is longer than 6000 byte (your 
MaxByte setting).
I saw mails with a size of 25.000 bytes and 10 words of human readable content.
ASSP tries to get the human readable content of HTML mails for analyzing, but 
if this is not possible, it uses the available data.
The CSS and header content is very different in every mail. Even assp 
normalizes this content anyway, this leads in to much more different HMMdb and 
spamDB records - most of them are useless for spam detection.

Have a look in to the GUI for - Use this HTML Parser (HTMLParser).
I use HTML::Strip.

My current setting for MaxBytes (20.000) is only a long time running try out. I 
want to see, how the detection works from 20.000 to 50.000 bytes setting in 
10.000 bytes steps. Each setting is used for ~1 month. MaxBytes 50.000 has 
passed the test and was perfect - like expected - because 100% of spam mails 
(without an attachment) are perfectly analyzed and detected. How ever, this 
setting leads in to a ~25% performance penalty for the rebuild task (in 
relation to 20.000) using my corpus.

>CPU Model: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz

An nice CPU - but with ASSP's single threaded rebuild task it is slower than my 
older Intel(R) Xeon(R) CPU X5680 @ 3.33GHz. 
http://cpuboss.com/cpus/Intel-Xeon-X5680-vs-Intel-Xeon-E5-2640-v2

Collin, don't care about the overall rebuild speed. It runns at night and it 
does'nt hurt, if it takes an hour more or less. Two steps are time critical: 
populating spamDB and populating HMMdb. As you said "The db part looks to be 
fine". But wait 
It looks like, there is something wrong with the temporary rebuild databases 
used for HMM. This can be also the cause for a very very slow rebuild. >>> The 
rebuild was actually quicker a while back, maybe 40m

>2018-01-05 00:07:42 Start populating Hidden Markov Model. HMM-check is 
>disabled for this time!

>2018-01-05 00:07:43 Total processing time: 7,663 second(s)

This is ONE second time difference - totaly impossible - even if HMMdb is hold 
in RAM 

Is it right, that you use 

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

I’ll try upping Maxbytes  to 2 and see what happens. I’ve also turned off 
usedb4rebuild to see what happens in relation to your other message.

As far as hmmdb goes, I checked everything over and can’t see anything wrong 
although the numbers don’t add up to the ones in the log. The db entries don’t 
have dates against them so I’m not sure how I would check to see if they are 
recent.

-rw-r--r-- 1 root root 0 Jan  5 22:00 BDB-error.txt
-rw-r--r-- 1 root root434175 Jan  5 22:00 __db.001
-rw-r--r-- 1 root root   3325951 Jan  5 22:00 __db.002
-rw-r--r-- 1 root root  65544191 Jan  5 22:13 __db.003
-rw-r--r-- 1 root root663552 Jan  6 00:12 rb_Helo.bdb
-rw-r--r-- 1 root root 334389248 Jan  6 00:08 rb_spam.bdb
-rw-r--r-- 1 root root 332099584 Jan  6 00:13 rbtmp.hamHMM.bdb
-rw-r--r-- 1 root root 168296448 Jan  6 00:13 rbtmp.hamHMM.totals.bdb
-rw-r--r-- 1 root root 339763200 Jan  6 00:13 rbtmp.spamHMM.bdb
-rw-r--r-- 1 root root 335945728 Jan  6 00:13 rbtmp.spamHMM.totals.bdb
-rw-r--r-- 1 root root 12288 Jan  5 23:21 trashlist.bdb

mysql> select count(*) from hmmdb;
|  5194934 |

mysql> mysql> select count(*) from hmmham;
|  1248444 |

mysql> select count(*) from hmmhamtot;
|  1123064 |

mysql> select count(*) from hmmspam;
|  1654660 |

mysql> select count(*) from hmmspamtot;
|  1495532 |


From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 06 January 2018 06:54
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

> I’m wondering why I have so many more records when Maxbytes is less and the 
> total data is less.

This is caused by HTML mails - mostly SPAM mails.

You may have a look in to some spam mails with a size of 20.000 and more bytes. 
You'll find some, which are starting with alot of HTML header stuff (CSS and 
script and so on). Most times this content is longer than 6000 byte (your 
MaxByte setting).
I saw mails with a size of 25.000 bytes and 10 words of human readable content.
ASSP tries to get the human readable content of HTML mails for analyzing, but 
if this is not possible, it uses the available data.
The CSS and header content is very different in every mail. Even assp 
normalizes this content anyway, this leads in to much more different HMMdb and 
spamDB records - most of them are useless for spam detection.

Have a look in to the GUI for - Use this HTML Parser (HTMLParser).
I use HTML::Strip.

My current setting for MaxBytes (20.000) is only a long time running try out. I 
want to see, how the detection works from 20.000 to 50.000 bytes setting in 
10.000 bytes steps. Each setting is used for ~1 month. MaxBytes 50.000 has 
passed the test and was perfect - like expected - because 100% of spam mails 
(without an attachment) are perfectly analyzed and detected. How ever, this 
setting leads in to a ~25% performance penalty for the rebuild task (in 
relation to 20.000) using my corpus.

>CPU Model: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz

An nice CPU - but with ASSP's single threaded rebuild task it is slower than my 
older Intel(R) Xeon(R) CPU X5680 @ 3.33GHz. 
http://cpuboss.com/cpus/Intel-Xeon-X5680-vs-Intel-Xeon-E5-2640-v2

Collin, don't care about the overall rebuild speed. It runns at night and it 
does'nt hurt, if it takes an hour more or less. Two steps are time critical: 
populating spamDB and populating HMMdb. As you said "The db part looks to be 
fine". But wait 
It looks like, there is something wrong with the temporary rebuild databases 
used for HMM. This can be also the cause for a very very slow rebuild. >>> The 
rebuild was actually quicker a while back, maybe 40m

>2018-01-05 00:07:42 Start populating Hidden Markov Model. HMM-check is 
>disabled for this time!

>2018-01-05 00:07:43 Total processing time: 7,663 second(s)

This is ONE second time difference - totaly impossible - even if HMMdb is hold 
in RAM 

Is it right, that you use BerkeleyDB for the rebuild? If so -

check the 'tmpDB/rebuildDB/BDB-error.txt' file. It should be zero byte long!

In doubt: shutdown assp, clean the folder  'tmpDB/rebuildDB/', start assp, run 
a rebuild.


Thomas



Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:05.01.2018 21:14
Betreff:Re: [Assp-test] Meltdown/Spectre







From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 05 January 2018 17:16
To: ASSP development mailing list 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Subject: Re: [Assp-test] Meltdown/Spectre



>>time 7,663 seconds, data 486.61 Mbyte

>This is very slow. To be honest - I'm lost for words!

>My rebuild results are:
Mine are very different

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 05 January 2018 17:16
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

>>time 7,663 seconds, data 486.61 Mbyte

>This is very slow. To be honest - I'm lost for words!

>My rebuild results are:
Mine are very different

2018-01-04 22:00:00 Maxbytes: 6,000
2018-01-05 00:03:00 start populating Spamdb with 2,466,760 records - Bayesian 
check is now disabled!
2018-01-05 00:07:42 Start populating Hidden Markov Model. HMM-check is disabled 
for this time!
2018-01-05 00:07:43 Total processing time: 7,663 second(s)
2018-01-05 00:07:43 Total processing data: 486.61 Mbyte
2018-01-05 00:08:37 Uploading Griplist via Direct Connection

The db part looks to be fine considering the times and the extra records that 
mine added. I’m wondering why I have so many more records when Maxbytes is less 
and the total data is less.

My two MX have directly mounted Gluster replicas running off a Fibre channel 
SAN and the rebuild only runs on one.

I have a 4GB tmpDB mounted as tmpfs:

tmpfs  4.0G  1.3G  2.8G  32% /usr/local/assp/tmpDB

Hardware for each is Citrix XenServer 7.2 running on HP DL servers
CPU Model: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
112GB RAM in each with 12GB allocated to each VM
Hard drives aren’t SSD but are on a 1+0 array – I forget how many drives are in 
it but there’s a few. SAN is a Dell Powervault, I’d need to check on the spec.

The VMs are Ubuntu 16.04.3 LTS
16 cores allocated in 4 socket with 4 cores per socket

Primary
top - 20:02:52 up 82 days,  3:40,  1 user,  load average: 0.41, 0.18, 0.11
Tasks: 241 total,   1 running, 240 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 12318500 total,   180648 free,  6131216 used,  6006636 buff/cache
KiB Swap:  8253436 total,  7765076 free,   488360 used.  5702644 avail Mem

Secondary/rebuild
top - 20:02:30 up 66 days,  6:59,  2 users,  load average: 0.05, 0.05, 0.07
Tasks: 250 total,   1 running, 249 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.1 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 12318500 total,   448412 free,  7276144 used,  4593944 buff/cache
KiB Swap:  8253436 total,  6071240 free,  2182196 used.  3396112 avail Mem

ASSP uses 2.3g memory
Clamd about 1G
Gluster 2.2G

Perl is v5.22.1. I believe 5.26 is coming in 18.04 LTS at the end of April 
according to the release schedule. I’ll plan an upgrade sometime after that.

The rebuild was actually quicker a while back, maybe 40m but one of the version 
changes must have had an impact. I couldn’t say which though as I only really 
keep an eye on the amount of data processed and the norm/confidence.

>From my point of view the real bottleneg for the rebuild task is, that only 
>one core (thread) is used by this >task, even there are 12 or more available.
>Because of this (my bad) software design, the speed of a single core matters 
>too much. I think about for >a while to change this. I hope, I'll get this 
>fixed/improved in 2018.

Improvements are always welcome to make a great product even better 

I hope 2018 is good to you.
All the best,
Colin.




Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:05.01.2018 16:01
Betreff:Re: [Assp-test] Meltdown/Spectre



Hi Thomas,



Thank you for the input – I do recall previously discussing ISP mode and 
realising that it was for bigger deployments than ours.



We have three servers. Two handling inbound and one specifically for Office 365 
relaying. The two inbound probably do about 50,000 messages per day between 
them according to infostats.



CPU Usage on both frontends is 1.62% avg and 1.49% avg respectively. I only 
have a single MySQL db (general load average is around 0.1 ) and I’ve been 
watching the hypervisor reports on its performance. I did set up a Gluster sync 
between the two frontends so they have access to the same corpus without having 
to do it over the network – that helped with performance however I’ve never 
been able to get the rebuild run to be particularly quick (Last night’s was 
total processing time 7,663 seconds, data 486.61 Mbyte). I haven’t brought it 
up here because it doesn’t really have much of an effect and it is likely in my 
setup rather than an ASSP issue.



So I think I’ll get away with it on my setup, hopefully this information will 
be helpful to other people who are trying to figure out if they’ll be impacted.



All the best,

Colin Waring.



From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 05 January 2018 13:49
To: ASSP development mailing list 
&l

Re: [Assp-test] Meltdown/Spectre

[Colin Waring]

Hi Thomas,

Thank you for the input – I do recall previously discussing ISP mode and 
realising that it was for bigger deployments than ours.

We have three servers. Two handling inbound and one specifically for Office 365 
relaying. The two inbound probably do about 50,000 messages per day between 
them according to infostats.

CPU Usage on both frontends is 1.62% avg and 1.49% avg respectively. I only 
have a single MySQL db (general load average is around 0.1 ) and I’ve been 
watching the hypervisor reports on its performance. I did set up a Gluster sync 
between the two frontends so they have access to the same corpus without having 
to do it over the network – that helped with performance however I’ve never 
been able to get the rebuild run to be particularly quick (Last night’s was 
total processing time 7,663 seconds, data 486.61 Mbyte). I haven’t brought it 
up here because it doesn’t really have much of an effect and it is likely in my 
setup rather than an ASSP issue.

So I think I’ll get away with it on my setup, hopefully this information will 
be helpful to other people who are trying to figure out if they’ll be impacted.

All the best,
Colin Waring.

From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 05 January 2018 13:49
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Meltdown/Spectre

I remember an ISP issue, who used 10 assp instances with one enterprise MySQL 
backend cluster, sharing all tables for all instances.
In havy workload times (100.000 or even more mails per hour), the MySQL server 
was brought to its end - no matter how many physical resouces were made 
available. Even holding the complete assp DB in the DB-server RAM has not 
solved the problem.
With 100.000 mails per hour and  ~50 DB queries per mail (HMMdb and spamDB), 
the DB server has to process at least 5 million queries in one hour.
If we exclude HMMdb and spamDB, depending on the configuration, there can be 
additionaly 10 to 20 DB queries per mail (for all the other DB-tables). Even 
this can lead in to a very high DB workload!
The URIBL-check can also be very resource expensive (read and write !!!). 
Assume a mail with 100 different URIs is seen the first time - 100 
unsuccessfull cache DB-queries, followed by 100 DNS queries, followed by 100 
cache DB-writes.

To prevent this issue, assp V2 has a buildin ISP mode for HMMdb and spamDB.
In short:
- the corpus of all instances is synchronized to a master instance (rsync for 
example)
- HMMdb and spamDB are hold in memory in each instance and each worker
- HMMdb and spamDB are build on the master system and are distributed as files 
to all other instances using an external script (methode of your choice)
- all other tables are shared traditionaly - but each instance uses a 
configurable DB cache to prevent repeated DB-queries for the same results (for 
example IP checks, helo )

This ISP mode requires at least 16GB RAM per instance, if a maximum of 15 SMTP 
workers is used. Using more than 15 workers in an instance, produces a large 
overhead without any performance improvement.

Collin, I don't know the workload and configuration of your systems - but the 
math is simple.

An possible solution between the standard mode and the ISP mode can be:
- each assp instance has its own DB backend
- all DB-backends are bidirectional synchronized (asynchron) to a 
DB-master-server-cluster

Depending on the overall workload, the DB-master-server-cluster must be an 
enterprise cluster or something like that.
If we assume 10 assp instances, each record change in one instance will lead in 
to one store and nine write sync ops at the master cluster!

If we assume five DB-write ops per mail -> 100 000 mail/h in all instances -> 
500 000 store ops/h + 4.5M sync ops/h at the master cluster.
Yes - the workload at the cluster will be very high, but it is no longer time 
critical and will balance over all the time.
The disadvantage is, that the tables in all instances are never 100% sychron 
and the last instance "winns" in writing the same DB-record. The async state of 
the tables in all DB-backends increases with the overall workload.

You may also think about a ring synchronization between the 10 assp 
DB-backends. The cluster will not be required and the DB-backends will have a 
manageable workload - but the delay of syncing a single record and the data 
inconsitency over all instances will be increased.

Thomas






Von:"Colin Waring" 
<co...@dolphinict.co.uk<mailto:co...@dolphinict.co.uk>>
An:"ASSP development mailing list" 
<assp-test@lists.sourceforge.net<mailto:assp-test@lists.sourceforge.net>>
Datum:05.01.2018 10:45
Betreff:[Assp-test] Meltdown/Spectre



Hi All,



I’m wondering if anyone has updated their ASSP/db backends and monitored the 
performance impact yet.



I’m currently wo

[Assp-test] Meltdown/Spectre

[Colin Waring]

Hi All,

I'm wondering if anyone has updated their ASSP/db backends and monitored the 
performance impact yet.

I'm currently working on assessing just how bad this is going to be with how 
many systems I've got to coordinate hypervisor/OS/microcode updates on so I'm 
checking around with everyone to see who's already got some answers.

All the best,
Colin Waring.

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP ramps up CPU usage after a time

[Colin Waring]

There is indeed a logout option.

Top left just above the left hand menu. The advice has always been to ensure 
that you log out (then press cancel twice) when finished rather than closing 
the browser, along with only using the root user when absolutely necessary.

All the best,
Colin Waring.


-Original Message-
From: James Moe [mailto:ji...@sohnen-moe.com] 
Sent: 15 October 2017 22:59
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] ASSP ramps up CPU usage after a time

On 10/02/2017 11:39 AM, James Moe wrote:
>
> opensuse v42.2
> linux v4.4.87-18.29-default x86_64
> assp  v 2.5.5(17223)
> perl 5.18.2
> 
> After some up time, usually in the range of 1 - 3 weeks, ASSP starts 
> consuming considerably more CPU time.
>
  Finally tracked down what is actually happening here.
  Apparently one of the worker threads runs the web interface when it is 
accessed from a browser. As soon as I logged into the interface the usage went 
from 0.6% to 11% using Worker_1 thread.
  If at some point I need to log in again (the browser closed erasing all 
cookies), another worker thread is assigned to the interface, increasing the 
CPU usage to 23%.
  Since there is no "logout" option, I presume this would continue until ASSP 
ran out of workers to handle the interface?

--
James Moe
moe dot james at sohnen-moe dot com
520.743.3936
Think.

--
Check out the vibrant tech community on one of the world's most engaging tech 
sites, Slashdot.org! http://sdm.link/slashdot 
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Block Reports and filtering

[Colin]

Hi,

Further to my emails on the user list regarding trying to get our 
HMM/Bayes working better I have turned on blocking and am doing my best 
to keep on top of reclassifying mistakes.


As a result I've got a block report coming to me every 4 hours of 
everything that has been blocked which leads me to 3 issues.


1) Although the frequency is set to 4 hours there is no way I can set it 
to only report on last then 1 day - it appears fractions are ignored in 
the report request. Would it be possible to have the report request 
process the first digit after the decimal? At least then I'd be able to 
generate reports an specify it in 2.4 hour blocks rather than 24 hour 
blocks.


2) I've filtered out none bayes/hmm blocks. When an email address has a 
blocked message, the header for that email address is included in the 
report even when all of the emails have been filtered from the display 
and it says "found no blocked messages". Would it be possible to have 
the header for each email address included only if there are blocked 
messages to be displayed on the report?


3) Finally, I have two particular customers who were being affected 
badly by HMM/Bayes blocks so I have had to set them as test mode for 
HMM/Bayes. Unfortunately this means that they don't appear on the block 
report so I can't correct the mistakes. I would like to suggest that if 
anything is set on test mode such as this there needs to be an admin 
report so that work can be done to get them off test mode. This could be 
as simple as including them on the block report with something to 
identify them as test mode. Leaving as is will be training the database 
the wrong way and my only options are to grep the mail logs or trawl 
through the spam corpus looking for anything out of place.


All the best,

Colin Waring.


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Odd behaviour with phishing message

[Colin]

Hi James,

Thanks for the reply, it turns out that I'm seeing odd behaviour for 
this recipient as they are in ptrSpamLovers.


The behaviour isn't what I would have expected - I see these in the 
collected message:


X-Assp-allLovePTRSpam: 1
X-Assp-allLoveSpam: 1

They are not in spamLovers, so apparently them being in PTR spam lovers 
also adds them to the main spam lovers. The message concerned didn't 
actually have a failed PTR so I wouldn't have expected it to be bypass 
the spam filtering.


All the best,

Colin.

On 20/07/2017 21:30, James Moe wrote:


On 07/20/2017 02:25 AM, Colin wrote:


2017-07-18 22:20:23 m1-12823-00551 [Worker_3] [TLS-in] 89.253.223.149
<josep...@lakomvent.ru> to: recipi...@domain.tld recipient delayed:
recipi...@domain.tld
2017-07-18 22:20:23 m1-12823-00551 [Worker_3] [TLS-in] 89.253.223.149
<josep...@lakomvent.ru> to: recipi...@domain.tld [SMTP Status] 451 4.7.1
Greylisting, Please try again after 1 minute


   This only indicates the message was delayed. Where are the logs when
the sender retried to send it?




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Odd behaviour with phishing message

[Colin]

Interestingly I’ve had this reported to me today as well.

There was a message from the beginning of the month about this but I was 
away and don’t think anyone picked it up.


In all cases, the message is a message that has been greylisted 
according to the logs yet it has been delivered to the recipient.


Here’s an example:

2017-07-18 22:20:23 m1-12823-00551 [Worker_3] [TLS-in] 89.253.223.149 
 [SMTP Reply] 250 OK
2017-07-18 22:20:23 m1-12823-00551 [Worker_3] [TLS-in] 89.253.223.149 
 adding new triplet: 
(89.253.223.0,josep...@lakomvent.ru,recipi...@domain.tld) on host 
my.servername.tld
2017-07-18 22:20:23 m1-12823-00551 [Worker_3] [TLS-in] 89.253.223.149 
 to: recipi...@domain.tld recipient delaying 
queued: recipi...@domain.tld
2017-07-18 22:20:23 m1-12823-00551 [Worker_3] [TLS-in] 89.253.223.149 
 to: recipi...@domain.tld [SMTP Reply] 250 Accepted
2017-07-18 22:20:23 m1-12823-00551 [Worker_3] [TLS-in] 89.253.223.149 
 to: recipi...@domain.tld recipient delayed: 
recipi...@domain.tld
2017-07-18 22:20:23 m1-12823-00551 [Worker_3] [TLS-in] 89.253.223.149 
 to: recipi...@domain.tld [SMTP Status] 451 4.7.1 
Greylisting, Please try again after 1 minute


Received message headers:

Received: from server.recipient.tld (1.1.1.1) by
server.recipient.tld (1.1.1.1) with Microsoft SMTP Server
(version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 
15.1.845.34

via Mailbox Transport; Tue, 18 Jul 2017 22:25:37 +0100
Received: from server.recipient.tld (1.1.1.1) by
server.recipient.tld (1.1.1.1) with Microsoft SMTP Server
(version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.1.845.34; Tue, 18 Jul 2017 22:25:37 +0100
Received: from my.server.name (2.2.2.2 ) by
server.recipient.tld (1.1.1.1) with Microsoft SMTP Server
(version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 
15.1.845.34

via Frontend Transport; Tue, 18 Jul 2017 22:25:36 +0100
Received: from [127.0.0.1] (helo=vps-1033709-9570.host4g.ru)
by my.server.name with esmtp (Exim 4.86_2)
(envelope-from )
id 1dXa0l-00031j-1Q
for recipi...@domain.tld; Tue, 18 Jul 2017 22:26:34 +0100
Received: from vps-1033709-9570.host4g.ru ([89.253.223.149] 
helo=vps-1033709-9570.host4g.ru)
by my.server.name with SMTPS(TLSv1_2 
ECDHE-RSA-AES128-GCM-SHA256) (2.5.6); 18 Jul 2017 22:26:30 +0100

Received: by vps-1033709-9570.host4g.ru (Postfix, from userid 48)
id 88D4B2029E6F; Wed, 19 Jul 2017 00:20:20 +0300 (MSK)
From: Joseph C. 
To: Recipient Name 
Subject: [ Possibly Spam ] Enjoy your life, let's program works!
Thread-Topic: [ Possibly Spam ] Enjoy your life, let's program works!
Thread-Index: AQHTAAxl0AYbou0ktEiU2mMiTdVDrw==
Date: Tue, 18 Jul 2017 21:20:20 +
Message-ID: <8b391e9f77fb7215610f423dcbce0...@lakomvent.ru>
Content-Language: en-GB
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-AuthSource: server.recipient.tld
X-MS-Has-Attach:
X-MS-Exchange-Organization-Network-Message-Id: 
76a6f9b5-1c7b-4506-cc1a-08d4ce23882f

X-MS-TNEF-Correlator:
x-assp-envelope-from: josep...@lakomvent.ru
x-assp-intended-for: recipi...@domain.tld
x-php-originating-script: 48:ewocuqmz.php(1166) : runtime-created 
function(1)

: eval()'d code(1) : eval()'d code
x-spam-status: yes
x-mailer: PHPMailer 5.2.23 (https://github.com/PHPMailer/PHPMailer)
Content-Type: multipart/alternative;
boundary="_000_8b391e9f77fb7215610f423dcbce06aalakomventru_"
MIME-Version: 1.0


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Unable to detect any running worker

[Colin]

There's a lot to consider here,

First up, you could do with figuring out the rates at which Exim is 
dumping mail into the queue then throttling it using Exim to see if the 
problem goes away. That way you'll know whether it is a problem with the 
rate or the number/type of email. Look at things like smtp_accept_max or 
maybe queue_smtp_domains to make deliveries go through the queue rather 
than open up a new SMTP thread for every message.


Size of email and encoding/attachments will likely make things take longer.

Secondly, you need to use debugging to find out if something is 
happening that is causing ASSP to take a long time to handle messages.


Thirdly, resources. I've a max of 112 concurrent connections showing on 
the stats page though it is only 43 since last restart on Sunday so the 
general average is lower. I have two VMs running Ubuntu with 16 vCPUs. 
12GB on the primary and 16GB on the secondary as this runs the rebuild. 
MySQL is a separate machine again with 16 vCPUs and 8GB ram.


So ASSP can easily handle the througput you're looking at and more, you 
need to look for bottlenecks and other errors. The actual issue will 
have ocurred at least 30s before the logs you have posted at 08:27:17 as 
that is when the timeout counter started that expired at 08:27:47.


"Cannot pack NaN" makes me suspicious as well for the usual - check all 
perl modules and ancillary files are up to date as well as the main 
assp.pl. Something isn't right.


Then there's the another question about the config. Is there a 
particular reason the Exim server needs to run through ASSP? All my 
servers accept email then hand off to Exim for delivery. There are 
plenty of servers that use ASSP as a smart host, but I'd question 
putting a server that dumps mail like that through. The reason for that 
is to think about the types of emails and the effect on the corpus. If 
you're dumping a mailing list through then you're going to affect the 
bayes/hmm database. You could redlist but then why waste the resources 
and not just have Exim send direct?


I know it's been a week or so since you posted, hopefully you've done 
some or all of that by now as it is fairly standard troubleshooting 
rather than anything specific to ASSP. If you've confirmed your setup is 
in order and can pull some logs that show ASSP actually causing a 
problem then that's what the list is for.


All the best,
Colin.

On 07/07/2017 16:39, MK wrote:

Using ASSP CVS 2.5.6/17184.

I have a server that pumps about 1800 messages into a queue and exim 
on that server makes connections to ASSP to forward the mail. 
Basically ASSP is the outgoing mail server.


It get through about 140 messages, at which point the SMTP connections 
time out (per exim's logs). I'm not sure the concurrency it generates 
to do so, but the connections to the proxy SMTP server it sends to 
gets to about 40 right away and then drops off (so I assume that means 
my concurrent connections about 40)


Meanwhile, ASSP shows:
...[all is fine to here]...
Jul-07-17 08:27:46 [Main_Thread] Info: unable to detect any running 
worker for a new connection - wait (max 30 seconds)

...[repeated]...
Jul-07-17 08:27:47 [Main_Thread] Info: unable to detect any running 
worker for a new connection - wait (max 30 seconds)
Jul-07-17 08:27:47 [Main_Thread] Info: ConnectionTransferTimeOut (30 
seconds) is now reached
Jul-07-17 08:27:47 [Main_Thread] Warning: Main_Thread is unable to 
transfer connection to any worker - try again!
Jul-07-17 08:27:47 [Main_Thread] Error: Main_Thread is unable to 
transfer connection to any worker within 120 seconds - restart ASSP

!
Jul-07-17 08:27:47 [Main_Thread] Initializing shutdown sequence
Jul-07-17 08:27:47 [Shutdown] Info: removing all SMTP and Proxy listeners
Jul-07-17 08:27:47 [Worker_4] Info: shutdown: Worker_4: Cannot pack 
NaN with 'C' at sub main::ipNetwork line 11.
Jul-07-17 08:27:47 [Worker_3] Info: shutdown: Worker_3: Cannot pack 
NaN with 'C' at sub main::ipNetwork line 11.
Jul-07-17 08:27:47 [Worker_5] Info: shutdown: Worker_5: Cannot pack 
NaN with 'C' at sub main::ipNetwork line 11.

Jul-07-17 08:27:47 [Worker_3] Worker_3 finished
Jul-07-17 08:27:47 [Worker_4] Worker_4 finished
Jul-07-17 08:27:47 [Worker_5] Worker_5 finished
Jul-07-17 08:27:47 [Worker_2] Info: shutdown: Worker_2: Cannot pack 
NaN with 'C' at sub main::ipNetwork line 11.

Jul-07-17 08:27:47 [Worker_2] Worker_2 finished
Jul-07-17 08:27:47 [Shutdown] Waiting for all SMTP-Workers to be finished
Jul-07-17 08:27:47 [Worker_1] Info: shutdown: Worker_1: Cannot pack 
NaN with 'C' at sub main::ipNetwork line 11.


Once ASSP restarts and the retry interval is received, ASSP tries 
again, makes it through about 200 messages and then the same outcome.



Of course what it's doing is flooding ASSP with SMTP connections.
The host is in AccetAllMail (yes I know we're not using relayport, but 
we need to make sure the SMTP server can handle a flood of connections 
gracefully)
The maxSM

Re: [Assp-test] ASSP start up errors

[Colin]

Have you made sure you have the latest ASSP_AFC? All require perl 
modules installed and up to date?


It is easy to fall into the trap of only updating assp.pl and not 
checking for any of the other many files that may have been updated!



On 29/06/2017 21:25, James Moe wrote:

Hello,
   linux 4.4.70-18.9-default x86_64
   assp 2.5.5 (17073)
   perl 5.18.2

   Error messages noted when ASSP starts.
   Is there a recommended way to load ASSP_AFC? Or is this a PERL
configuration issue?

using Perl /usr/bin/perl version 5.018002 (5.18.2), all Perl features
for 5.18 are enabled
compiling code and check code integrity - please wait .
checking config in /usr/local/bin/assp2/assp.cfg[OK]
error: preload plugin ASSP_AFC failed in 'use' -
Bareword "Archive::Extract::TGZ" not allowed while "strict subs" in use
at /usr/local/bin/assp2/Plugins/ASSP_AFC.pm line 1877.
...[other similar errors]...
Bareword "ARCHIVE_OK" not allowed while "strict subs" in use at
/usr/local/bin/assp2/Plugins/ASSP_AFC.pm line 1950.
Bareword "ARCHIVE_WARN" not allowed while "strict subs" in use at
/usr/local/bin/assp2/Plugins/ASSP_AFC.pm line 1950.
Compilation failed in require at (eval 29) line 2.
BEGIN failed--compilation aborted at (eval 29) line 2.

the assp.pl code of version 2.5.5(17073) passed the integrity check
ASSP uses AsspSelfLoader 2.03 - check   [OK]
...[ other OKs ]...

   Here is the result from "cpan Archive::Extract::TGZ":

Could not expand [Archive::Extract::TGZ]. Check the module name.
I can suggest names if you install one of Text::Levenshtein::XS,
Text::Levenshtein::Damerau::XS, Text::Levenshtein, and
Text::Levenshtein::Damerau::PP
Skipping Archive::Extract::TGZ because I couldn't find a matching namespace.


   cpan indicates that "Archive::Extract" is current.






--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] lot of JS error with new GUI/LAyout

[Colin]

Hi Renaud,

I run Ubuntu 16.04.1 LTS as well and do not have the issues you desribe. 
I think you need to start looking at logs to see what is happening as we 
can only guess at what is not configured right such as make sure you've 
installed all required perl modules and they are up to date. I know that 
several modules did not build correctly for me so I had to correct them.

I don't use LXC so can't comment on that. Either your ASSP or system 
logs will point you in the direction. Failing that you need to start 
profiling the system and perl to find out what is holding you back.

All the best,

Colin Waring.


On 12/02/2017 14:29, Renaud wrote:
> Hi Thomas, Colin,
>
> For 10.4.2.1 it's a from scratch (download on sourceforge the install
> package) installation under unbuntu xenial amd64 distrib with his is
> perl 5, version 22, subversion 1 (v5.22.1).
>
> Other important information it's an LXC container (Proxmox).
>
> One think the UI is extremely slow, more than 60s to load completely,
> with 500ms for the TTFB. The error come maybe from that point but I
> don't know why ASSP take so long time.
>
> The container have 4 cores and 4 GB of memory
>
> Thanks for your feedback,
> Renaud
>
> Thomas Eckardt a écrit :
>> did you clean the browser cache?
>>
>> Thomas
>>
>>
>>
>>
>>
>> Von:Renaud <ml+a...@manda.tagmail.eu>
>> An:assp-test@lists.sourceforge.net
>> Datum:10.02.2017 11:10
>> Betreff:[Assp-test] lot of JS error with new GUI/LAyout
>> 
>>
>>
>>
>> Hi,
>>
>> I've lot of errors with JS in both desktop and mobile view... Mobile
>> layout work better than the desktop one but I couldn't do simple action
>> like show help for an option or just apply the new config (it works in
>> the mobile view).
>>
>> it happen on two different servers ASSP 2.5.5(17013) which is completely
>> start from the ground  and an ASSP 2.5.5(17036) instance running since 2
>> years and upgrading continuously.
>>
>> The kind of error I have:
>> - TypeError: document.getElementById(...) is null[En savoir plus]
>> :5:380:29
>>  showHelp http://xx:5/:380:29
>>  onclick
>>
>> - TypeError: document.forms.ASSPconfig.theButtonX is undefined[En savoir
>> plus]  10.4.2.1:5:1:1
>>  onclick http://10.4.2.1:5/:1:1
>>
>> My browser is Firefox 51 but I try also with chrome with the same
>> behaviour.
>>
>> Thanks for your help
>> Renaud
>>
>>
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>>
>>
>> DISCLAIMER:
>> ***
>> This email and any files transmitted with it may be confidential,
>> legally privileged and protected in law and are intended solely for the
>> use of the
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> ***
>>
>> --
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Any Outlook users out there? Reporting / analyze question

[Colin]

Hi Ken,

The majority of our users are Exchange/Outlook based. Is yours Outlook 
with Exchange or Outlook with POP/IMAP/SMTP?


I have never seen problem number 2. Reports including multiple 
attachments always work so I cannot help with that. We have Office 365, 
Exchange 2010, Exchange 2013 and now Exchange 2016 and have not seen any 
issues. Various versions of Outlook too. Have you made sure that you are 
not including a signature with the report? I’ve seen many instances 
where signatures cause reports to fail or have odd results so we ensure 
everyone is instruction to remove them.


Problem 1, I have never set DoAdditionalAnalyze before. I have just set 
it and tried. The report that came through was corrupt. Every line ends 
in =0D


I saved to an HTML file and stripped them all out.

I can however say that I am seeing the same problem as you. All of the 
bayes and HMM bad words are out of the headers. The thank you message 
includes both the actual sender and an 
ms...@eurpro01.prod.exchangelabs.com 
<mailto:ms...@eurpro01.prod.exchangelabs.com> address as well from the 
headers.


I have then downloaded the .eml file that ASSP collected from exactly 
the same message and sent that using Thunderbird. I still see some 
information from the headers in Bad Words, but much less. To me it looks 
like the analyse is including the headers for all reports. The problem 
is more evident in the Outlook message because that includes Exchange 
receipt and processing headers. IIRC the analysis only works with a set 
number of lines/bytes at the beginning at the message hence this becomes 
a bigger problem when using Exchange.



I've done a few preliminary searches directly in the database and I see 
the exact same entries appearing there which is a little 
concerning..have you checked your database to confirm if it is just an 
issue with the analyser for you?


All the best,
Colin.

On 19/12/2016 20:01, K Post wrote:

Thanks for chiming in Andrew!!

This is through an exchange server.  The user enters the address from 
remembered addresses or by directly entering the internet address 
(which exchange knows isn't hosted internally).  By "Outlook user" I 
presume you mean Exchange users as Outlook's just a client and not a 
server.I >think< this is all a moot point though as ASSP is 
getting the message correctly - or correctly enough - to save it as 
expected in the corrected corpus.  It's just the analyze report that's 
analyzing the report email itself vs the content of the reported 
email.  And I've got no idea what's going on when I forward multiple 
reports as attachments under one email.  No idea how long that's been 
broken for me.



On Mon, Dec 19, 2016 at 5:44 AM, Andrew Macpherson <and...@oa5.com 
<mailto:and...@oa5.com>> wrote:


Just a thought…. Does Outlook think the mailbox for the report
address is an outlook user or an internet mail user?  (Check in
the sending address book)

Andrew Macpherson <and...@oa5.com <mailto:and...@oa5.com>>
(Twitter @OA5dotCom)

The Old Church, 22-24 Church St, Milnathort, KY13 9XH, GB
Phone tel:+441577861848 <tel:%2B441577861848> GSM
tel:+447899961797 <tel:%2B447899961797>

LEGAL CLAIMER:  Any claims made at this point in a message
are completely invalid as they are presented after the information
they
attempt to assert rights over has been disclosed without prior caveat




> On 19 Dec 2016, at 01:09, K Post <nntp.p...@gmail.com
<mailto:nntp.p...@gmail.com>> wrote:
>
> Can any of you report back on this?  THANKS
>
> On Sun, Dec 4, 2016 at 4:32 PM, K Post <nntp.p...@gmail.com
<mailto:nntp.p...@gmail.com>> wrote:
> I'm curious if there are any ASSP admins out there who use
Outlook on a PC.
>
> We're having 2 minor issues with Spam/NotSpam reports sent from
Outlook and I'm wondering if it's just our installation or if
others are seeing the same thing.  Thomas understandably doesn't
want to install Outlook, so I'm turning to you, the admin users of
ASSP for some quick help.
>
> Note: we send reports to assp by doing a Forward as Attachment,
which preserves the headers.
>
>
> Problem #1: Analyze reports don't work.
> When we send a Spam/NotSpam report, the report itself is saved
perfectly in the corpus.  Headers are intact, the message is
there.  All is well.  However, if we have Spam and Ham Reports
will trigger an additional Analyze Report (DoAdditionalAnalyze)
set to send an analyze report, the report gets sent, but it's all
wrong.  It seems to analyze the headers of the report itself, not
the reported message.  It also almost always triggers and error in
the log like:
> Dec-04-16 16:13:51 Warning: DKIM returned 'no domain to fetch
policy for '
> (that wa

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Colin Waring]

16256 works acceptably but shuts down once or twice a day. 16270 or 16274_1 
gave me problems with delays.

I suspect the shutting down is a symptom of a different problem as it has 
happened for a while.

On 30 Sep 2016 17:57, Thomas Eckardt <thomas.ecka...@thockar.com> wrote:
Hmm ... not OK.

for my records:

build 16256 is running fine
builds 16270 and higher make problems

right?

Thomas





Von:cw <colin.war...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 17:19
Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
servers



I've had to roll back now unfortunately as I'm getting email problems
again
:(

On Fri, Sep 30, 2016 at 3:50 PM, cw <colin.war...@gmail.com> wrote:

> Mixed results on this. So far no problems with running workers being
> logged but the GUI has become incredibly unresponsive. By unresponsive I
> mean I waited a good couple of minutes for the shutdown_list page to
load.
> The dot on the main page is red yet the workers page is all green.
> Scratch that, it has refreshed again and I have a worker stuck:
> Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
> filename name (stuck)
> 30s later and it is healthy again..
>
> On the server I haven't upgraded the shutdown_list page comes up within
> seconds. I'm not sure whether to leave it running or whether this is
> evidence of the same kind of unresponsiveness that cause me to have to
roll
> back earlier this week.
>
> On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:
>
>> I wish I'd spotted this before writing out the other message. I'll give
>> it a test now for you.
>>
>> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> Collin, this should no longer happen using the updated 2.5.2 16274_1
at
>>> CVS /test
>>>
>>> Thomas
>>>
>>>
>>>
>>> Von:cw <colin.war...@gmail.com>
>>> An: ASSP development mailing list
<assp-test@lists.sourceforge.net>
>>> Datum:  29.09.2016 16:40
>>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>>> servers
>>>
>>>
>>>
>>> Hi Thomas,
>>> I moved up to 16270 following this thread of discussion but then had a
>>> day
>>> working away. I've come back to huge issues with delays, mails not
going
>>> through and many, many of these in the logs:
>>>
>>> Info: unable to detect any running worker for a new connection - wait
>>> (max
>>> 30 seconds)
>>>
>>> When I say many, I have over 21,000 lines in today's log file. I also
>>> found
>>> the GUI unresponsive or not connecting at all and ASSP restarting
quite
>>> regularly.
>>>
>>> I've dropped back to 16256 and things are instantly better. Do you
think
>>> going up to 16273 might improve things over 16270 or am I better
holding
>>> off for now?
>>> All the best,
>>> Colin.
>>>
>>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
>>> <thomas.ecka...@thockar.com>
>>> wrote:
>>>
>>> > I just released 2.5.2 build 16273 at CVS test folder
>>> >
>>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>>> >
>>> > This release should make a very large difference for SSL/TLS mails
sent
>>> by
>>> > hosts that uses small SSL-frame size.
>>> >
>>> > Tell me your test results.
>>> >
>>> >
>>> > Thomas
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Von:K Post <nntp.p...@gmail.com>
>>> > An: ASSP development mailing list
<assp-test@lists.sourceforge.net
>>> >
>>> > Datum:  28.09.2016 19:42
>>> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses
/
>>> > servers
>>> >
>>> >
>>> >
>>> > But I want a postman driving a Ferarri with monster truck tires that
>>> can
>>> > roll over the traffic (and if wishes are being granted, I'd prefer
the
>>> car
>>> > in a deep blue instead of classic red).
>>> >
>>> > We regularly see people attaching large files or a bunch of smaller
>>> ones
>>> > that add up to a big email, I'm talking lots and lots of different
>>> people
>>> > from outside the organization sending to us, and this happens on a
>>> daily
>&

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Colin Waring]

I have been running IO::Socket::SSL 2.0.33 though have just updated to 2.0.38. 
I don't think this is going to be related as I have seen this issue for a long 
time and will undoubtedly have had previous versions of OpenSSL.

Don't forget that I see the issue from more than just Google.

I'm quite pushed for time at the moment. Ken, what did you do specifically to 
grab the necessary debugs? - save me having to stop and think :)

All the best,
Colin Waring.


Colin Waring
Technical Manager
Dolphin ICT Limited
T
+44 (0)151 438 2246 Ext 2003
www.dolphinict.co.uk
co...@dolphinict.co.uk
US15a, Armstrong House, First Avenue, Robin Hood Airport, Doncaster, DN9 3GA





Dolphin ICT Limited. NOTICE & DISCLAIMER Dolphin ICT Limited, a private limited 
company, with company registration number 6206916, registered in the United 
Kingdom, the registered office of which is at US15a, Armstrong House, First 
Avenue, Robin Hood Airport, Doncaster, DN9 3GA VAT registration number GB 918 
1896 88. 



-Original Message-
From: K Post [mailto:nntp.p...@gmail.com]
Sent: 27 September 2016 04:53
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

I have IO::Socket::SSL 2.036 installed instead of 2.020.  Could this have 
anything to do with any of this?

On Mon, Sep 26, 2016 at 11:49 PM, K Post <nntp.p...@gmail.com> wrote:

> THANK YOU again for taking all the time on this.  It's nuts that this 
> only seems to happen (to me and others reporting) with TLS on and mail 
> sent through google servers.
>
> I've confirmed the version of Convert::Scalar to be 1.11
>
> I'll get you a debug log privately, but here's what I'm seeing with 
> the latest version:
>
> 11mb attachment, tls on, newest version, but without the 
> $main::neverQueueSize = 4194304; line took 620 seconds.  That's better 
> than the 772seconds that saw before I but still pretty terrible - and 
> of course, that's only one test.
>
> I see a message which I assume is now expected:
> message is too large ( SIZE 15700413 byte > neverQueueSize 1200
> byte) to be queued for further internal processing! Skipping DKIM, 
> Plugins and charset conversion. for that message
>
> I saw a X-ASSP-KEEP line in the header too.  Don't know what that means.
> Haven't seen that before.
>
> Once I added the $main::neverQueueSize = 4194304; line to 
> ASSP_Correct.pm, speed improves for sure.  It took 327 seconds.  Still 
> really slow considering that without TLS it only takes 19 seconds.
> Similar line noting the 4MB size limit Removing the full message 
> analysis seems like a shame especially since it doesn't seem to even 
> stutter if TLS is off.
>
> So more questions for your consideration
> 1) What is TLS doing that slows things down so much for GOOGLE mails 
> only (or at least only google that I've seen be slow)
> 2) What encryption related modules need checking?
> 3) Why would things be fine on your old Windows 2003 rig, but clearly 
> not okay on my (presumably) faster machine
> 4) What is similar between my machine and the others who reported TLS 
> problems with Google.  I know one at least was a Linux rig.
>
>
>
>
>
>
> On Mon, Sep 26, 2016 at 4:02 AM, Thomas Eckardt < 
> thomas.ecka...@thockar.com> wrote:
>
>> First, thank you for the debug file.
>>
>> There is one big problem. The debug file explains the general 
>> behavior of the slowing down connection while the data size is growing.
>> It not explains, why this should only happens at connections from 
>> gmail.com and only if TLS is used.
>>
>> looking at the following timeline - the *** lines are from me and are 
>> showing the count of read-socketcalls within this second
>>
>> 
>> Sep-23-16 21:14:37 [Worker_2] > IO::Socket::INET=GLOB(0x11c1e3bc) (6)<DATA[CR][LF]
>> Sep-23-16 21:14:37 [Worker_2] > Sep-23-16 21:14:38 [Worker_2] > Sep-23-16 21:14:39 [Worker_2] > (each 1440 byte) 164 ...
>> Sep-23-16 21:14:40 [Worker_2] > (each 1440 byte) 167 ...
>> Sep-23-16 21:14:41 [Worker_2] > (each 1440 byte) 108 ...
>> Sep-23-16 21:14:42 [Worker_2] > (each 1440 byte) 95 ...
>> Sep-23-16 21:14:43 [Worker_2] > (each 1440 byte) 82 ...
>> Sep-23-16 21:14:44 [Worker_2] > (each 1440 byte) 74 ...
>> Sep-23-16 21:15:09 [Worker_2] > (each 1440 byte) 43 ...
>> Sep-23-16 21:15:39 [Worker_2] > (each 1440 byte) 35 ...
>> Sep-23-16 21:16:39 [Worker_2] > (each 1440 byte) 21 ...
>> Sep-23-16 21:18:39 [Worker_2] > (each 1440 byte) 12 ...
>> Sep-23-16 21:22:41 msg79676-04975 209.85.223.177 
>> <nntp.p...@gmail.com>
>> to:
>> testtls@[[ OUR DOMAIN ]].org info: message is too large (

Re: [Assp-test] invalidFormatHeloRe

[Colin Waring]

Hi Thomas,

Thanks for the reply. That was actually a typo in my email, I did mean 
validFormatHeloRe not invalidFormatHeloRe.

The file on SourceForge is out of date. It hasn't been updated in nearly three 
years and still has w{2,6} in it - I checked this prior to posting as my way of 
keeping things up to date is comparing them with 
http://assp.cvs.sourceforge.net/viewvc/assp/assp2

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 09 September 2016 08:38
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] invalidFormatHeloRe

>invalidFormatHeloRe

No - 'validFormatHeloRe' makes this rule.

This regex was change at the begin of this year (I think) - the default is

file:files/validhelo.txt

validhelo.txt:

^(?:\w[\w\.\-]*\.\w{2,64})$
^[a-fA-F0-9]{1,4}:([a-fA-F0-9:]{1,4}){1,}(?:(?:\.\d+){3})?$


Thomas




Von:cw <colin.war...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  08.09.2016 13:26
Betreff:[Assp-test] invalidFormatHeloRe



Hi,



I’m not an expert at Regexs otherwise I’d look at this myself.



I’ve had someone emailing me about problems getting mail through and at first 
glance it was due to an invalid HELO. At second glance, the HELO is actually 
valid and points to a domain that has a valid DNS record. The HELO is 
server.kalo.digital



This fails the default regex for invalidFormatHeloRe because the regex
stipulates that the last part of the HELO has to be between 2 and 6
characters long. This doesn’t take into account the more recent TLDs that
have been forced on the Internet of which .digital is one being 7
characters.



I can’t find anything in RFC1123 that specifically states the number of
characters for the TLD so is this a problem with the Regex rather than the
usage of TLDs with more than 6 characters?



All the best,

Colin Waring.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Colin Waring]

I have to say I've seen this and I posted about it back in January.

https://sourceforge.net/p/assp/mailman/message/34783916/

Back then I saw problems with Gmail, Yahoo Mail and SMTPRoutes. Since then I've 
occasionally fielded calls from different people saying that emails aren't 
coming through and the solution has been to add the IP to noTLSip. The problem 
was much more significant back in January because I was getting lots of 
complaints whereas now it is only occasional.

I'm on a completely different architecture to you.

Ubuntu 14.04.4 LTS, OpenSSL 1.0.1f (latest from apt), Perl v5.18.2, Net::SSLeay 
1.74, IO::Socket::SSL 2.033, Net::SMTP::SSL 1.03

I've been using cpanm and cpanoutdated to manage module updates, checking from 
within cpan I can see that a number of modules haven't been done that way so 
I'm running upgrade from within CPAN itself to get things up to date. One of 
the updates is Net:SSLeay 1.77 so I'll see what that does.

All the best,
Colin Waring.


Colin Waring
Technical Manager
Dolphin ICT Limited
T
+44 (0)151 438 2246 Ext 2003
www.dolphinict.co.uk
co...@dolphinict.co.uk
US15a, Armstrong House, First Avenue, Robin Hood Airport, Doncaster, DN9 3GA





Dolphin ICT Limited. NOTICE & DISCLAIMER Dolphin ICT Limited, a private limited 
company, with company registration number 6206916, registered in the United 
Kingdom, the registered office of which is at US15a, Armstrong House, First 
Avenue, Robin Hood Airport, Doncaster, DN9 3GA VAT registration number GB 918 
1896 88. 



-Original Message-
From: K Post [mailto:nntp.p...@gmail.com]
Sent: 01 August 2016 23:06
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: [Assp-test] Inbound TLS from gmail.com addresses / servers

I originally thought that we had a problem with all TLS inbound email.  As it 
turns out, my conclusion appears to have been wrong.


   - There are some SLOW servers outside that are just plain slow (nothing
   I can do there),

   - TLS seems to work reasonably fast with most inbound mail, though
   significantly slower than without TLS  (5 seconds for an 11mb file without
   tls, vs 45 seconds with TLS on)

   - GMAIL.com inbound TLS emails are SLOW, no matter what settings I tweak


With inbound gmail.com message. if I have TLS off, an 11mb attachment is 
delivered through ASSP in under 5 seconds.  With TLS on it takes close to
10 minutes, which gets close to gmail's limit.

I've tested with Outlook.com and that same 11mb attachment comes in through 
ASSP with TLS on in about 45 seconds.

Sending a 30mb attachment from gmail FAILS because it takes too long. gmail 
will try for I believe 10 minutes to send a message, then it quits and retries. 
 After a couple tries, it sends an NDR.

This is a Windows 2012 R2 server, latest ASSP dev, OpenSSL 1.0.2h installed 
from slproweb.com/products/Win32OpenSSL.html (though I've also tried with the 
OpenSSL I downloaded a while back from the ASSP sourceforge site.
 net::ssleay 1.74 (openssl 1.0.2g).  I'm almost certain that the OpenSSL 
installation is not used by ASSP, but I've not been able to get confirmation of 
that here.

Just updated IO::Socket::SSL to 2.033.
Net::SMTP:SSL 1.02.

CPU usage as reported by assp is 4.78%.  It's not on the fastest machine in the 
world (it's a hypver-v guest on a decent machine), but it seems speedy enough.  
24gb ram.  We've got similar physical hosts running Exchange as a guest without 
any speed issues whatsoever.

Any other info I can provide to help figure this out?

Disabling TLS for any gmail inbound mail isn't a feasible option, plus I don't 
know if it really is just google, or just the way that google connects which 
others might too...

Thank you all.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] More MX and A record lookup issues

[Colin Waring]

You need debug logs and set something up to monitor your DNS traffic. You need 
to be certain whether the issue is with ASSP handling DNS or your DNS setup. 
This information is the only thing that will really let you track your issue 
down.

All the best,
Colin Waring.

-Original Message-
From: K Post [mailto:nntp.p...@gmail.com] 
Sent: 19 May 2015 14:57
To: ASSP development mailing list
Subject: [Assp-test] More MX and A record lookup issues

Running 15135 on a Windows 2012 box.

I've got a message that was ultimately erroneously rejected due to total score. 
 Contributing to this score is ASSP being (for some reason) unable to find A or 
MX records for the sending IP.  This isn't the first time I've seen this.  My 
last suggestion of potentially having ASSP retry dns lookups if neither A or MX 
returns anything was dismissed as crazy.  I don't know what else to suggest.  
Here's what I'm seeing:

In analyze everything looks great:
• domain bounce.e.hautelook.com (in Mail From:) has a valid MX record:
bounce-mx.exacttarget.com
• domainMX bounce-mx.exacttarget.com has a valid A record: 66.231.91.54 • 
domain e.nordstromrack.com (in From , Reply-To) has a valid MX record:
reply-mx.s6.exacttarget.com
• domainMX reply-mx.s6.exacttarget.com has a valid A record: 198.245.82.46 • 
198.245.83.134 SenderBase: status=white SenderBase, data=[CN=US, 
ORG=EXACTTARGET, DOM=hautelook.com, BLS=, HNM=Y, CIDR=20, HN= 
mta6.e.hautelook.com] Senderbase should have given a bonus, the A and MX record 
is there, so it shouldn't have counted against the message.

But in the message in the corpus, I see:
X-ASSP-Message-Score: 10 (MX missing: bounce.e.hautelook.com (Mail From:))
X-ASSP-IP-Score: 10 (MX missing: bounce.e.hautelook.com (Mail From:))
X-ASSP-Message-Score: 15 (A record missing: bounce.e.hautelook.com (Mail
From:))
X-ASSP-IP-Score: 15 (A record missing: bounce.e.hautelook.com (Mail From:)) 
Senderbase doesn't seem to have run either

I see nothing else to indicate that the machine is having DNS problems of any 
kind.  It's looking to a set of internal DNS servers that are fast and reliable 
- they're used for all of our servers and none of them have any dns issues.

It's not light exacttarget, a major mailing company used by big companies, 
temporarily removed the A and MX records for this hostname.

Any idea of what could be going on and how to correct it?  Could it be that 
this is happening to others but I'm the only one going through almost every 
questionally blocked message by hand (hate this part)??


Thanks
--
One dashboard for servers and applications across Physical-Virtual-Cloud Widest 
out-of-the-box monitoring support with 50+ applications Performance metrics, 
stats and reports that give you Actionable Insights Deep dive visibility with 
transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
--
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] speed of adding records to spamdb table

[Colin Waring]

My assumption would be that is the estimate of the number of seconds it will 
take the process to complete.

Our rebuild takes about 10 seconds to populate the database, you do need to do 
some network tuning and make sure your database is optimised for purpose, I 
can't help you with MS SQL though.

All the best,
Colin Waring.

-Original Message-
From: K Post [mailto:nntp.p...@gmail.com] 
Sent: 23 April 2015 15:25
To: ASSP development mailing list
Subject: [Assp-test] speed of adding records to spamdb table

Working to get the rebuild process to complete.  Win32.  MS SQL DB.

Are these speeds normal??  I'm a little confused to the sec numbers, It's not 
5,000+ seconds, I don't think. and I don't know why secs would be decreasing.  
confused.

Apr-23-15 10:21:37 Added 176152 of 998035 records for table spamdb - finished 
in 5081 sec
Apr-23-15 10:21:38 Added 176346 of 998035 records for table spamdb - finished 
in 5078 sec
Apr-23-15 10:21:40 Added 176540 of 998035 records for table spamdb - finished 
in 5081 sec
Apr-23-15 10:21:42 Added 176928 of 998035 records for table spamdb - finished 
in 5077 sec
Apr-23-15 10:21:47 Added 177704 of 998035 records for table spamdb - finished 
in 5073 sec
Apr-23-15 10:21:48 Added 177785 of 998035 records for table spamdb - finished 
in 5075 sec
Apr-23-15 10:21:49 Added 177940 of 998035 records for table spamdb - finished 
in 5074 sec
Apr-23-15 10:21:53 Added 178560 of 998035 records for table spamdb - finished 
in 5071 sec
Apr-23-15 10:21:55 Added 178870 of 998035 records for table spamdb - finished 
in 5069 sec
Apr-23-15 10:21:57 Added 179180 of 998035 records for table spamdb - finished 
in 5068 sec
Apr-23-15 10:21:59 Added 179490 of 998035 records for table spamdb - finished 
in 5066 sec
Apr-23-15 10:22:01 Added 179800 of 998035 records for table spamdb - finished 
in 5065 sec
--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own 
process in accordance with the BPMN 2 standard Learn Process modeling best 
practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ 
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ClamAV win32 Sane

[Colin Waring]

The subject tests shouldn't require AFC at all, as the subject comes early on 
in the message clamav should catch it normally.

I'm not sure if there's a debug option for the scanning or afc. You could turn 
on general debug, run the test then turn it off again.

For clamd itself you might need to make sure the logging is configured for 
Windows:

LogFile C:/ClamAv/Logs/clamd.log
LogTime yes
LogClean yes
LogFileMaxSize 0

The latter two won't be needed for normal operation as they will produce larger 
log files.

All the best,
Colin Waring.

-Original Message-
From: K Post [mailto:nntp.p...@gmail.com] 
Sent: 16 March 2015 15:28
To: ASSP development mailing list
Subject: Re: [Assp-test] ClamAV win32 Sane

Thank you Colin!!

I have almost the same settings as yours.   The only differnce is DoASSP_AFC
is set to both.  I tried yesterday with AFC off though, and it's still not 
caught.

When tests 1 and 3 get caught, it does appear that the sane signatures are 
catching them:

Mar-14-15 16:06:08 msg63566-10522 209.85.220.175 
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org ClamAV:
scanned 2232 bytes in whitelisted message - FOUND 
Sanesecurity.TestSig_Type4_Bdy.3.UNOFFICIAL
Mar-14-15 16:06:08 msg63566-10522 209.85.220.175 
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
Message-Score: added 50 (vdValencePB) for virus detected:
'Sanesecurity.TestSig_Type4_Bdy.3.UNOFFICIAL', total score for this message is 
now 35
Mar-14-15 16:06:08 msg63566-10522 [VIRUS] 209.85.220.175 
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org [spam 
found] (virus detected: 'Sanesecurity.TestSig_Type4_Bdy.3.UNOFFICIAL') [3rd in 
body] - messages/discarded/3rd_in_body--67.txt;

But, yeah, when it's only the subject that has the test, I see AFC pluggin 
being called, but no hit!

Not sure where else to look or what else to try.  It's certainly not the end of 
the world, but I worry based on the Sane guy saying how important this one is - 
that headers are often what's in the signature files.


On Mon, Mar 16, 2015 at 5:34 AM, Colin Waring co...@dolphinict.co.uk
wrote:

 Your log looks to me like the settings simply aren't calling Clam to 
 scan the message rather than clam missing the message.

 I have ScanWL, ScanNP, ScanLocal, ScanCC and UseAvClamd enabled and 
 you need to make sure that AvClamdPort is correct for your system.
 DoASSP_AFC is set to enabled but only set to do attachments. If you 
 haven't got the main clam settings enabled, you'll need to make sure 
 that ASSP_AFCSelect is set to one of the options that scans the whole message.

 2015-03-15 15:34:57 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com info: found message size announcement:
 1.56 kByte
 2015-03-15 15:34:57 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com IP 209.85.214.176 matches 
 whiteListedIPs - with 209.85.128.0/17
 2015-03-15 15:34:57 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com [SMTP Reply] 250 OK
 2015-03-15 15:34:57 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld [SMTP 
 Reply]
 250 Accepted
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld [SMTP 
 Reply]
 354 Enter message, ending with . on a line by itself
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld Whitelisted 
 sender address: sen...@gmail.com for recipient recipi...@domain.tld
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld 
 DKIM-Signature found
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld info: 
 domain gmail.com has published a DMARC record
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld ClamAV:
 scanned 1774 bytes in whitelisted message - FOUND
 Sanesecurity.TestSig_Type4_Hdr.2.UNOFFICIAL(740814f660dc883f8fe4646084
 30ae9f:1774)
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld Message-Score:
 added 50 (vdValencePB) for virus detected:
 'Sanesecurity.TestSig_Type4_Hdr.2.UNOFFICIAL(740814f660dc883f8fe464608
 430ae9f:1774)', total score for this message is now 50
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS-out] 
 [VIRUS]
 209.85.214.176 sen...@gmail.com to: recipi...@domain.tld [spam 
 found] (virus detected:
 'Sanesecurity.TestSig_Type4_Hdr.2.UNOFFICIAL(740814f660dc883f8fe464608
 430ae9f:1774)') 
 [rrg63Uhj2UCyECcruX7D83A4qd5UA5vnlgwJp6b6fmPZpObZJAbftehuhRAXFby] - 
 /usr/local/assp/store/quarantine/rrg63Uhj2UCyECcruX7D83A4qd5UA5vnlgwJp
 6b6fmPZpObZJA--571715.eml;
 2015-03-15 15:34:58 m1-33697-05727 [Worker_6] [TLS-in] [TLS

Re: [Assp-test] ClamAV win32 Sane

[Colin Waring]

Howdy,

I think you need to pull some logs for both ASSP and clam. I've run the tests 
on my install and they all got blocked properly.

I'm not using Windows though so can't help with the setup.

All the best,
Colin Waring

On 14 Mar 2015 20:07, K Post nntp.p...@gmail.com wrote:
Correction, the first 2 sane tests slip through, 3rd IS trapped.

On Sat, Mar 14, 2015 at 4:05 PM, K Post nntp.p...@gmail.com wrote:

 I've got the sane signatures installed on a windows box with ASSP.

 Has anyone tried these tests?
 http://sanesecurity.com/support/signature-testing/

 I've tried this with and without the AFS plugin.  Same results.  All 3
 messages arrive.

 UseAVClamD is on
 DoFileScan is off

 When I run tests from http://www.emailsecuritycheck.net/, some of the
 tests are coming through as well, but some are caught.

 Any suggestions would be appreciated.

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ClamAV win32 Sane

[Colin Waring]

I'll look into them for you but it'll be tomorrow before I do.

All of them got blocked, though I did see the same effect on gmail from the 
HTML one.

All the best,
Colin Waring

On 15 Mar 2015 18:32, K Post nntp.p...@gmail.com wrote:
Colin-
really, I'm just interested in the results of the 2nd test in your log.  I
managed to get the html email one to be trapped - apparently sending html
mail from gmail is a bit different.  From outlook it trapped it.

The one where the spam string is in the subject however, doesn't seem to be
caught though.  It looks like one of our bombre is scoring the long
subject.  I don't now why that would stop a detection though.  It does look
like the ASSP_AFC is being called (it was enabled for this test).


Mar-15-15 14:27:37 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
Received-RWL: listed from list.dnswl.org; client-ip=209.85.220.177
Mar-15-15 14:27:37 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
Message-Score: added -2 for 209.85.220.0 in griplist (0.14), total score
for this message is now -42
Mar-15-15 14:27:37 msg44055-12284 [DKIM] 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org [scoring]
DKIM signature failed - none - sender policy is: neutral - author policy
is: neutral
Mar-15-15 14:27:37 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
Message-Score: added 10 (dkimValencePB) for DKIM none, total score for this
message is now -32
Mar-15-15 14:27:38 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org info:
SenderBase - query using SenderBase
Mar-15-15 14:27:38 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org
SenderBase -- used Senderbase -- country:US orgname:GOOGLE domain:google.com
Mar-15-15 14:27:39 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org HMM is
not available - hmmdb is still locked by a rebuild task
Mar-15-15 14:27:40 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org Bayesian
Check [monitoring] - Prob: 1.0 = spam
Mar-15-15 14:27:40 msg44055-12284 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org [Plugin]
calling plugin ASSP_AFC
Mar-15-15 14:27:40 msg44055-12284 [MessageOK] 209.85.220.177
testexter...@ourgoogledomain.org to: v...@testmail.ourcharity.org message
ok [rrg63Uhj2UCyECcruX7D83A4qd5UA5vnlgwJp6b6fmPZpObZJAbftehuhRAXFby] -
messages/okmail/rrg63Uhj2UCyECcruX7D83A4qd5UA5vnlgwJp6b6fmPZpObZJA--73.txt


I've got the sanesecurity.ftm database there, last modified 9/3/14

Thank you for your help!
--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Localdomains stopping working

[Colin Waring]

Thanks again for the reply,

I've stayed away from that because I always intended to have a central logging 
server thus would need syslog for that, it just hasn't happened yet!

I'll have to look into LDAP, it makes sense that you could use a group in the 
flat files and then manage everything through that. 

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 11 March 2015 07:37
To: ASSP development mailing list
Subject: Re: [Assp-test] Localdomains stopping working

Monitoring runs on localhost

You should have a look in to the assp-monitor.pl script. This script emulates a 
SYSLOG server. If syslog is configured in assp to send the log to ths 
assp-monitor SYSLOG server, the script will watch permanently if assp is 
running or not. You have to modify the script for your local needs, like: IP, 
Port, timing values, restart command and so on. But this is easy to see.
The advantage of this script is, that assp is monitored even the instance is 
idle for hours.

Some of our configuration files are generated externally, such as
localdomains

In this case assp rereads the file every 5 minutes (per default). Here we 
have the five minutes - and the reload is normal. Make sure your external 
collection script makes no mistake! 

I just set up different users so we could stop using root, clicked logout 
and got the login prompt.

You have to click cancel in the login prompt - this should be shown in the 
login prompt window.

The sequence in maillog.txt is like this:

Mar-11-15 07:59:52 [Main_Thread] Admin connection from user root on host 
***; page:/logout; session-ID:31d32662563be88bd596b72bb20bcb3c;
Mar-11-15 07:59:52 [Main_Thread] Logout from admin interface requested for 
user 'root' at '**'
Mar-11-15 07:59:52 [Main_Thread] Terminated WEB session 
31d32662563be88bd596b72bb20bcb3c for user 'root' at ''
Mar-11-15 07:59:52 [Main_Thread] Terminated WEB session 
6eb2b017b825cd3defc7c48c441ab01b for user 'root' at ''
Mar-11-15 07:59:52 [Main_Thread] Terminated WEB session 
3e8252de5c6b289718e69c86a8b68ad1 for user 'root' at ''

 Would there be a preferred way to have any updates sent to ASSP rather 
than overwriting the file?

I prefer using LDAP and the Groups feature for registering and classifying 
domains, IP's and users.
The concept of assp allows to have a central LDAP server where all 
domains, groups, IP's and users are registered.
As a result, the usage of the assp GUI is only required for major 
configuration changes - all other domain , IP and user based changes have 
to be only done in the LDAP directory.

Thomas





Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  10.03.2015 20:21
Betreff:Re: [Assp-test] Localdomains stopping working



Actually this raises a few other questions (sorry!).

Monitoring runs on localhost and the script basically calls the telnet 
command then searches the output for Connected. The web admin is 
configured to use https so the monitoring command should never actually 
set up a session with ASSP. I'll need to do a bit more with the script to 
change it to look for a particular response on port 3.

Some of our configuration files are generated externally, such as 
localdomains which comes from a combination of different systems. Would 
there be a preferred way to have any updates sent to ASSP rather than 
overwriting the file? I know this isn't causing the problem as the first 
thing I did was stop the scripts involved.

I just set up different users so we could stop using root, clicked logout 
and got the login prompt. When I tried to log back in I got user root is 
currently logged on from host 10.0.5.51 - no new sessions will be accepted 
until root has logged off. So it looks like even though I clicked logout 
the session didn't get cleaned up properly.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 10 March 2015 16:04
To: ASSP development mailing list
Subject: Re: [Assp-test] Localdomains stopping working

It doesn't authenticate and doesn't attempt to do anything with the
connection.
I wouldn't have thought that an unauthenticated connection would be 
able
to have any impact


The reason is the root login without an logout. assp caches the complete 
web communication for the root account. Because it is doing this, no other 
login is allowed while root has an active login.
Now for example - if the monitor (5) runs on the same system or is 
connected from the same IP (NAT) like a root-web session it may possible 
(should not, but who knows) that the monitor connection is misinterpreted. 

There is simply no web connection code in assp, that expects a non-browser 
session.
The web code of assp is written for browsers and it is not perfect in 
terms of security if http is used. For this reason https should be used

Re: [Assp-test] Localdomains stopping working

[Colin Waring]

Hi Thomas,

Thank you for the very in depth responses. You're a star as always. I'll give 
them a proper review later.

My first thought is that the monitoring script that I use only checks that it 
can open a connection. It doesn't authenticate and doesn't attempt to do 
anything with the connection. I wouldn't have thought that an unauthenticated 
connection would be able to have any impact on the configuration as that seems 
like a significant security issue.

The monitoring script runs every 60s not five minutes, I did previously look at 
SNMP but couldn't get any results so I'll add that to the high priority list. I 
use that script as it has other monitors in such as queue length, MTA 
monitoring and some system admin tasks.

We will definitely stop using the root login though. Strange how we haven't 
seen any issues at all until last week.

All the best,
Colin Waring

On 10 Mar 2015 10:38, Thomas Eckardt thomas.ecka...@thockar.com wrote:
Colin - I find it hard to believe. You brought home the bacon. :):):)

NEVER EVER use the web listerner 5 to monitor assp - this can lead in
to unexpected config changes or config reloads - in worth case you can
lose parts or the complete configuration.



These are very BASIC IT rules - and they also applies to assp:

Don't login to assp as 'root'. Use 'root' only, if you need to access
restricted configuration parameters.
NEVER forget to use the 'logout' button in the GUI - especially NOT if
root is logged on!


2015-03-09 09:38:34 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 106
records
2015-03-09 09:43:33 [Main_Thread] Adminupdate: [root 192.168.11.13] file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains'

2015-03-09 21:37:10 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 104
records
2015-03-09 21:42:11 [Main_Thread] Adminupdate: [root 192.168.11.13] file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains'

exactly 5 minutes difference - Colin, can you remember about this 5
minutes - is it an accidental circumstance, that the monitor to port 5
is running every 5 minutes ??
But - it is NOT a  accidental circumstance, that the last root web-session
was not logged out!

all has been said

2015-03-09 00:04:33 [Main_Thread] Info: added schedule : BlockReport -
for : *@domain.tld=*=1= - at : 0 0,4,8,12,16,20 * * * - next run is at
: 2015-03-09 04:00:00

this is normal - the MaintThread has changed the file after the
blockreport is done

2015-03-09 02:42:11 [Main_Thread] Option list file:
'/usr/local/assp/files/droplist.txt' reloaded (droplist) with 658 records

this is normal - the MaintThread has download the file

This is a huge problem, as localdomains errors cause mail to be
incorrectly rejected and leads to serious complaints. If I can't resolve
this within the next few days I'm likely to have to switch to a different
product which I really don't want to do.

good luck


Thomas






Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  10.03.2015 10:05
Betreff:Re: [Assp-test] Localdomains stopping working



Hi again,

This looks to be a more serious issue now affecting other config files. It
appears that ASSP reloads the flat files and gets the entries wrong.
192.168.11.X is my home office subnet that is allowed access to the admin
interface via VPN. This brings up two things.

1) At first glance it looks like ASSP is incorrectly and sometimes
partially reloading the localdomains file whenever a setting is changed
via the admin interface. Localdomains.txt did not change at all yesterday
yet we have differing numbers of entries indicating the file was only
partially loaded.
2) The first entry at 00:34:50 is impossible. The router for 192.168.11.X
was turned off at approximately 22:30 and not turned back on until 07:00
therefore there could not have been any admin update from the 192.168.11.X
subnet.
3) None of these coincide with actual connections to the admin interface.
There are no logs preceding that say IP 192.168.11.X matches
allAdminConnectionsFrom. The only admin connections to this instance were
at 2015-03-08 14:42:01 from .11 and 2015-03-09 08:02:14 from .13

2015-03-09 00:34:50 [Main_Thread] Adminupdate: [root 192.168.11.11] file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains' was
changed
2015-03-09 00:34:50 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 139
records
2015-03-09 09:38:34 [Main_Thread] Adminupdate: [root 192.168.11.13] file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains' was
changed
2015-03-09 09:38:34 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 106
records
2015-03-09 09:43:33 [Main_Thread] Adminupdate: [root 192.168.11.13] file
'/usr/local/assp/files

Re: [Assp-test] Localdomains stopping working

[Colin Waring]

Actually this raises a few other questions (sorry!).

Monitoring runs on localhost and the script basically calls the telnet command 
then searches the output for Connected. The web admin is configured to use 
https so the monitoring command should never actually set up a session with 
ASSP. I'll need to do a bit more with the script to change it to look for a 
particular response on port 3.

Some of our configuration files are generated externally, such as localdomains 
which comes from a combination of different systems. Would there be a preferred 
way to have any updates sent to ASSP rather than overwriting the file? I know 
this isn't causing the problem as the first thing I did was stop the scripts 
involved.

I just set up different users so we could stop using root, clicked logout and 
got the login prompt. When I tried to log back in I got user root is currently 
logged on from host 10.0.5.51 - no new sessions will be accepted until root has 
logged off. So it looks like even though I clicked logout the session didn't 
get cleaned up properly.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 10 March 2015 16:04
To: ASSP development mailing list
Subject: Re: [Assp-test] Localdomains stopping working

It doesn't authenticate and doesn't attempt to do anything with the
connection.
I wouldn't have thought that an unauthenticated connection would be 
able
to have any impact


The reason is the root login without an logout. assp caches the complete web 
communication for the root account. Because it is doing this, no other login is 
allowed while root has an active login.
Now for example - if the monitor (5) runs on the same system or is 
connected from the same IP (NAT) like a root-web session it may possible 
(should not, but who knows) that the monitor connection is misinterpreted. 
There is simply no web connection code in assp, that expects a non-browser 
session.
The web code of assp is written for browsers and it is not perfect in terms of 
security if http is used. For this reason https should be used and if anyhow 
possible a Client-SSL-certificate authentication should be configured mandatory.

You're a star as always.

No, I'm a gyp artist.
Call me Betelgeuse :):)


Colin, do a telnet to assp port 3 (webStatPort) and press two times enter - 
you'll get the right answer - 'healthy'
or the bad one - 'not healthy'. Both answers are configurable. I think your 
monitor don't need to know more.

Thomas



Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  10.03.2015 13:30
Betreff:Re: [Assp-test] Localdomains stopping working



Hi Thomas,

Thank you for the very in depth responses. You're a star as always. I'll give 
them a proper review later.

My first thought is that the monitoring script that I use only checks that it 
can open a connection. It doesn't authenticate and doesn't attempt to do 
anything with the connection. I wouldn't have thought that an unauthenticated 
connection would be able to have any impact on the configuration as that seems 
like a significant security issue.

The monitoring script runs every 60s not five minutes, I did previously look at 
SNMP but couldn't get any results so I'll add that to the high priority list. I 
use that script as it has other monitors in such as queue length, MTA 
monitoring and some system admin tasks.

We will definitely stop using the root login though. Strange how we haven't 
seen any issues at all until last week.

All the best,
Colin Waring

On 10 Mar 2015 10:38, Thomas Eckardt thomas.ecka...@thockar.com wrote:
Colin - I find it hard to believe. You brought home the bacon. :):):)

NEVER EVER use the web listerner 5 to monitor assp - this can lead in to 
unexpected config changes or config reloads - in worth case you can lose parts 
or the complete configuration.



These are very BASIC IT rules - and they also applies to assp:

Don't login to assp as 'root'. Use 'root' only, if you need to access 
restricted configuration parameters.
NEVER forget to use the 'logout' button in the GUI - especially NOT if root is 
logged on!


2015-03-09 09:38:34 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 106 
records
2015-03-09 09:43:33 [Main_Thread] Adminupdate: [root 192.168.11.13] 
file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains'

2015-03-09 21:37:10 [Main_Thread] Option list file:
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 104 
records
2015-03-09 21:42:11 [Main_Thread] Adminupdate: [root 192.168.11.13] 
file
'/usr/local/assp/files/localdomains.txt' for config 'localDomains'

exactly 5 minutes difference - Colin, can you remember about this 5 minutes - 
is it an accidental circumstance, that the monitor to port 5 is running 
every 5 minutes ??
But - it is NOT a  accidental

Re: [Assp-test] Localdomains stopping working

[Colin Waring]

 I'm likely to have to switch to a different product which I 
really don't want to do.

All the best,
Colin Waring.

-Original Message-
From: Colin [mailto:colin.war...@gmail.com] 
Sent: 03 March 2015 17:44
To: ASSP development mailing list
Subject: [Assp-test] Localdomains stopping working

Howdy,

We've had this a couple of times in the last week or so:

2015-03-03 15:17:15 [Main_Thread] Saving config
2015-03-03 15:17:15 [Main_Thread] Info: no configuration changes detected - 
nothing to save - file /usr/local/assp/assp.cfg is unchanged
2015-03-03 15:17:15 [Main_Thread] Adminupdate: [root ] file 
'/usr/local/assp/files/localdomains.txt' for config 'localDomains' was changed
2015-03-03 15:17:15 [Main_Thread] Option list file: 
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with
104 records

On the face of it, looks fine as it loads all the entries but after this point 
ASSP acts as though the file is empty. All inbound mail gets bounced with:

[SMTP Error] 530 Relaying not allowed (enable smtp authentication on your email 
client)

I've verified with the MTA that this isn't an MTA error, ASSP is generating 
this before passing the connection on to it.

The localdomains.txt file is updated automatically by a script so that could be 
the trigger for the reload.

Any thoughts?


--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored 
by Intel and developed in partnership with Slashdot Media, is your hub for all 
things parallel software development, from weekly thought leadership blogs to 
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test



--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.4.4 build 15067

[Colin Waring]

Thanks for the explanation Thomas,

Most of the changes I've been making are aimed at redundancy over performance. 
For example I intended to build a MySQL cluster and put it behind a load 
balancer so that we can handle the DB server going offline for maintenance etc.

I do have one issue that I've never been sure about whether it is performance 
related. Quite regularly, ASSP will accept connections and hold them for 
anywhere from a few seconds up to 10-20 seconds and then carry on. It is 
noticeable enough that when I'm using the web admin to change between a few 
settings I'll quite often see it. Most of the time it doesn't cause any 
problems as it always finishes processing after the delay.

Every now and then though it doesn't come back. ASSP won't respond to any 
shutdown commands so I have to kill the process, remove the pid file and start 
it back up manually. My monitoring scripts only kick in if they can't connect 
to port 25/5.

I suspect this won't help as that doesn't look to be performance related and 

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 09 March 2015 05:38
To: ASSP development mailing list
Subject: Re: [Assp-test] fixes in assp 2.4.4 build 15067

Colin,

If I understand more of how experimental this is and the next step is 
to HA the database

At this time the code is very experimental and very special. It aims to fix 
SMTP performance problems for an ISP, which holds around 20.000 domains.
The concept of the central RDB (for HMM and Bayesian) backend is not fast 
enough to process several hundred thousands or million mails a day.
If 100.000 mails have to be processed with HMM and/or Bayesian in a day, this 
will lead in to 6.000.000 - 60.000.000 SQL queries a day (only for HMM).
What DB engine (cluster) is able to do this? And this is only the average 
calculation - what about the peaks?
The code is currently specialized for the environment of this single ISP and is 
not generic enough to go public. 
There are currently no changes made to enhance the implementation of other 
features, like blockreporting or anything else.

Thomas





Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  08.03.2015 15:18
Betreff:Re: [Assp-test] fixes in assp 2.4.4 build 15067



Hi Thomas,

I'd be very interested to know more details on the ultimate aim with the ISP 
option. I support the idea of subscription for the higher end as it will help 
create funding for you past donations.

Is the aim of the addition to add support for extended scalability or do you 
have ideas for the future to make additional features available? If you 
remember we exchanged emails a while back about some of the features that I 
could see benefiting a larger setup and we are looking into how to implement 
things at the moment.

I've already implemented clustered file systems and the next step is to HA the 
database. The biggest concern for me in scaling up is the block reports being 
generated on each server individually.

If I understand more of how experimental this is and what could go wrong then I 
may be able to help with testibg.

All the best,
Colin Waring

On 8 Mar 2015 12:39, Thomas Eckardt thomas.ecka...@thockar.com wrote:
Hi all,

fixed in assp 2.4.4 build 15067:

- on some windows systems 'Win32::Unicode' was detected as unavailable, even it 
was correctly installed

- the alpha index was not working in build 15059

- HMM was not working, if 'spamdb' was set to a plain file, placed in a 
subfolder like: db/spamdb


added:

- This build contains experimental code to setup assp in very large ISP 
environments, with a very high workload
  caused by HMM, Bayesian and DNS.
  Such a setup requires an enormous and expensive amount of hardware resources, 
a very high knowledge in
  system design and OS scripting.
  minimum requirements:
  - assp: 64Bit OS, all SSD, 16GB RAM, 8 CPU cores, 64Bit Perl (multiple 
larger systems expected)
  - external high available enterprise database server
  - high available and very fast DNS-servers

  This ISP setup option is subject to become a payed licensed feature.


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known 
virus in this email!
***

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored 
by Intel and developed in partnership with Slashdot Media, is your hub for all 
things parallel software development, from weekly thought leadership blogs to 
news, videos

Re: [Assp-test] fixes in assp 2.4.4 build 15067

[Colin Waring]

Hi Thomas,

I'd be very interested to know more details on the ultimate aim with the ISP 
option. I support the idea of subscription for the higher end as it will help 
create funding for you past donations.

Is the aim of the addition to add support for extended scalability or do you 
have ideas for the future to make additional features available? If you 
remember we exchanged emails a while back about some of the features that I 
could see benefiting a larger setup and we are looking into how to implement 
things at the moment.

I've already implemented clustered file systems and the next step is to HA the 
database. The biggest concern for me in scaling up is the block reports being 
generated on each server individually.

If I understand more of how experimental this is and what could go wrong then I 
may be able to help with testibg.

All the best,
Colin Waring

On 8 Mar 2015 12:39, Thomas Eckardt thomas.ecka...@thockar.com wrote:
Hi all,

fixed in assp 2.4.4 build 15067:

- on some windows systems 'Win32::Unicode' was detected as unavailable,
even it was correctly installed

- the alpha index was not working in build 15059

- HMM was not working, if 'spamdb' was set to a plain file, placed in a
subfolder like: db/spamdb


added:

- This build contains experimental code to setup assp in very large ISP
environments, with a very high workload
  caused by HMM, Bayesian and DNS.
  Such a setup requires an enormous and expensive amount of hardware
resources, a very high knowledge in
  system design and OS scripting.
  minimum requirements:
  - assp: 64Bit OS, all SSD, 16GB RAM, 8 CPU cores, 64Bit Perl
(multiple larger systems expected)
  - external high available enterprise database server
  - high available and very fast DNS-servers

  This ISP setup option is subject to become a payed licensed feature.


Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
***

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Issue: Spamlover email saved to both spam/notspam folders

[Colin]

baysSpamLoversRed seems to be the appropriate setting, although the 
description of it doesn't appear all that clear.

On 06/03/2015 10:46, Mr. Courtney Creighton wrote:
 Hi,

 I've recently added a bunch of spamlover mail users, who just want spam
 mails marked. But I've noticed that my notspam directory is also getting
 copies of the bayesian detected spam for the spamlovers put into it.
 There's no mention in the maillogs about that action, but assp is
 apparently duplicating spam into the notspam folder as well.

 It's the exact copy of the spam mail that is also in the spam folder.

 I can't find any setting allowing this. All my collections settings
 appear to be correct. I've been running assp for more than 10 years,
 tweaking the config as I go along. It seems unlikely that something in
 my config puts spam into the notspam folder and I wouldn't have noticed
 this previously. I seem to remember a previous bug where this copying
 mail into multiple folders was happening. Is it back?

 Can anyone else confirm this? As far as I know at this point, it may
 just be something that is happening for only spamlover users.

 You can run this command (Linux) from your assp root directory and see
 if it turns up any results:
   find notspam -type f -print | xargs grep -l X-Assp-spamlover: 1

 -C

 --
 Dive into the World of Parallel Programming The Go Parallel Website, sponsored
 by Intel and developed in partnership with Slashdot Media, is your hub for all
 things parallel software development, from weekly thought leadership blogs to
 news, videos, case studies, tutorials and more. Take a look and join the
 conversation now. http://goparallel.sourceforge.net/
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Localdomains stopping working

[Colin]

Howdy,

We've had this a couple of times in the last week or so:

2015-03-03 15:17:15 [Main_Thread] Saving config
2015-03-03 15:17:15 [Main_Thread] Info: no configuration changes 
detected - nothing to save - file /usr/local/assp/assp.cfg is unchanged
2015-03-03 15:17:15 [Main_Thread] Adminupdate: [root ] file 
'/usr/local/assp/files/localdomains.txt' for config 'localDomains' was 
changed
2015-03-03 15:17:15 [Main_Thread] Option list file: 
'/usr/local/assp/files/localdomains.txt' reloaded (localDomains) with 
104 records

On the face of it, looks fine as it loads all the entries but after this 
point ASSP acts as though the file is empty. All inbound mail gets 
bounced with:

[SMTP Error] 530 Relaying not allowed (enable smtp authentication on 
your email client)

I've verified with the MTA that this isn't an MTA error, ASSP is 
generating this before passing the connection on to it.

The localdomains.txt file is updated automatically by a script so that 
could be the trigger for the reload.

Any thoughts?


--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Corrupt messages

[Colin]

Hi,

I've seen this before but it slipped my mind. This is an example of a 
message where after a few lines every line of a message gets corrupted. 
It looks like a regex has replaced from the first alphanumeric up until 
the first whitespace with X.

This message was blocked dut to having no from header. The header is 
there but corrupted in the corpus. It must have been corrupted before 
message processing for ASSP to not see the from header.

X-Assp-Envelope-From: sen...@domain.tld
X-Assp-Intended-For: recipi...@domain.tld
X-Assp-Delay: not delayed (whitelisted); 1 Mar 2015 21:14:59 +
X-Assp-Message-Score: 50 (From missing)
X-Assp-IP-Score: 50 (From missing)
X-Assp-Whitelisted: Yes (whiteListedIPs '1.1.1.0/18')
X-Assp-Tag: MessageLimit
X-Spam-Status:yes
X-Assp-Spam-Reason: MessageScore 50, limit 50
X-Assp-Message-Totalscore: 50
Received: from host.tld.com ([1.1.1..5]
 helo=host.tld.com) by mail.smtphost.co.uk with SMTP
 (2.4.4); 1 Mar 2015 21:14:52 +
XXX v=1; a=rsa-sha1; c=relaxed/relaxed; s=mandrill; 
d=recipientdomain.tld;
  
XX
 i=i...@recipientdomain.tld;
  
  XX


 a=rsa-sha1; c=nofws; q=dns; s=mandrill; 
d=recipientdomain.tld;
  XX

X
X from pmta03.mandrill.prod.atl01.rsglab.com (127.0.0.1) by 
host.tld.com id hue0la1sau8v for recipi...@domain.tld; Sun, 1 Mar 2015 
21:12:41 + (envelope-from sen...@domain.tld)
XXX v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandrillapp.com;
  XXX q=dns/txt; s=mandrill; t=1425244361; h=From :
  XXX : To : Message-Id : Date : MIME-Version : Content-Type :
  X : From : Subject : Date : X-Mandrill-User :
  X bh=GcgOa6X59ZJsf0KFehKYcucCDDWWvCnVHXPUrNIVBBI=;
  
  
  XX
X Recipient Name i...@recipientdomain.tld
 Trade Account Registration
 sen...@domain.tld
X from [92.63.138.71] by mandrillapp.com id 
1d4e8afcc1264b0ca1ed9f0675fc3f42; Sun, 01 Mar 2015 21:12:41 +
XXX recipi...@domain.tld
XXX Please forward a copy of this message, including all 
headers, to ab...@mandrill.com
XXX You can also report abuse here: 
http://mandrillapp.com/contact/abuse?id=idnumber
 md_30007454
XXX 30007454.20150301211241.54f380c91a9320.22340...@host.tld.com
X Sun, 01 Mar 2015 21:12:41 +
X 1.0
X text/html; charset=utf-8
XX 7bit

XX type=text/css
 X table, #email-format tr, #email-format td, 
#email-format p{margin:0; padding:0; border:0; font-family:Arial; 
font-size:12px;}
 X table td img { border: 0 none; }


XX width=760 cellspacing=0 cellpadding=0 border=0 
style=text-align: left; border: 1px solid #d4d4d4; border-radiXXX 5px; 
id=email-format
 XXX
 



--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Net::SMTP::SSL Broken

[Colin]

Hi All,

This isn't an ASSP bug, but a heads up to anyone building a new system. 
As it turns out, apparently Net::SMTP::SSL hasn't been updated in many 
years. Recent changes in libnet (post 1.27) mean that Net::SMTP::SSL 
will no longer pass build tests.

The cpan bug is here:

https://rt.cpan.org/Public/Bug/Display.html?id=99454

The discussion there implies that Net::SMTP::SSL is going to become a 
pseudo package for Net::SMTP as this supports SSL natively now.

On a test machine I am building (Ubuntu 14.04 LTS with perl 5.18.2) I 
had to do the following to get Net::SMTP::SSL to install:

perl -MCPAN -e shell
o conf urllist push http://backpan.perl.org/
install SHAY/libnet-1.27.tar.gz

If I do any perl module updates in the future I'm going to have to be 
really careful not to let this upgrade because it could break things again.

All the best,
Colin Waring.


--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Bug. Webadmin Left Navigation from MailLog not working?

[Colin]

This happens with any of the top links in Chrome. Any link that 
dynamically rewrites the main display area disables the ability to use 
the #JumpTo links in the left menu.

You'd need some special javascript handler on all of the links to catch 
it and check if the main display area had changed which I think is a bit 
excessive. Currently the solution is to just refresh the page but a 
simpler solution might be a Config link up at the top that changes the 
main area back to the list of config values.

All the best,
Colin Waring.

On 11/02/2015 22:40, Peter Hinman wrote:
 I've seen this in Chrome.  The URL in the address bar changes when I
 click on a link, but the page itself doesn't get updated.  I have to
 click on the address bar and then hit enter.  Seems like the target
 has been removed from the link?

 Peter Hinman
 International Bridge / ParcelPool.com

 On 2/10/2015 4:26 PM, K Post wrote:
 I take that back.  Restarting did NOT fix this.  (I mistakenly restarted
 and then tried on the production system, not in the lab),  Problem is still
 there.

 On Tue, Feb 10, 2015 at 6:20 PM, K Post nntp.p...@gmail.com wrote:

 Restarting ASSP resolved this.  I haven't been able to recreate.


 On Tue, Feb 10, 2015 at 3:38 PM, K Post nntp.p...@gmail.com wrote:

 Running 150025 in a lab.

 If I access the mail log, then expand a menu item from the left, say
 listenport under network setup  I can click, and it'll show
 https://mylabip:8100/#listenPort in the url bar, but it doesn't seem to
 navigate away from the log.

 This happens in chrome and IE.  Clicking on Main does bring up the
 expected UI.



 --
 Dive into the World of Parallel Programming. The Go Parallel Website,
 sponsored by Intel and developed in partnership with Slashdot Media, is your
 hub for all things parallel software development, from weekly thought
 leadership blogs to news, videos, case studies, tutorials and more. Take a
 look and join the conversation now. http://goparallel.sourceforge.net/
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


 --
 Dive into the World of Parallel Programming. The Go Parallel Website,
 sponsored by Intel and developed in partnership with Slashdot Media, is your
 hub for all things parallel software development, from weekly thought
 leadership blogs to news, videos, case studies, tutorials and more. Take a
 look and join the conversation now. http://goparallel.sourceforge.net/
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Non-missing MX

[Colin]

Hi Scott,

This would break many things. If we are to accept the message then 
bounce it later, where does the bounce go? Do we try to figure it out 
and send it to www instead of web. or do we just end up with a 
non-deliverable NDR sat in our queues? The former is really bad because 
someone could set up a shill domain and point it at a target to use for 
mailbombing.

Really, the way that ASSP acts is in accord with how things should be 
set up on the Internet. The people responsible for webengineer.com 
should defined MX records for web. if they wish to use it to send email. 
If you want to over-ride this yourself, the best thing to do is to set a 
whitelist entry on your server.

All the best,
Colin Waring

On 10/02/2015 15:54, Scott MacLean wrote:
 I have a client who is being blocked because ASSP is reporting that
 their domain does not have a valid MX. The domain in question is
 www.webengineer.com:

 Feb-09-15 16:00:22 NB-15619-06295 [Worker_2] [MissingMX] {IPAddr}
 #@web.webengineer.com to: #@.org [scoring] MX missing
 (cache): web.webengineer.com

 However, doing a lookup against web.webengineer.com shows:

 ;www.webengineer.com.   IN  MX

 ;; ANSWER SECTION:
 www.webengineer.com.3600IN  CNAME   webengineer.com.
 webengineer.COM.3585IN  MX  1 mail.webengineer.com.

 So www.webengineer.com is a CNAME to webenginner.com, which in fact DOES
 have an MX, which is mail.webengineer.com.

 Is it possible to have ASSP follow that CNAME and do a recursive lookup
 for MX in this type of case?



 --
 Dive into the World of Parallel Programming. The Go Parallel Website,
 sponsored by Intel and developed in partnership with Slashdot Media, is your
 hub for all things parallel software development, from weekly thought
 leadership blogs to news, videos, case studies, tutorials and more. Take a
 look and join the conversation now. http://goparallel.sourceforge.net/
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Resend request unreachable host emails

[Colin]

Hi,

I've for some reason started getting a few complaints from people 
receiving emails that mean absolutely nothing to them.

The emails have the subject forward resend request queued for host 
(hostname).

I understand exactly what these emails are, but I question them being 
sent to the end user. Is there an option to set these so they are sent 
to our admin email address instead so that we know immediately if there 
is a problem?

Are there any other ASSP status messages that might benefit from this too?

All the best,
Colin Waring.

--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] failed BlockReport forward queue

[Colin]

Further to this, this looks like an easy to replicate problem.

If I set up ASSP-A and ASSP-B to forward requests to each other, they 
work. If I change the IP address of ASSP-B and ASSP-A receives a resend 
request that it needs to forward to ASSP-B it will save the request to 
the .store file

If I then update ASSP-A block report forwarding with the new IP, the 
request gets forwarded and the message delivered. The entry remains in 
the .store file and gets reprocessed every time ASSP looks at it thus 
leading to many copies of the message going through.

I haven't tried breaking the link between the two without changing IPs 
so I don't know if this issue will occur under normal behaviour or 
under unusual circumstances when making configuration changes.

All the best,
Colin Waring.

On 10/02/2015 17:50, Colin Waring wrote:
 -Original Message-
 From: Colin [mailto:colin.war...@gmail.com]
 Sent: 05 February 2015 14:16
 To: ASSP development mailing list
 Subject: [Assp-test] failed BlockReport forward queue

 Hi,

 [Worker_1] Info: checking failed BlockReport forward queue, having 5 
 entries

 Where is this stored? I have one mailserver holding onto 5 resend requests 
 that have all succeeded. It keeps instructing the other mailserver to send a 
 fresh copy of the message which is understandably annoying to clients seen as 
 it is happening every few minutes!

 I even moved the source file out of the corpus but ASSP still manages to find 
 it from somewhere. Restarting ASSP doesn't clear it out either and I can't 
 find any files that hold this info.




 --
 Dive into the World of Parallel Programming. The Go Parallel Website, 
 sponsored by Intel and developed in partnership with Slashdot Media, is your 
 hub for all things parallel software development, from weekly thought 
 leadership blogs to news, videos, case studies, tutorials and more. Take a 
 look and join the conversation now. http://goparallel.sourceforge.net/
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] failed BlockReport forward queue

[Colin]

Hi,

[Worker_1] Info: checking failed BlockReport forward queue, having 5 
entries

Where is this stored? I have one mailserver holding onto 5 resend 
requests that have all succeeded. It keeps instructing the other 
mailserver to send a fresh copy of the message which is understandably 
annoying to clients seen as it is happening every few minutes!

I even moved the source file out of the corpus but ASSP still manages to 
find it from somewhere. Restarting ASSP doesn't clear it out either and 
I can't find any files that hold this info.




--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] assp 100% cpu but basicly idle

[Colin]

How are you monitoring the usage? I remember getting caught out by what 
you are describing when I first started monitoring performance.

Don't forget that top by default shows you the performance figures 
relating to a single core. If you have a multiple core system then you 
will regularly see more than 100% and a load higher than 1. Useful 
options are I (capital i) to change the CPU% to relative to the total 
and push f and select the P column to show which CPU/core each process 
is running on.

I also saw what you were seeing in strace as natural behaviour. What you 
should find is that the WAKE events are preceeded by a poll event doing 
POLLIN. If there aren't any connections waiting it will return resource 
temporarily unavailable and it will go back to WAIT.

I do see a lot of gettimeofday in strace too and I think I get a couple 
of minutes delay in things being written to the logs which is odd.

What monitoring service do you have and how does it identify ASSP as not 
responding?

All the best,
Colin.

On 28/01/2015 09:51, krz...@gmail.com wrote:
 I have assp (ASSP version 2.4.1(14132)) running on multiple servers.
 Those servers have exacly same configuration, os (mirrored), hardware.
 On one of these servers assp is causing high cpu ussage. After
 restarting assp cpu usage is rising constantly and slowly. After
 about 4 hours it it 100% on i7-4770. After about 8-16 hours assp is so
 slow to respond that monitoring service sees it as not running and
 restarts it. Server has low traffic volume, much lower that most of my
 other servers. There is nothing in assp logs (even on highest
 verbosity for connection logging) - a smtp session every 5-30 seconds
 maybe. Assp ASSP Worker/DB/Regex Status shows workers in
 ThreadGetNewCon status. Strace executed on assp pid shows a lot more
 of FUTEX_WAKE_OP_PRIVATE in comparison than on assp on other servers.
 There is no problems with other software on server and nothing on
 dmesg so I don't think it is a hardware problem. I even tried
 reinstalling fresh pel 5.18 with new modules by mod_inst.pl. Can
 anyone help?

 futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
 {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
 sched_yield()   = 0
 futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
 {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
 sched_yield()   = 0
 sched_yield()   = 0
 futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
 {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
 futex(0x7fb2da74fa20, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN
 (Resource temporarily unavailable)
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
 futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
 {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
 futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
 {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
 futex(0x7fb2da74fa20, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN
 (Resource temporarily unavailable)
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
 futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
 {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
 sched_yield()   = 0
 futex(0x7fb2da74fa5c, FUTEX_WAIT_PRIVATE, 88147493, NULL) = -1 EAGAIN
 (Resource temporarily unavailable)
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
 futex(0x7fb2da74fa58, FUTEX_WAKE_PRIVATE, 1) = 1
 futex(0x7fb2da74fa5c, FUTEX_WAIT_PRIVATE, 88147495, NULL) = -1 EAGAIN
 (Resource temporarily unavailable)
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
 futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
 {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
 futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
 {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
 futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
 futex(0x7fb2da74fa58, FUTEX_WAKE_PRIVATE, 1) = 1
 futex(0x7fb2da74fa5c, FUTEX_WAIT_PRIVATE, 88147501, NULL) = -1 EAGAIN
 (Resource temporarily unavailable)

 --
 Dive into the World of Parallel Programming. The Go Parallel Website,
 sponsored by Intel and developed in partnership with Slashdot Media, is your
 hub for all things parallel software development, from weekly thought
 leadership blogs to news, videos, case studies, tutorials and more. Take a
 look and join the conversation now. http://goparallel.sourceforge.net/
 ___
 Assp-test mailing list
 Assp-test

Re: [Assp-test] Socket poll cycle

[Colin]

On 28/01/2015 09:48, Thomas Eckardt wrote:
 I was mainly asking about tracking down why connections are going
 unanswered by ASSP


 unanswered ?
Yes, I have been dealing with a large mail provider investigating why 
there are delays on emails inbound to us. They have provided me with 
logs showing no response to their connections.

I set iptables to log every SYN packet to port 25 so I have a record of 
every inbound connection. I see SYN packets logged that match the time 
of the logs from this provider.

The ASSP logs show no evidence of an inbound connection within five 
minutes each side of the time logged by iptables. There are no 
connection debug logfiles written to the debug directory relating to it 
either.

 Warning: the operating system socket poll cycle has taken
 3.10847902297974 seconds - this is very much is too long
 This is only a warning. It tells you, that it has taken a long time
 (typical are 0.01 to 1 seconds - accepted are up to 3.0 seconds) to query
 the OS for the state of the current connected sockets - nothing else. The
 OS has answered and everything is running well.

 btw. you see this warning (only), because 'ConnectionLog' is higher than
 standard.

 Thomas

Thanks for the explanation, the logging setting is high because of 
trying to track down where these connections are going missing. If this 
isn't related to the missing connections then I have no evidence at all 
of where to start looking. I suspect that it isn't related as the timing 
doesn't coincide and this is logged far less frequently than the missing 
connections issue occurs.




 Von:cw colin.war...@gmail.com
 An: ASSP development mailing list assp-test@lists.sourceforge.net
 Datum:  28.01.2015 09:11
 Betreff:Re: [Assp-test] Socket poll cycle



 Thanks for the reply,

 I was mainly asking about tracking down why connections are going
 unanswered by ASSP when the OS has already seen them and iptables has
 logged them. I have connection debug set to highest and nothing shows up
 relating to these connections.

 Is there any condition under which ASSP will hit a limit and not log
 message stating so?

 The socket poll logs were something I saw in the logs maybe 20 times a day
 whilst trying to find something relating to the first issue. Most of the
 socket poll logs come from workers not the main thread and the system load
 is quite low.

 CPU usage is less than 20% at peak and the load only gets up to 3 on an 8
 core CPU which is a little over 25%. I did increase to 12 cores to see if
 extra power would help but it didn't.
 On 28 Jan 2015 06:05, Thomas Eckardt thomas.ecka...@thockar.com wrote:

 Warning: the operating system socket poll cycle
 Like stated - the call to an OS function - in this case socket-poll -
 has
 too long. This may happen - but sould not happen too often.
 It could be caused, if the core where the MainThread is running on, is
 used extensive at the moment or the complete system is under havy load
 for
 some seconds.
 There is nothing you do to track this down in assp

 Thomas



 Von:Colin colin.war...@gmail.com
 An: ASSP development mailing list assp-test@lists.sourceforge.net
 Datum:  27.01.2015 19:50
 Betreff:[Assp-test] Socket poll cycle



 Hi,

 I've been trying to track down some performance issues lately. The load
 and CPU on the server is quite low so we have plenty of resources but I
 see this in the logs every day:

 Warning: the operating system socket poll cycle has taken
 3.10847902297974 seconds - this is very much is too long

 I'm also seeing iptables log incoming syn packets on port 25 that never
 get answered by ASSP and there is no evidence of them in the logs. I'm
 presuming there must be a configuration issue or something that can be
 optimised within ASSP to track this down but I've not been able to find
 anything appropriate.

 Thanks for suggestions,
 Colin.




 --
 Dive into the World of Parallel Programming. The Go Parallel Website,
 sponsored by Intel and developed in partnership with Slashdot Media, is
 your
 hub for all things parallel software development, from weekly thought
 leadership blogs to news, videos, case studies, tutorials and more. Take
 a
 look and join the conversation now. http://goparallel.sourceforge.net/
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test




 DISCLAIMER:
 ***
 This email and any files transmitted with it may be confidential,
 legally
 privileged and protected in law and are intended solely for the use of
 the
 individual to whom it is addressed.
 This email was multiple times scanned for viruses. There should be no
 known virus in this email

[Assp-test] Socket poll cycle

[Colin]

Hi,

I've been trying to track down some performance issues lately. The load 
and CPU on the server is quite low so we have plenty of resources but I 
see this in the logs every day:

Warning: the operating system socket poll cycle has taken 
3.10847902297974 seconds - this is very much is too long

I'm also seeing iptables log incoming syn packets on port 25 that never 
get answered by ASSP and there is no evidence of them in the logs. I'm 
presuming there must be a configuration issue or something that can be 
optimised within ASSP to track this down but I've not been able to find 
anything appropriate.

Thanks for suggestions,
Colin.


--
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] I'm sending messages from Yahoo?

[Colin]

 byte(s) sent to socket
 2014.12.11 10:24:27 LOG7[4403986432]: Remote socket (FD=11) closed
 2014.12.11 10:24:27 LOG7[4403986432]: Local socket (FD=10) closed
 2014.12.11 10:24:27 LOG7[4403986432]: Service [ssmtp] finished (0 left)

 So looks like the remote IP is 41.43.219.15 in this case (not our IP).

 James.

 On 11 Dec 2014, at 8:46 pm, Colin colin.war...@gmail.com wrote:

 Dec-11-14 10:23:56 [Worker_2] 127.0.0.1 info: authentication - plain is used

 This line gives me cause for concern for you. Something running on
 localhost sent or proxied this message AND used valid credentials to
 send the message.

 What do the collected emails show?  Are they definitely junk messages?
 If so you need to turn up logging to find out which credentials have
 been used and change those. Next step would be to see what process on
 localhost is passing these messages to ASSP and lock it down.

 I did a little bit of poking around on your IP to see if anything
 obvious stood out, but didn't want to do anything intrusive without
 asking. The only thing I can see is it looks like you have two different
 MTAs running. Port 25 responds with a Symantec banner and port 587
 responds with a Postfix banner. I'm not sure if one may be proxying and
 less secure but I didn't test.

 You could update OpenSSL that Apache is using from za to zc as there
 have been a lot of OpenSSL vulnerabilities this year. I don't know if
 that is likely to have any relevance though.

 On 11/12/2014 00:21, James Brown wrote:
 I’m a bit puzzled by this. I’ve noticed in the logs emails coming from and 
 going to email addresses that have nothing to do with my domain.

 Eg:

 Dec-11-14 10:23:53 [Worker_2] Connected: session:7FAD1B6519F8 
 127.0.0.1:51769  127.0.0.1:25  127.0.0.1:10026
 Dec-11-14 10:23:56 [Worker_2] 127.0.0.1 info: authentication - plain is used
 Dec-11-14 10:24:12 id-53842-01613 [Worker_2] [MessageOK] 127.0.0.1 
 cupra0...@yahoo.com to: mj.bas...@orange.fr message ok [Re Josette et 
 Michel Basset] - /Applications/assp/notspam/1613.eml
 Dec-11-14 10:24:14 [Worker_1] Finished message - received DATA size: 17.27 
 kByte - sent DATA size: 17.49 kByte
 Dec-11-14 10:24:14 [Worker_1] Disconnected: session:7FACFD3C7970 127.0.0.1 
 - processing time 62 seconds
 Dec-11-14 10:24:25 id-53858-12500 [Worker_2] [MessageOK] 127.0.0.1 
 cupra0...@yahoo.com to: mj.bur...@orange.fr message ok [To MJ Burgat] - 
 /Applications/assp/notspam/12500.eml
 Dec-11-14 10:24:26 [Worker_2] Finished message - received DATA size: 1.78 
 kByte - sent DATA size: 2.18 kByte
 Dec-11-14 10:24:26 [Worker_2] Disconnected: session:7FAD1B6519F8 127.0.0.1 
 - processing time 33 seconds

 My domain is bordo.com.au http://bordo.com.au/, not yahoo.com or 
 orange.fr http://orange.fr/.

 I’ve done external tests and they all show that I’m not an open relay.

 I think I need to remove 127.0.0.1 from acceptAllMail, and turn on 
 DoLocalSenderDomain.

 Does this sound right?

 Anything else I should look at?

 ASSP version 2.4.4(14343)

 Thanks,

 James.

 --
 Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
 from Actuate! Instantly Supercharge Your Business Reports and Dashboards
 with Interactivity, Sharing, Native Excel Exports, App Integration  more
 Get technology previously reserved for billion-dollar corporations, FREE
 http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test

 --
 Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
 from Actuate! Instantly Supercharge Your Business Reports and Dashboards
 with Interactivity, Sharing, Native Excel Exports, App Integration  more
 Get technology previously reserved for billion-dollar corporations, FREE
 http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test

 --
 Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
 from Actuate! Instantly Supercharge Your Business Reports and Dashboards
 with Interactivity, Sharing, Native Excel Exports, App Integration  more
 Get technology previously reserved for billion-dollar corporations, FREE
 http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT

Re: [Assp-test] Google drops NoTLS?

[Colin]

The SMTP error is from your MTA. Neither Google nor ASSP dropped this 
message. Your MTA rejected it with 502 command not implemented.

Have a look at those logs to see why.

All the best,
Colin Waring.

On 11/12/2014 13:55, Pontus Hellgren wrote:
 Hi there!

 Got some people complaining about not getting mail from domains hosted at
 googles mailservers.

 Made a fast check at the ASSP logs and found a bunch of these:
 Dec-11-14 14:44:23 [Worker_1] Connected: session:AA61610
 209.85.214.182:52540  x.x.x.x:25   y.y.y.y:125
 Dec-11-14 14:44:24 [Worker_1] 209.85.214.182 info: got STARTTLS request from
 209.85.214.182
 Dec-11-14 14:44:24 [Worker_1] 209.85.214.182 [SMTP Error] 502 command not
 implemented
 Dec-11-14 14:44:24 [Worker_1] Disconnected: session:AA61610 209.85.214.182 -
 processing time 1 seconds

 Is this ASSP dropping the connection for some reason or is Google being
 rude, not delivering the mail unless we implement TLS?

 Running ASSP version 2.4.4(14307) on Ubuntu 14.04.1 LTS

 Regards,
 Pontus



 --
 Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
 from Actuate! Instantly Supercharge Your Business Reports and Dashboards
 with Interactivity, Sharing, Native Excel Exports, App Integration  more
 Get technology previously reserved for billion-dollar corporations, FREE
 http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration  more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.4.4 build 14334

[Colin]

I spotted this one last night and it is to do with Net::SMTP::SSL version.

CPAN has version 1.01 of Net::SMTP::SSL and ASSP will not work with this.

Sourceforge has 1.02 so you need to download this to ASSPdir/lib/Net/SMTP

http://assp.cvs.sourceforge.net/viewvc/assp/assp2/lib/Net/SMTP/SSL.pm

ASSP will now start.

All the best,
Colin Waring.

On 01/12/2014 08:19, Pascal Dreissen wrote:
 Hi Thomas,

 This version is not starting up at all:

 Not enough arguments for Net::SMTP::DESTROY_SSLNSNot enough arguments for 
 Net::SMTP::assp_starttls at sub main::init line 311, near ()
   at assp.pl line 6286.

 Met vriendelijke groet / best regards,

 Pascal Dreissen
 applemooz

 On 30 Nov 2014, at 20:55, Thomas Eckardt thomas.ecka...@thockar.com wrote:

 Hi all,

 fixed in assp 2.4.4 build 14334:

 - improved DNS handling and debug

 - prevents now DNS query timeouts for all checks, if the sender domain is
 invalid

 - fixes a thrown exception
   Prototype mismatch: sub Net::SMTP::assp_starttls (-1) vs none at sub
 Net::SMTP::assp_starttls line 28.

 - in case of an connection error, the partial debug mode was unexpected
 enabled for some time


 changed:

 - the rebuild spamdb report contains some information about assp/tmpDB
   more information are available if ASSP_FC.pm is installed


 - the default value for DoRFC822 is changed from 'recipient' to  'sender'

 - the default value for DNStimeout is changed from '5' to  '2'

 - the default value for DNSretrans is changed from '2' to  '1'


 added:

 - the server information screen contains some more information about DNS
 query times


 Thomas


 DISCLAIMER:
 ***
 This email and any files transmitted with it may be confidential, legally
 privileged and protected in law and are intended solely for the use of the

 individual to whom it is addressed.
 This email was multiple times scanned for viruses. There should be no
 known virus in this email!
 ***

 --
 Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
 from Actuate! Instantly Supercharge Your Business Reports and Dashboards
 with Interactivity, Sharing, Native Excel Exports, App Integration  more
 Get technology previously reserved for billion-dollar corporations, FREE
 http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test

 --
 Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
 from Actuate! Instantly Supercharge Your Business Reports and Dashboards
 with Interactivity, Sharing, Native Excel Exports, App Integration  more
 Get technology previously reserved for billion-dollar corporations, FREE
 http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration  more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] fixes in assp 2.4.4 build 14331

[Colin]

Hi Thomas,

This one one of the things I noticed the other day although I 
misunderstood the logs then. I have every single collection option set 
because I want everything collected so no mail goes missing bar redlist. 
For that reason I do have DoNotCollectRedRe, DoNotCellectRedList and 
DoNotCollectBounces set.

In my case the following is logged with debugging on:

 2014-11-28 11:47:07 [Worker_1] doing line ###DATA#

 2014-11-28 11:47:07 [Worker_1] Maillog

 2014-11-28 11:47:07 [Worker_1] matchSL - sen...@domain.tld 
mailto:sen...@domain.tld - noCollecting

2014-11-28 11:47:07 m1-x-x [Worker_7] 1.1.1.1 sen...@domain.tld 
mailto:sen...@domain.tld to: recipi...@domain.tld 
mailto:recipi...@domain.tld info: Maillog - no log - log-condition is zero


noCollecting is completely blank for me (as is noCollectRe) so it would 
seem that this may be an incorrect match against that?

All the best,
Colin Waring.

On 01/12/2014 12:05, Thomas Eckardt wrote:
 log-condition is zero
 This means, that you don't collect all files - for this type of files
 'nocollect' is set.

 Thomas




 Von:aquilinux aquili...@gmail.com
 An: ASSP development mailing list assp-test@lists.sourceforge.net
 Datum:  01.12.2014 12:58
 Betreff:Re: [Assp-test] fixes in assp 2.4.4 build 14331



 Hi.
 I'm testing this feature in my test environment.
 no sure what i should see but when i set SessionLog to verbose in addition
 to normal log I see only a lot of:

 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] 199.103.5.118 
 sen...@out.com to: m...@mydomain.com info: Maillog - no log - log-condition
 is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] 199.103.5.118 
 sen...@out.com to: m...@mydomain.com info: Maillog - no log - log-condition
 is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] 199.103.5.118 
 sen...@out.com to: m...@mydomain.com info: Maillog - no log - log-condition
 is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] 199.103.5.118 
 sen...@out.com to: m...@mydomain.com info: Maillog - no log - log-condition
 is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] 199.103.5.118 
 sen...@out.com to: m...@mydomain.com info: Maillog - no log - log-condition
 is zero
 Dec-01-14 12:35:50 m1

Re: [Assp-test] fixes in assp 2.4.4 build 14331

[Colin]

All my logging settings. All either okmail, spam, notspam, quarantine or 
discard folder. So I do in fact have them all set to collect.

NonSpamLog:=2
baysNonSpamLog:=4
SpamLog:=1
noProcessingLog:=4
npAttachLog:=6
wlAttachLog:=6
extAttachLog:=6
SpamVirusLog:=5
spamBombLog:=1
scriptLog:=1
blDomainLog:=1
spamHeloLog:=1
forgedHeloLog:=1
invalidHeloLog:=1
spamBucketLog:=1
baysSpamLog:=1
SPFFailLog:=1
RBLFailLog:=1
URIBLFailLog:=1
SRSFailLog:=1
spamPTRLog:=1
spamMXALog:=1
spamISLog:=1
spamSBLog:=1
spamMSLog:=1
spamPBLog:=1
DKIMLog:=1
BackLog:=1
freqNonSpam:=1
freqSpam:=1


On 01/12/2014 12:55, Thomas Eckardt wrote:
 2014-11-28 11:47:07 [Worker_1] matchSL - sen...@domain.tld
 mailto:sen...@domain.tld - noCollecting

 this is a debug output - means matchSL checks  sen...@domain.tld for
 noCollecting

 if a match is found - it would be logged (see regexLogging)

 log-condition is zero
 shows that the logging condition for this mail is set to 'no collection'
 !!!

 I have every single collection option set
 because I want everything collected so no mail goes missing bar redlist.

 This seems not to be the case!

 Thomas




 Von:Colin colin.war...@gmail.com
 An: assp-test@lists.sourceforge.net
 Datum:  01.12.2014 13:37
 Betreff:Re: [Assp-test] fixes in assp 2.4.4 build 14331



 Hi Thomas,

 This one one of the things I noticed the other day although I
 misunderstood the logs then. I have every single collection option set
 because I want everything collected so no mail goes missing bar redlist.
 For that reason I do have DoNotCollectRedRe, DoNotCellectRedList and
 DoNotCollectBounces set.

 In my case the following is logged with debugging on:

   2014-11-28 11:47:07 [Worker_1] doing line ###DATA#

   2014-11-28 11:47:07 [Worker_1] Maillog

   2014-11-28 11:47:07 [Worker_1] matchSL - sen...@domain.tld
 mailto:sen...@domain.tld - noCollecting

 2014-11-28 11:47:07 m1-x-x [Worker_7] 1.1.1.1 sen...@domain.tld
 mailto:sen...@domain.tld to: recipi...@domain.tld
 mailto:recipi...@domain.tld info: Maillog - no log - log-condition is
 zero


 noCollecting is completely blank for me (as is noCollectRe) so it would
 seem that this may be an incorrect match against that?

 All the best,
 Colin Waring.

 On 01/12/2014 12:05, Thomas Eckardt wrote:
 log-condition is zero
 This means, that you don't collect all files - for this type of files
 'nocollect' is set.

 Thomas




 Von:aquilinux aquili...@gmail.com
 An: ASSP development mailing list assp-test@lists.sourceforge.net
 Datum:  01.12.2014 12:58
 Betreff:Re: [Assp-test] fixes in assp 2.4.4 build 14331



 Hi.
 I'm testing this feature in my test environment.
 no sure what i should see but when i set SessionLog to verbose in
 addition
 to normal log I see only a lot of:

 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log-condition is zero
 Dec-01-14 12:35:50 m1-33747-02917 [Worker_1] [TLS-out] [PTRinvalid]
 199.103.5.118 sen...@out.com to: m...@mydomain.com info: Maillog - no
 log
 -
 log

[Assp-test] Understanding logs/troubleshooting high load

[Colin]

Hi there,

I'm spending a lot of time trawling through logs at the moment. Our 
primary mail server is experiencing high load yet CPU usage is less than 
50%, memory less than 75% and disk usage minimal.

I've turned on debugging and set absolutely everything to maximum 
logging and have noticed some oddities that I could do with some help 
understanding. They could be normal, or could mean something.

After turning session logging to max I get loads of these, I'd say 
anywhere up to 50 each message:

2014-11-28 11:41:40 m1-x-x [Worker_7] 1.1.1.1 
sen...@domain.tld to: recipi...@domain.tld info: Maillog - no log - 
log-condition is zero

As far as I can tell, it logs one of these for each line of data that is 
processed as follows:

 2014-11-28 11:47:07 [Worker_1] doing line ###DATA#
 2014-11-28 11:47:07 [Worker_1] Maillog
 2014-11-28 11:47:07 [Worker_1] matchSL - sen...@domain.tld - noCollecting
2014-11-28 11:47:07 m1-x-x [Worker_7] 1.1.1.1 
sen...@domain.tld to: recipi...@domain.tld info: Maillog - no log - 
log-condition is zero

So this begs two questions, firstly when there is nothing to be logged, 
does it need to write a line saying so?

Secondly, is there a reason for matchSL running for every single line? 
My instinct is that the decision on whether to log or not based on 
sender address should be made once when the header is received and 
repeating it many times is wasting resources.

I also get a constant stream of these:

2014-11-28 11:40:54 [Worker_1] error
 2014-11-28 11:40:54 [Worker_1] error
 2014-11-28 11:40:54 [Worker_1] error
 2014-11-28 11:40:54 [Worker_1] error
 2014-11-28 11:40:54 [Worker_1] error
2014-11-28 11:40:54 [Worker_1] error

All workers except worker 1 log them once, immediately after a seterror 
and it seems to follow a greylisting rejection.

Worker 1 also logs occassional seterrors that match up with greylisting 
rejections. The oddity is that it also logs a continual stream of errors 
that don't seem to match up with anything.

I've tried to get some more information out of strace, but it leaves me 
equally puzzled.

I see a lot of EAGAIN (Resource temporarily unavailable), but no 
information about which resource is unavailable. I also understand that 
these are logged as normal behaviour when a process listens for 
connections on a TCP port and there is nothing there.

Monitoring file and network activity shows it all pretty low with every 
call being returned quickly.

Still, I see ASSP not responding to connections on the admin interface 
for minutes at a time. When I turned on debug mode, I turned it on at 
11:20:15.  It wasn't until 11:27:14 that ASSP wrote the following to the 
logs:

[Main_Thread] Info: starting partial debug mode to file 
/usr/local/assp/debug/1417174034.dbg

Any thoughts or suggestions on this one?

Thanks,
Colin

--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration  more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] unable to detect connected IP addresses

[Colin]

I think that I may finally have stumbled upon an error that is part of 
my troubles:

2014-11-28 17:15:17 [Worker_7] Worker_7 wakes up
2014-11-28 17:15:17 [Worker_7] Info: Worker_7 got connection from MainThread
2014-11-28 17:15:17 [Worker_7] Error: This system is some time unable to 
detect connected IP addresses - check that you use the latest C-library, 
Perl-version and Perl module versions
2014-11-28 17:15:17 [Worker_7] Error: unable to detect the remote 
connected IP address - localIP:port, 5.159.231.219:25 - remoteIP:port, : 
- local-socket,IO::Socket::INET=GLOB(0x7fe698fd1f30)
2014-11-28 17:15:17 [Main_Thread] Info: Main_Thread freed by idle 
Worker_7 in 0.126 seconds - got (ok)
2014-11-28 17:15:17 [Worker_7] Worker_7 will sleep now

System is Ubuntu 14.04 LTS fully up to date but for reference:

Perl(v5.18.2)
ldd (Ubuntu EGLIBC 2.19-0ubuntu6.3) 2.19

uname -a
Linux mail.smtphost.co.uk 3.13.0-39-generic #66-Ubuntu SMP Tue Oct 28 
13:30:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Perl modules were all updated last week with cpan-outdated -p|cpanm

--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration  more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Rebuild not completing - update

[Colin]

Hi Folks,

Answering my own question here. Finally got some time to investigate 
this one and it looks like the rebuild is needing more space in tmpDB.

It looks like when tmpDB runs out of space, the rebuild has not 
identified this. Instead it keep trying over and over again using 100% 
CPU and not doing anything with the insufficient disk space errors that 
are being returned. I've increased the memory allocated to tmpDB now.

Would it be possible to have the rebuild process do a periodic check on 
the tmpDB for free space? This check would be able to return an error to 
the rebuild log and possibly even raise a notification somewhere.

Even simpler could be a process at the end of the rebuild run that adds 
a line saying X amount of space was required in tmpDB for this rebuild 
run to complete. That would enable us to keep an eye on out for 
patterns in the usage and see quickly if the usage is increasing or 
approaching the size of the tmpDB.

All the best,
Colin Waring.

--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration  more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Rebuild not completing

[Colin]

Hi,

With the recent batch of updates I am finding that rebuildspamdb doesn't 
complete any more.

I've updated all the perl modules, made sure the lib/plugin folders are 
up to date etc.

The rebuild appears to run normally until it gets to:

2014-11-23 14:40:10 Generating consolidated Hidden-Markov-Model database 
from 9,396,082 record model

At this point it just stops. Nothing is output to the debug and the 
worker says add HMM sequences 3259000 but never progresses.

Any suggestions?

Cheers.

--
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration  more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] MTA says 552, ASSP doesn't pass on to client?

[Colin]

Hi Folks,

We've had this today filling the logs on our server.

Turns out the MTA is returning 552 but observing the SMTP session 
reveals that the 552 isn't getting back to the client. The client 
reports 421 connection dropped.

Client is Exchange 2010 using authenticated SMTP. The same session is 
used to deliver any other queued messages until it gets to this message 
where the session drops.

2014-11-11 09:48:13 m1-99293-00962 [Worker_4] [TLS-in] 1.1.1.1 
sen...@domain.tld info: found message size announcement: 67.12 MByte
2014-11-11 09:48:13 m1-99293-00962 [Worker_4] [TLS-in] 1.1.1.1 
sen...@domain.tld message proxied without processing - message size 
(70382399) is above 50 (npSizeOut).
2014-11-11 09:48:13 m1-99293-00962 [Worker_4] [TLS-in] 1.1.1.1 
sen...@domain.tld warning: got reply '552 Message size exceeds maximum 
permitted' from 127.0.0.1
2014-11-11 09:58:13 m1-99293-00962 [Worker_4] [TLS-in] 1.1.1.1 
sen...@domain.tld info: no (more) data readable from 1.1.1.1 
(connection closed by peer) - last command was 'MAIL FROM'
2014-11-11 09:58:13 [Worker_4] Disconnected: session:7FD9B0EFE5C0 
1.1.1.1 - command list was 'EHLO,STARTTLS,EHLO,AUTH,MAIL FROM,RCPT 
TO,DATA,MAIL FROM' - used 18 SocketCalls - processing time 604 seconds

Any ideas why the 552 wouldn't be passed on?

All the best,
Colin Waring.


--
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] RWL/DNSBL priority?

[Colin]

Hi All,

I'm wondering if this behaviour is correct. My understanding was that 
whitelisted IPs are excluded from other checks which includes the DNSBL. 
This message is in fact a legitimate message containing a gift 
certificate and I can't get a resend because it wasn't collected.

I have the following set:

ValidateRWL - 1
RWLwhitelisting - 1
RWLminhits - 1
ValidateRBL - Score
ForceRBLCache - 0
RBLWL - 0

2014-10-22 12:07:03 m1-76023-07040 [Worker_6] 195.140.184.159 
sen...@domain.tld [SMTP Reply] 250 OK
2014-10-22 12:07:03 m1-76023-07040 [Worker_6] 195.140.184.159 
sen...@domain.tld to: recipi...@domain.tld [SMTP Reply] 250 Accepted
2014-10-22 12:07:03 m1-76023-07040 [Worker_6] 195.140.184.159 
sen...@domain.tld to: recipi...@domain.tld [SMTP Reply] 354 Enter 
message, ending with . on a line by itself
2014-10-22 12:07:03 m1-76023-07040 [Worker_6] 195.140.184.159 
sen...@domain.tld to: recipi...@domain.tld Regex:Red 'autoreply'
2014-10-22 12:07:04 m1-76023-07040 [Worker_6] 195.140.184.159 
sen...@domain.tld to: recipi...@domain.tld Received-RWL: from 
(list.dnswl.org-127.0.15.0,trust=0 (category=Email Marketing 
Providers);) - high trust is 0 - client-ip=195.140.184.159
2014-10-22 12:07:04 m1-76023-07040 [Worker_6] 195.140.184.159 
sen...@domain.tld to: recipi...@domain.tld Message-Score: added 50 for 
DNSBL: failed, 195.140.184.159 listed in bl.mailspike.net, total score 
for this message is now 50
2014-10-22 12:07:04 m1-76023-07040 [Worker_6] 195.140.184.159 
sen...@domain.tld to: recipi...@domain.tld [scoring] (DNSBL: failed, 
195.140.184.159 listed in (bl.mailspike.net-127.0.0.11; ))
2014-10-22 12:07:04 m1-76023-07040 [Worker_6] 195.140.184.159 
sen...@domain.tld to: recipi...@domain.tld deleting spamming 
safelisted tuplet: (195.140.184.0,bounce.customerservice.mbna.co.uk) age: 1s
2014-10-22 12:07:04 m1-76023-07040 [Worker_6] [MessageLimit] 
195.140.184.159 sen...@domain.tld to: recipi...@domain.tld [spam 
found] (MessageScore 50, limit 50) [Amazon Gift Certificate];
2014-10-22 12:07:04 m1-76023-07040 [Worker_6] 195.140.184.159 
sen...@domain.tld to: recipi...@domain.tld [SMTP Error] 554 5.7.1 
Message not accepted - forward bounces to spamhelp@ smtphost.co.uk for 
assistance


--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Big increase in spam yesterday

[Colin]

Hi folks,

We saw a marked increase in spam getting through yesterday. I've finally 
got some time to compare the headers and noticed a bit of a problem.

Example message sent from russell...@ziggo.nl to a local user account 
mail...@domain.tld

The message was whitelisted incorrectly:

X-Assp-Whitelisted: Yes (whiteListedDomains '@domain.tld')

It seems we're getting a lot of messages where the whitelist check is 
incorrectly applying to the to: header causing them to be allowed.

I don't think anything has been changed on the system this week which 
would lead to this, so where next?

All the best,
Colin Waring.


--
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Running ASSP with MS Exchange?

[Colin]

Hi,

You're unlikely to get more complete notes from me - we have sufficient 
procedures for restoring our systems from backups and rebuilding from 
scratch and have other priorities at the moment I'm afraid.

We used this setup across a number of busy on site Exchange servers - I 
think the thing I missed out of the notes was the /etc/staticroutes 
file. This originated from someone else's way of routing messages using 
cPanel but I modified it several years back:

http://forums.cpanel.net/f43/exim-smart-relay-verification-123501.html#post538101

There is even a discussion in the archives for this lists going back a 
few months where I worked on getting the setup to cooperate with 
Microsoft's hosted Exchange. Be aware that hosted Exchange will not 
authenticate its outbound connections, you have to make a unique setup 
to accept mail from their IPs.

None of it seems messy to me and we can knock up a brand new mail relay 
instance within half an hour using those notes. We do have two ASSP 
instances and a separate MySQL box plus a load of configuration 
synchronisation scripts to automate things for us though. Once ASSP is 
in place you need to turn off spam filtering in Exchange otherwise 
you'll get people confused as to where spam goes.

All the best,
Colin Waring

On 24/09/2014 09:58, Pontus Hellgren wrote:
 Hi again!

 Thanks Colin, it's quick notes, but I get the ideea, I will surely get back
 to them when they are more complete. (since I have no time to laborate and
 test stuff before they work)
 I would love, if possible, more complete notes before I jump on and try your
 setup. (maybe in a later scenario)
 Keep me updated!

 And, we would/will/are hosting it ourselfs.

 The scenario for me is this: MS will(have) stopped supporting their product
 forefront which is used in another solution(hosted Exchange with multiple
 domains and servers) and I'm now evaluating what to replace forefront with.
 ASSP runs well in the much simpler solution (not running exchange, and with
 some tweaks) and we love it.
 Running ASSP in front of Exchange seems messy and seems bound to create
 trouble... or not!
 Problem is, it's a live environment so lots of changes needed(over all) are
 not welcome at the moment, I guess!

 Thanks for all input and suggestions!

 Regards,
 Pontus


 -Original Message-
 From: Colin [mailto:colin.war...@gmail.com]
 Sent: den 23 september 2014 14:13
 To: assp-test@lists.sourceforge.net
 Subject: Re: [Assp-test] Running ASSP with MS Exchange?

 How are you intending to run ASSP?

 Will it be hosted or on premises?

 We have a hosted solution where we run ASSP on a Ubuntu box with an Exim
 MTA. It sits in place like a normal relay without any special connectors or
 rules as follows:

 Internet - ASSP - Exim - Exchange on premises Exchange on premises -
 ASSP - Exim - Internet

 The Exchange box is completely firewalled off from the rest of the world and
 receives only TLS encrypted mail on port 25 from our ASSP IP.
 Exim is configured to do user validation and authentication. It calls
 forward to the Exchange box to validate the receipient before accepting it.
 Having Exim do authentication means that we can set our ASSP hostname as a
 simple outbound smart host with username/password authentication over a TLS
 connection again on port 25.

 One of the big advantages of Exim is that when the Exchange box or Internet
 goes down it will queue mail for the host. You can use a queue viewer to
 check your mail for anything important or even set an Exim filter that sends
 a copy of your mail to a backup address for the duration of an outage - we
 had this yesterday when the whole area around our office had a power cut for
 most of the day.

 I've been meaning to put some of our info back to the list to help others
 out for a while. We have a lot of other edits, for example our Exim auth is
 synced from our hosting platform as is the localdomains files. You can find
 a copy of my setup notes here:
 http://www.dolphinict.co.uk/Ubuntu-ASSP.txt I hope people find them useful.
 You will need to understand Linux to use them and you will need to do
 additional configuration to get things working, my config is mostly for the
 back end of the system and I haven't included any notes on configuring ASSP
 itself past the init script.

 All the best,
 Colin Waring.

 -Original Message-
 From: Pontus Hellgren [mailto:pon...@scandinavianhosting.se]
 Sent: September 22, 2014 02:50
 To: 'ASSP development mailing list'
 Subject: [Assp-test] Running ASSP with MS Exchange?

 This is a request for information about how to run ASSP with Exchange and
 no error report.
 Please redirect this if there is another list for it!

 * Any caveats to avoid? (what not to do or what to actually do to not
 get in trouble with MS Exchange)
 * Any new links with setup information for ASSP running in front of
 MS Exchange
 * Any useful information.

 I ask since I have been asked to do a testrun for a case, but I

Re: [Assp-test] Running ASSP with MS Exchange?

[Colin]

How are you intending to run ASSP?

Will it be hosted or on premises?

We have a hosted solution where we run ASSP on a Ubuntu box with an Exim 
MTA. It sits in place like a normal relay without any special connectors 
or rules as follows:

Internet - ASSP - Exim - Exchange on premises
Exchange on premises - ASSP - Exim - Internet

The Exchange box is completely firewalled off from the rest of the world 
and receives only TLS encrypted mail on port 25 from our ASSP IP.
Exim is configured to do user validation and authentication. It calls 
forward to the Exchange box to validate the receipient before accepting 
it. Having Exim do authentication means that we can set our ASSP 
hostname as a simple outbound smart host with username/password 
authentication over a TLS connection again on port 25.

One of the big advantages of Exim is that when the Exchange box or 
Internet goes down it will queue mail for the host. You can use a queue 
viewer to check your mail for anything important or even set an Exim 
filter that sends a copy of your mail to a backup address for the 
duration of an outage - we had this yesterday when the whole area around 
our office had a power cut for most of the day.

I've been meaning to put some of our info back to the list to help 
others out for a while. We have a lot of other edits, for example our 
Exim auth is synced from our hosting platform as is the localdomains 
files. You can find a copy of my setup notes here: 
http://www.dolphinict.co.uk/Ubuntu-ASSP.txt I hope people find them 
useful. You will need to understand Linux to use them and you will need 
to do additional configuration to get things working, my config is 
mostly for the back end of the system and I haven't included any notes 
on configuring ASSP itself past the init script.

All the best,
Colin Waring.

 -Original Message-
 From: Pontus Hellgren [mailto:pon...@scandinavianhosting.se]
 Sent: September 22, 2014 02:50
 To: 'ASSP development mailing list'
 Subject: [Assp-test] Running ASSP with MS Exchange?

 This is a request for information about how to run ASSP with Exchange and no 
 error report.
 Please redirect this if there is another list for it!

 * Any caveats to avoid? (what not to do or what to actually do to not get in 
 trouble with MS Exchange)
 * Any new links with setup information for ASSP running in front of MS 
 Exchange
 * Any useful information.

 I ask since I have been asked to do a testrun for a case, but I have limited 
 time so I do not wanna do one or manny trail and error runs.

 Thanks in advance,
 Pontus



 --
 Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI 
 DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you 
 Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI 
 DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer 
 http://pubads.g.doubleclick.net/gampad/clk?id=154622311iu=/4140/ostg.clktrk
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test

 --
 Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
 Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
 Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
 Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
 http://pubads.g.doubleclick.net/gampad/clk?id=154622311iu=/4140/ostg.clktrk
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Problem with syncing passwords

[Colin]

When I moved from a Ubuntu 12.04 to Ubuntu 14.04 system I noticed that 
the following values weren't syncing correctly:

msgid
batv
adminusersdb
adminusersdbpass
myuser
mypassword
notify
notifyre
ssh_cipher_list

My solution at that point was to turn off sync and restore the .cfg so 
everything went back to how it was. I was part way through migrating our 
infrastructure to a new datacentre so it got shelved in the notes as 
working. look at later. It only surfaced again yesterday when one of 
our other guys went to log on to the secondary to look at some stats and 
couldn't.

In this case, the only value that I experimented with was 
webAdminPassword, I haven't touched any other password values.

Do you think I'm likely to have anything not working?
All the best,
Colin Waring.

On 02/09/2014 17:50, Thomas Eckardt wrote:
 It looks like master sends the password hash
 to the secondary whilst the secondary expects the unencrypted password.
 YES.

 What a BUG.

 The slave sets the password to the string shown in the masters GUI

 You can log into the secondary if you open up assp.cfg and
 copy out the password hash.
 If you has already done it this way - you has lost ALL encrypted config
 variables and the adminusersdb at the slave - sorry!

 Thomas





 Von:Colin colin.war...@gmail.com
 An: ASSP development mailing list assp-test@lists.sourceforge.net
 Datum:  02.09.2014 17:36
 Betreff:[Assp-test] Problem with syncing passwords



 Hi,

 I discovered that if you use the sync to keep passwords up to date
 between servers it doesn't work.

 Take a simple two server setup Master - Secondary and give each server
 the same password. Set up the sync to include the password (web
 interface) and then change the password on the master.

 You can no longer log into the secondary with either the old or new
 password. You can log into the secondary if you open up assp.cfg and
 copy out the password hash. It looks like master sends the password hash
 to the secondary whilst the secondary expects the unencrypted password.

 All the best,
 Colin Waring.

 --
 Slashdot TV.
 Video for Nerds.  Stuff that matters.
 http://tv.slashdot.org/
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test





 DISCLAIMER:
 ***
 This email and any files transmitted with it may be confidential, legally
 privileged and protected in law and are intended solely for the use of the

 individual to whom it is addressed.
 This email was multiple times scanned for viruses. There should be no
 known virus in this email!
 ***

 --
 Slashdot TV.
 Video for Nerds.  Stuff that matters.
 http://tv.slashdot.org/
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] ASSP_OCR stuck workers question

[Colin]

I have the same issue but never got round to investigating.

I've just tried to turn ASSP_OCR on and only use it for PDFs and I find 
that I can't. I untick procWhiteASSP_OCR, DoSimpleTextASSP_OC, 
DoPDFImageASSP_OCR and DoImageASSP_OCR.

When I click apply it reloads the page and they're all ticked again. 
assp.cfg values all show 1. If I amend the .cfg manually the values get 
over-written.

I had to turn it off again as most of my threads got stuck within a few 
minutes of turning it on.

All the best,
Colin Waring.

On 02/09/2014 15:55, Thomas Eckardt wrote:
 Don't use the image processing in the OCR plugin - only use the PDF
 processing.
 This issue seems to be related to a thread semaphore problem - I'll fix
 this, if I'll find the time.
 I plan to remove the image processing from this plugin until end of this
 year.

 Why does the documentation say
 ASSP_OCRocrmaxprocesses should be less than the number of cpu cores?
 OCR (ImageMagick + tesseract) will use 100% of one core per concurrent
 processed image until it has finished.
 I expect a maximum of 0.05% better spam detection using the OCR-image
 processing - for an up to 95% higher CPU usage, this is t less.

 It is much more efficient to use the 'ASSP_AFCDetectSpamAttachRe' in the
 ASSP_AFC plugin.

 Thomas



 Von:Dirk Kulmsee d.kulm...@netgroup.de
 An: assp-test@lists.sourceforge.net
 Datum:  02.09.2014 16:27
 Betreff:[Assp-test] ASSP_OCR  stuck workers question



 Hi,

 I am currently running ASSP 2.4.4 (14241) on Debian Linux with Perl 5.20.
 The ASSP_OCR module is 2.18.

 I had all worker processes stuck in ASSP_OCR one by one:

   

 2014-09-02 10:59:21 [Main_Thread] Info: Loop in Worker_1 was not active
 for
 461 seconds

 2014-09-02 10:59:21 [Main_Thread] Info: Worker_1 : last sigoff in
 ASSP_OCR,
 /opt/assp/Plugins/ASSP_OCR.pm, 282, main::sigoffTry, 1, , ,  at 14-2-8
 10:51:40 1409647900.23592 - 282

 2014-09-02 10:59:21 [Main_Thread] Info: Worker_1 : last sigon in main, sub
 main::URIBLok, 15, main::URIBLok_Run, 1, , ,  at 14-2-8 10:51:40
 1409647900.2248 - 272

 2014-09-02 10:59:21 [Main_Thread] Info: Worker_1 : last action was : call
 Plugin ASSP_OCR with

 2014-09-02 10:59:21 [Main_Thread] Warning: try to terminate
 inactive/stucking Worker_1

 2014-09-02 11:19:26 [Main_Thread] Info: Loop in Worker_2 was not active
 for
 466 seconds

 2014-09-02 11:19:26 [Main_Thread] Info: Worker_2 : last sigoff in
 ASSP_OCR,
 /opt/assp/Plugins/ASSP_OCR.pm, 282, main::sigoffTry, 1, , ,  at 14-2-8
 11:11:40 1409649100.27879 - 282

 2014-09-02 11:19:26 [Main_Thread] Info: Worker_2 : last sigon in main, sub
 main::URIBLok, 15, main::URIBLok_Run, 1, , ,  at 14-2-8 11:11:40
 1409649100.26713 - 241

 2014-09-02 11:19:26 [Main_Thread] Info: Worker_2 : last action was : call
 Plugin ASSP_OCR with

 2014-09-02 11:19:26 [Main_Thread] Warning: try to terminate
 inactive/stucking Worker_2

 2014-09-02 11:36:11 [Main_Thread] Info: Loop in Worker_3 was not active
 for
 271 seconds

 2014-09-02 11:36:11 [Main_Thread] Info: Worker_3 : last sigoff in
 ASSP_OCR,
 /opt/assp/Plugins/ASSP_OCR.pm, 282, main::sigoffTry, 1, , ,  at 14-2-8
 11:31:40 1409650300.57724 - 282

 2014-09-02 11:36:11 [Main_Thread] Info: Worker_3 : last sigon in main, sub
 main::URIBLok, 15, main::URIBLok_Run, 1, , ,  at 14-2-8 11:31:40
 1409650300.56076 - 241

 2014-09-02 11:36:11 [Main_Thread] Info: Worker_3 : last action was : call
 Plugin ASSP_OCR with

 2014-09-02 11:36:11 [Main_Thread] Warning: try to terminate
 inactive/stucking Worker_3

 2014-09-02 13:49:57 [Main_Thread] Info: Loop in Worker_4 was not active
 for
 196 seconds

 2014-09-02 13:49:57 [Main_Thread] Info: Worker_4 : last sigoff in
 ASSP_OCR,
 /opt/assp/Plugins/ASSP_OCR.pm, 282, main::sigoffTry, 1, , ,  at 14-2-8
 13:46:41 1409658401.38248 - 282

 2014-09-02 13:49:57 [Main_Thread] Info: Worker_4 : last sigon in main, sub
 main::URIBLok, 15, main::URIBLok_Run, 1, , ,  at 14-2-8 13:46:41
 1409658401.36525 - 241

 2014-09-02 13:49:57 [Main_Thread] Info: Worker_4 : last action was : call
 Plugin ASSP_OCR with

 2014-09-02 13:49:57 [Main_Thread] Warning: try to terminate
 inactive/stucking Worker_4

   

 Later I found a live example for this. A simple email status report
 containing four little PNG icons stuck the worker process, leaving log
 lines
 like these:

   

 2014-09-02 13:59:26 m1-59166-11063 [Worker_1] [Plugin] 88.198.3.4 [OIP:
 81.209.171.97] server2...@someone.de to: al...@mydomain.de ASSP_OCR:
 (att)
 file text1.ecelp9600 found in mime part 1

 2014-09-02 13:59:26 m1-59166-11063 [Worker_1] [Plugin] 88.198.3.4 [OIP:
 81.209.171.97] server2...@someone.de to: al...@mydomain.de ASSP_OCR:
 (att)
 file logo.png found in mime part 2

 2014-09-02 13:59:26 m1-59166-11063 [Worker_1] [Plugin] 88.198.3.4 [OIP:
 81.209.171.97] server2...@someone.de to: al...@mydomain.de ASSP_OCR:
 processing (attatched) file logo.png

 2014-09-02 13:59:26 m1-59166-11063 [Worker_1] [Plugin] 88.198.3.4 [OIP:
 81.209.171.97

[Assp-test] Problem with syncing passwords

[Colin]

Hi,

I discovered that if you use the sync to keep passwords up to date 
between servers it doesn't work.

Take a simple two server setup Master - Secondary and give each server 
the same password. Set up the sync to include the password (web 
interface) and then change the password on the master.

You can no longer log into the secondary with either the old or new 
password. You can log into the secondary if you open up assp.cfg and 
copy out the password hash. It looks like master sends the password hash 
to the secondary whilst the secondary expects the unencrypted password.

All the best,
Colin Waring.

--
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Office 365 High Risk Delivery Pool

[Colin]

Hi folks,

I was wondering if someone could assist me with a bit of testing.

Apparently Microsoft route any suspicious outbound emails through a 
specific set of IP addresses so that they don't risk tarnishing the 
reputation of the main Office 365 emails. That is quite a clever idea. 
Unfortunately  am finding that emails going via the HRDP do not get 
delivered to our ASSP servers. The error that comes back in the bounce 
message is a Winsock error from their mailservers and they are telling 
me that our servers are not responding to their connection request.

I do not believe this is the case as ASSP doesn't log anything from the 
IP address concerned at all, however there is a possibility that 
additional security software might be dropping it (denyhosts, fail2ban etc).

The issue I am having is that customers forward the ASSP block report to 
me asking for something to be adjusted and I can't reply unless I remove 
the report from the reply, so I would like to ask for a couple of 
volunteers. I would like to send a copy of one of the failing emails to 
a couple of addresses on different servers that run ASSP to see if they 
all fail or if it is something specific to our configuration. If anyone 
is willing to help, please send me your email off list. I will send you 
and email requesting your reply if it comes through.

All the best,
Colin Waring.

--
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] FW: Email interface kicking in on external mail?

[Colin]

Hi Thomas,

The mail flow is this:

Outbound OC - HE - ASSP - Internet
Inbound Internet - ASSP - HE - OC

Inbound works fine as we can set up an inbound connector on Office 365 
and tell it to accept mail for specific domains from our ASSP IP address.
Outbound is the issue. HE communicates uses outbound connectors. The 
only thing you can configure in an outbound connector is the IP address 
it delivers to. There is no ability to specify a username and password, 
there is no ability to specify a different port.

In the end, I have assigned an extra IP address to the ASSP server. I 
have bound the normal traffic to the main IP, port 25 and bound the 
relay port to the second IP, port 25. I've made sure that the second IP 
is locked down. The data centre firewall, iptables and allowRelayCon are 
configured to only accept port 25 mail locally or from the IP blocks 
that Microsoft use.

The only improvement I could make would be to limit the sender domains 
allowed by connections to relayPort.

All the best,
Colin Waring.


On 11/08/2014 12:39, Thomas Eckardt wrote:
 Collin,

 the infrastructure behind your Office 365 implementation is still unclear
 to me.
 It does not matter if this szenario is used by an ISP or a local company.

 assuming the following:

 - you have local Office 365 clients - OC
 - you have a local assp instance -assp
 - you have a hosted Exchange 365 instance - HE

 Where local means 'local' in terms of assp - this could be any client and
 assp in the world. All OC's should connect to assp using the 'relayPort'
 or the 'listenPort2'. Foreign connections should go to the assp
 'listenPort'.

 OC is getting mails from HE using POP3 - that's clear to me
 OC (and local printers/faxmachines/scanners/notifyers) sends all mails
 (local and outgoing) to assp and assp forwards the mails to HE using TLS
 (and injected AUTH for the local
 printers/faxmachines/scanners/notifyers) - that's clear to me
 Because assp should scan incoming foreign mails for spam, the domain MX
 points to assp - assp forwards the good mails to a local MTA(forwarder),
 which sends the mails to the HE .


 get ASSP and Office 365 talking seen as Office 365 can't do outbound
 authentication

 Now the question:
   
 - all OC must (IMHO) use TLS and AUTH to connect to the HE directly - why
 they can't do this through assp?
 - in which case  the HE is connecting to assp via SMTP - the only case
 where AUTH will be a problem ?

 Please help me to understand the problem - it seems that you do something
 different?

 Thomas


 Von:Colin colin.war...@gmail.com
 An: ASSP development mailing list assp-test@lists.sourceforge.net
 Datum:  09.08.2014 12:07
 Betreff:Re: [Assp-test] FW:  Email interface kicking in on
 external mail?



 Thanks for the clarification.

 This was an attempt to get ASSP and Office 365 talking seen as Office
 365 can't do outbound authentication. Unfortunately it has meant that
 anyone using Office 365 was treated as a local user which is something
 that we cannot have so I will have to take it all out and find another
 solution to Office 365.

 My personal preference for the email interface would be to be able to
 restrict it and just have it work on a defined domain (ie smtphost.co.uk
 for us) but if you're happy with just the requirement to define unique
 addresses then that's OK as it is your software!

 All the best,
 Colin Waring.

 -Original Message-
 From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
 Sent: 08 August 2014 11:07
 To: ASSP development mailing list
 Subject: Re: [Assp-test] Email interface kicking in on external mail?

 don't use 'acceptAllMail' for foreign IP's - I never used it for any IP,
 because it is an old legacy problematic feature - use the 'relayPort'
 instead
 I know, that is must be used in some cases for local IP's. For example ,
 if you can't define the destination-port for a SMTP-server in another
 application (report/notifications).

 'assphelp' is the default for 'EmailHelp'

   From the GUI:

 Enable Email Interface (EmailInterfaceOk)  •
 Checked means that you want ASSP to intercept and parse mail to the
 following usernames at any localdomains. The domain '@assp.local' is
 automatically a local domain and can be used for the email-interface.
 read:   'at any localdomains' 

 How ever - IP's connected to the relayPort are authenticated to relay
 and
 to use the emailinterface .

 The usernames used in the emailinterface/BlockReport have to be unique
 for
 all local domains - this is a simple conclution - and every username
 should show, what it is used for.
 This requires no additionally exception lists or definitions - only a
 clear setup.

 Thomas






 --
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test



 DISCLAIMER

Re: [Assp-test] FW: Email interface kicking in on external mail?

[Colin]

Hi Thomas,

Thanks for the suggestion however all our hosted customers do in fact 
use RPC over HTTP as this allows them to make full use of hosted 
Exchange contacts, tasks, calendar etc which IMAP/SMTP would not.

All on premises Exchange solutions allow multiple ways of authenticating 
connections and changing port numbers. I'm astounded that Microsoft have 
stripped that functionality out of their hosted product, maybe they have 
done it to try and force everyone to use their own security/archiving 
solutions or maybe they're just useless!

I did indeed use that list of addresses to set all my allow 
lists/firewalls. Annoyingly however there is no method to subscribe and 
receive notifications if it changes. The first thing we know about a 
change is when mail starts bouncing (as it did when they added an IP 
block at the beginning of July).

I had a conversation on the phone with a guy from Microsoft this 
morning, he repeated himself for about 10 minutes confirming the 
limitations of their product and that I had found a workaround then said 
he'd put me through to a manager to discuss how they could improve the 
hosted service so this wouldn't be an issue. I then got some lovely hold 
music for a while before I hung up. Presumably they don't really want my 
feedback!

At least we have a workaround that achieves the job now so we can look 
at putting ASSP in place for other clients.

All the best,
Colin Waring.

On 11/08/2014 15:03, Thomas Eckardt wrote:
 Collin,

 what speaks against to switch the outbound from

 OC - HE - ASSP - Internet

 to

 OC - ASSP(relayPort) - HE - Internet

 I know, this looks not very common, but it will work. It protects the HE
 also from a possibly hijacked OC (if assp is configured to do so).
 I think it is possible to define an DNS delivery outbound connector in
 the HE like in any other Exchange server.
 I only know one reason that speaks against this switching - the OC uses
 IMAP or RPC or RPC over HTTP or HTTP. But it should be possible to
 configure the OC to use IMAP for all except outbound mail, which
 should be configured to use SMTP (-assp).

 There is no ability to specify a username and password,
 there is no ability to specify a different port.
 Are we back in the good old Exchange 4 times ??? Even an 11 years old
 SBS2003 could do it better.

 If you can't do the switch for any reason, you can define an IP-address
 group for the Office 365 EOP ranges like:
 (possibly you've already done it)

 (from
 http://technet.microsoft.com/en-us/library/dn163583(v=exchg.150).aspx)

 [Office365EOP]
 65.55.88.0/24
 94.245.120.64/26
 207.46.51.64/26
 207.46.163.0/24
 213.199.154.0/24
 213.199.180.128/26
 216.32.180.0/24
 216.32.181.0/24
 2a01:111:f400:7c00::/54

 These addresses are the only and are only used for customer O365 relay
 connection by microsoft.

 The group definition [Office365EOP] could than be used anywhere you can
 define IP-addresses and ranges in assp.

 Splitting the IP's of assp in to public and privat is fine. How ever, you
 need to configure assp to check the local sender address
 ('DoLocalSenderAddress') for outbound mails - but at least for local
 domains in the sender address ('DoLocalSenderDomain') - to prevent other
 HE admins in the world to use your assp as an open relay-host - as you
 said!

 Never define any EOP in 'acceptAllMail' ! (allowRelayCon is OK - really
 good)

 I don't know if it is possible to fake the sender address/domain in Office
 365 - if so, this would be very very problematic - you'll be lost, if
 there is no way to sign or to tag a relayed mail.
 Collin - have a look in to some O365 outbound mails. MS has every time
 written some sender/domain - unique tags or X-headers or something like
 that in there mails, if they were processed by an Exchange server. If we
 can find something like that, it would be relative easy to implement a
 'validateOffice365' in the Relay section of ASSP. This could be also the
 first 'Received:' line (the last down from top).

 Just another idea. I know MS uses SSL or TLS for the customer relay
 connections. Are you able to define your own certificate/key for the relay
 connection in the HE ? If so, V2 is able to verify the client certificate
 and to drop wrong connections. I'm afraid, nobody at MS thought thus far,
 because on the short way they lost the relay port anyway

 Thomas



 Von:Colin colin.war...@gmail.com
 An: assp-test@lists.sourceforge.net
 Datum:  11.08.2014 14:01
 Betreff:Re: [Assp-test] FW:  Email interface kicking in on
 external mail?



 Hi Thomas,

 The mail flow is this:

 Outbound OC - HE - ASSP - Internet
 Inbound Internet - ASSP - HE - OC

 Inbound works fine as we can set up an inbound connector on Office 365
 and tell it to accept mail for specific domains from our ASSP IP address.
 Outbound is the issue. HE communicates uses outbound connectors. The
 only thing you can configure in an outbound connector is the IP address
 it delivers

Re: [Assp-test] FW: Email interface kicking in on external mail?

[Colin]

Thanks for the clarification.

This was an attempt to get ASSP and Office 365 talking seen as Office 
365 can't do outbound authentication. Unfortunately it has meant that 
anyone using Office 365 was treated as a local user which is something 
that we cannot have so I will have to take it all out and find another 
solution to Office 365.

My personal preference for the email interface would be to be able to 
restrict it and just have it work on a defined domain (ie smtphost.co.uk 
for us) but if you're happy with just the requirement to define unique 
addresses then that's OK as it is your software!

All the best,
Colin Waring.

 -Original Message-
 From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
 Sent: 08 August 2014 11:07
 To: ASSP development mailing list
 Subject: Re: [Assp-test] Email interface kicking in on external mail?

 don't use 'acceptAllMail' for foreign IP's - I never used it for any IP,
 because it is an old legacy problematic feature - use the 'relayPort'
 instead
 I know, that is must be used in some cases for local IP's. For example ,
 if you can't define the destination-port for a SMTP-server in another
 application (report/notifications).

 'assphelp' is the default for 'EmailHelp'

  From the GUI:

 Enable Email Interface (EmailInterfaceOk)  •
 Checked means that you want ASSP to intercept and parse mail to the
 following usernames at any localdomains. The domain '@assp.local' is
 automatically a local domain and can be used for the email-interface.
 read:   'at any localdomains' 

 How ever - IP's connected to the relayPort are authenticated to relay and
 to use the emailinterface .

 The usernames used in the emailinterface/BlockReport have to be unique for
 all local domains - this is a simple conclution - and every username
 should show, what it is used for.
 This requires no additionally exception lists or definitions - only a
 clear setup.

 Thomas







--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Email interface kicking in on external mail?

[Colin]

OK, thanks for the clarification Thomas.

I have now confirmed that the email itself was sent to 
h...@customerdomain.tld rather than i...@customerdomain.tld

I always thought that the email interface applied only to the defined 
server domain or @assp.local. I'm surprised we haven't had clashes with 
email addresses crop up before so I've changed the help@ to assphelp@

Would it be possible to have an extra option which we can enable and 
over-ride the defaults? If not enabled then it would use localdomains. 
If enabled then it would contain a list of domains in a file and the 
email interface only works for those domains? This would prevent any 
possibility of users setting up an email address that clashes.

All the best,
Colin Waring.

On 07/08/2014 19:46, Thomas Eckardt wrote:
 Do acceptAllMail IPs somehow become classified as local or
 authenticated


 YES

 accepAllMail - means accept ALL mails.from these IP's
 IP addresses in 'acceptAllMail' and IP's connecting to the 'relayPort'
 have to follow the assp rules.

 Thomas







 Von:Colin colin.war...@gmail.com
 An: ASSP development mailing list assp-test@lists.sourceforge.net
 Datum:  07.08.2014 19:51
 Betreff:[Assp-test] Email interface kicking in on external mail?



 Hi Folks,

 I've had a report of mail not coming through to the recipient. The logs
 indicate that ASSP has incorrectly intercepted the message as a request
 for a help document on the email interface. The info for the help
 address says that it should only intercept local and authenticated
 requests. Do acceptAllMail IPs somehow become classified as local or
 authenticated even when they aren't?

 The to address in this case would have been i...@customerdomain.tld
 whilst the actual help address is h...@ourlocaldomain.tld. This message
 should never have been intercepted by the ASSP email interface, yet it
 was.

 There is no connection debug file for this one either and the log
 changed id half way through:

 2014-08-05 10:51:41 [Worker_6] 1.1.1.1 IP 1.1.1.1 matches acceptAllMail
 - with 213.199.154.0/24
 2014-08-05 10:51:41 [Worker_6] Connected: session:7F943CBAA0C0
 1.1.1.1:18903  5.159.231.219:25  127.0.0.1:57685  127.0.0.1:125 ,
 111-113
 2014-08-05 10:51:41 [Worker_6] 1.1.1.1 [SMTP Reply] 220
 mail.smtphost.co.uk ESMTP Exim 4.82 Ubuntu Tue, 05 Aug 2014 10:51:41 +0100
 2014-08-05 10:51:41 [Worker_6] 1.1.1.1 info: injected '250-STARTTLS'
 offer in to EHLO reply
 2014-08-05 10:51:41 [Worker_6] 1.1.1.1 info: send '250-STARTTLS' -
 injected for 127.0.0.1
 2014-08-05 10:51:41 [Worker_6] 1.1.1.1 [SMTP Reply] 250 HELP
 2014-08-05 10:51:41 [Worker_6] 1.1.1.1 info: got STARTTLS request from
 1.1.1.1
 2014-08-05 10:51:42 [Worker_6] 1.1.1.1 [SMTP Reply] 220 Ready to start TLS
 2014-08-05 10:51:42 [Worker_6] [TLS-in] 1.1.1.1 info: started TLS-SSL
 session for client 1.1.1.1
 2014-08-05 10:51:42 [Worker_6] [TLS-in] 1.1.1.1 [SMTP Reply] 250 HELP
 2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1
 sen...@domain.tld info: found message size announcement: 29.85 kByte
 2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1
 sen...@domain.tld [SMTP Reply] 250 OK
 2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1
 sen...@domain.tld email help
 2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1
 sen...@domain.tld [SMTP Reply] 250 OK
 2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1
 sen...@domain.tld [SMTP Reply] 354 OK Send help body
 2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1
 sen...@domain.tld report-header: no addresses found in header tags
 2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1
 sen...@domain.tld report-body: no addresses found in header tags
 2014-08-05 10:51:42 m1-32302-09130 [Worker_6] [TLS-in] 1.1.1.1 [SMTP
 Reply] 250 OK
 2014-08-05 10:51:42 m1-32302-09130 [Worker_6] [TLS-in] 1.1.1.1 [SMTP
 Reply] 250 Reset OK
 2014-08-05 10:51:42 m1-32302-09130 [Worker_6] [TLS-in] 1.1.1.1 warning:
 IO::Socket::SSL=GLOB(0x7f943cbaa0c0) got writeerror - Connection reset
 by peer -
 2014-08-05 10:51:42 m1-32302-09130 [Worker_6] [TLS-in] 1.1.1.1 info: no
 (more) data readable from 1.1.1.1 (connection closed by peer) - last
 command was 'QUIT'
 2014-08-05 10:51:43 [Worker_6] Disconnected: session:7F943CBAA0C0
 1.1.1.1 - command list was 'EHLO,STARTTLS,EHLO,MAIL FROM,RCPT
 TO,DATA,QUIT' - used 10 SocketCalls - processing time 2 seconds


 All the best,
 Colin Waring.

 --
 Infragistics Professional
 Build stunning WinForms apps today!
 Reboot your WinForms applications with our WinForms controls.
 Build a bridge from your legacy apps to the future.
 http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk

 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test





 DISCLAIMER

[Assp-test] Email interface kicking in on external mail?

[Colin]

Hi Folks,

I've had a report of mail not coming through to the recipient. The logs 
indicate that ASSP has incorrectly intercepted the message as a request 
for a help document on the email interface. The info for the help 
address says that it should only intercept local and authenticated 
requests. Do acceptAllMail IPs somehow become classified as local or 
authenticated even when they aren't?

The to address in this case would have been i...@customerdomain.tld 
whilst the actual help address is h...@ourlocaldomain.tld. This message 
should never have been intercepted by the ASSP email interface, yet it was.

There is no connection debug file for this one either and the log 
changed id half way through:

2014-08-05 10:51:41 [Worker_6] 1.1.1.1 IP 1.1.1.1 matches acceptAllMail 
- with 213.199.154.0/24
2014-08-05 10:51:41 [Worker_6] Connected: session:7F943CBAA0C0 
1.1.1.1:18903  5.159.231.219:25  127.0.0.1:57685  127.0.0.1:125 , 111-113
2014-08-05 10:51:41 [Worker_6] 1.1.1.1 [SMTP Reply] 220 
mail.smtphost.co.uk ESMTP Exim 4.82 Ubuntu Tue, 05 Aug 2014 10:51:41 +0100
2014-08-05 10:51:41 [Worker_6] 1.1.1.1 info: injected '250-STARTTLS' 
offer in to EHLO reply
2014-08-05 10:51:41 [Worker_6] 1.1.1.1 info: send '250-STARTTLS' - 
injected for 127.0.0.1
2014-08-05 10:51:41 [Worker_6] 1.1.1.1 [SMTP Reply] 250 HELP
2014-08-05 10:51:41 [Worker_6] 1.1.1.1 info: got STARTTLS request from 
1.1.1.1
2014-08-05 10:51:42 [Worker_6] 1.1.1.1 [SMTP Reply] 220 Ready to start TLS
2014-08-05 10:51:42 [Worker_6] [TLS-in] 1.1.1.1 info: started TLS-SSL 
session for client 1.1.1.1
2014-08-05 10:51:42 [Worker_6] [TLS-in] 1.1.1.1 [SMTP Reply] 250 HELP
2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1 
sen...@domain.tld info: found message size announcement: 29.85 kByte
2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1 
sen...@domain.tld [SMTP Reply] 250 OK
2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1 
sen...@domain.tld email help
2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1 
sen...@domain.tld [SMTP Reply] 250 OK
2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1 
sen...@domain.tld [SMTP Reply] 354 OK Send help body
2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1 
sen...@domain.tld report-header: no addresses found in header tags
2014-08-05 10:51:42 m1-32302-00497 [Worker_6] [TLS-in] 1.1.1.1 
sen...@domain.tld report-body: no addresses found in header tags
2014-08-05 10:51:42 m1-32302-09130 [Worker_6] [TLS-in] 1.1.1.1 [SMTP 
Reply] 250 OK
2014-08-05 10:51:42 m1-32302-09130 [Worker_6] [TLS-in] 1.1.1.1 [SMTP 
Reply] 250 Reset OK
2014-08-05 10:51:42 m1-32302-09130 [Worker_6] [TLS-in] 1.1.1.1 warning: 
IO::Socket::SSL=GLOB(0x7f943cbaa0c0) got writeerror - Connection reset 
by peer -
2014-08-05 10:51:42 m1-32302-09130 [Worker_6] [TLS-in] 1.1.1.1 info: no 
(more) data readable from 1.1.1.1 (connection closed by peer) - last 
command was 'QUIT'
2014-08-05 10:51:43 [Worker_6] Disconnected: session:7F943CBAA0C0 
1.1.1.1 - command list was 'EHLO,STARTTLS,EHLO,MAIL FROM,RCPT 
TO,DATA,QUIT' - used 10 SocketCalls - processing time 2 seconds


All the best,
Colin Waring.

--
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071iu=/4140/ostg.clktrk
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Trying to upgrade ASSPv2

[Colin]

Hi Doug,

I did a quick Google and it looks like you posted the same issue about a 
year ago:

http://www.zimbra.com/forums/administrators/63140-sslv3-alert-bad-record-mac.html

Do you recall what you did to resolve it then? From the other hits out 
there is seems that postfix can be particuraly unforgiving to AES 
connections (there are other people talking about having problems 
receiving email from Google via AES with Postfix).

I had RC4-SHA:HIGH as my preferred cipher after the beast SSL attacks, 
however with some of the latest revelations I should probably look at 
bringing AES back in.

Anyone else care to share their cipher lists and a quick explanation of 
why they picked it?

Mine is currently the following and this way because it was the only way 
to gain PCI compliance after the beast SSL attack.

RC4-SHA:HIGH:!ADH:!AES256-SHA:!ECDHE-RSA-AES256-SHA384:!AES128-SHA:!DES-CBC3-SHA:!DES-CBC3-MD5:!IDEA-CBC-SHA:!RC4-MD5:!IDEA-CBC-MD5:!RC2-CBC-MD5:!MD5:!aNULL:!EDH:!AESGCM

All the best,
Colin Waring.

On 20/07/2014 14:09, Doug Lytle wrote:
 I have a very old install of ASSPv2 2.3.4(13136) running on Debian
 GNU/Linux 6.0.3 (squeeze).

 This is for our Zimbra mail server that is also outdated, running on
 Ubuntu 10.04 64bit.  I'd like to update the mail server, but won't
 attempt it until I get the ASSP2 issues resolved.

 When building another VM to house the upgraded ASSP and putting it into
 place, I get attachment corruption.  Following the logs on the Zimbra
 side, I see a change in what is being used for the SSL cipher.  It goes
 from the normal:

 postfix/smtpd[12152]: Anonymous TLS connection established from
 assp.inet[10.0.0.10]: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)

 To:

 postfix/smtpd[11502]: Anonymous TLS connection established from
 assp.inet[10.0.0.10]: TLSv1 with cipher AES128-SHA (128/128 bits)

 So,

 Reviewing a previous post from Thomas

 http://sourceforge.net/p/assp/mailman/message/31259064/

 I started playing around with the cipher options on ASSP.  I forced:

 AES256:SHA256:RC4-SHA:HIGH:!ADH

 Now my logs on the Zimbra server so AES256 and I no longer have
 attachment corruption, but I now am experiencing two different issues.

 1.)  Sending test email from Seamonkey, I may have to hit send a couple
 times before it goes.
 2.)  I'm seeing the below logs in my Zimbra server:

 postfix/smtpd[22112]: warning: TLS library problem:
 22112:error:140943FC:SSL routines:SSL3_READ_BYTES:sslv3 alert bad record
 mac:s3_pkt.c:1199:SSL alert number 20

 Would this be because I'm missing a required cipher?

 Any suggestions would be appreciated.

 Doug

 --
 Want fast and easy access to all the code in your enterprise? Index and
 search up to 200,000 lines of code with a free copy of Black Duck
 Code Sight - the same software that powers the world's largest code
 search on Ohloh, the Black Duck Open Hub! Try it now.
 http://p.sf.net/sfu/bds
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Trying to upgrade ASSPv2

[Colin]

Hi Doug,

I've not had any issues with the ciphers I have listed but they should 
really be considered out of date. RC4 was recommended to avoid beast 
attacks however now most clients are patched against those attacks and 
new attacks have come out against RC4. It isn't completely insecure but 
I need to consider switching to a better list hence asking what other 
people use.

All the best,
Colin Waring.

On 21/07/2014 10:49, Doug Lytle wrote:
 Colin wrote:
 Do you recall what you did to resolve it then?
 I never did.

 People were starting to get the perception that our mail server was
 having issues, having it down so much trying to figure this out, so I
 left it alone for almost 8 months.  Figured I'd give it another try.

 I'll be playing around with the ciphers you've listed again, this
 upcoming Sunday morning.  Thanks for the input!



 --
 Want fast and easy access to all the code in your enterprise? Index and
 search up to 200,000 lines of code with a free copy of Black Duck
 Code Sight - the same software that powers the world's largest code
 search on Ohloh, the Black Duck Open Hub! Try it now.
 http://p.sf.net/sfu/bds
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test

--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Timeout issues

[Colin]

This is almost correct.

The problem appears to be that the ASSP timeout is not being honoured 
and it is falling back to the MTA.

I have smtpIdleTimeout set to 360 whilst the MTA timeout is set to 
14400. It is the MTA timeout that is being triggered yet ASSP logs 
TLS-Connection idle for 360 secs - timeout when the MTA drops the 
connection at 14400 seconds.

So far, I haven't had any more reports of messages being delayed or not 
getting through because of the other issue so it looks like disabling 
TLS between ASSP and Exim has completely gotten rid of the issue.

I suspect this is a different issue and it is just that Amazon's servers 
don't close the connections when they're done - presumably so they can 
charge their cloud customers for them!

All the best,
Colin Waring.

On 16/07/2014 05:31, Thomas Eckardt wrote:
 Now I get the whole session through to the end of DATA and the trailing .
 within a few seconds
 ASSP logs it and then leaves the connection open but does nothing
 The MTA then times out the connection after 14400s
 So I assume the following SMTP command sequence

 .
 MTA-ASSP-CLIENT: 354 send
 CLIENT-ASSP-MTA: data until[CR][LF].[CR][LF]
 MTA-ASSP-CLIENT: 250 queued in ..

 At this point the client has the following options

 RSET
 MAIL FROM:
 QUIT
 NOOP
 HELP

 If nothing is sent by the client, the connection will run in to a timeout

 Thomas




 Von:Colin colin.war...@gmail.com
 An: ASSP development mailing list assp-test@lists.sourceforge.net
 Datum:  15.07.2014 10:41
 Betreff:Re: [Assp-test] Timeout issues



 Hi Spyros,

 ASSP still does TLS for incoming connections. The only thing that I have
 disabled is the MTA (Exim) receiving inbound TLS connections - it will
 still send outbound emails via TLS. This means that the only affected
 connections are between ASSP and Exim and this occurs on the local
 loopback address. Exim does not even listen on any external interfaces.
 The only security risk is someone with access to the box being able to
 run tcpdump and by that point we're in serious trouble anyway!

 I looked into the tcpdump again yesterday and got further baffled. It
 seems that disabling TLS has changed the issue somewhat. The original
 issue was that ASSP would receive the message and deliver it to the MTA.
 Something would happen and the connection would go idle at the end of
 DATA until the MTA timed it out at 400s.

 Now I get the whole session through to the end of DATA and the trailing
 . within a few seconds. The MTA sends the OK and queue id number back to
 ASSP, ASSP logs it and then leaves the connection open but does nothing
 with it. The MTA then times out the connection after 14400s. The message
 has long since been delivered. Interestingly, it seems that Amazon Web
 Services IP addresses are responsible for the majority of these odd
 sessions in the logs.

 The only changes I have made are to disable TLS between ASSP and Exim as
 above and to increase the timeout from 400s to 14400s.

 All the best,

 Colin Waring.



 On 15/07/2014 09:25, Spyros Tsiolis wrote:
 
 On Sat, 12/7/14, Colin colin.war...@gmail.com wrote:

Subject: Re: [Assp-test] Timeout issues
To: assp-test@lists.sourceforge.net
Date: Saturday, 12 July, 2014, 21:32

Hi All,

Good news. Disabling TLS on
the mta has resolved the issue completely.
There isn't any idle time on the
connections any more and I've observed
a previously affected server (unable to deliver
a message to us for a
couple of days) send
through on its first retry attempt.

I'm not sure what the issue is as I am
using the same Exim config as I
have always
used. It could be Exim, it could be ASSP but I'm happy
with
TLS off as both are on the same box
communicating over the loopback
interface.

All
the best,
Coin Waring
 hi Colin,

 but now isn't your system open to attacks ?
 Since disabling TLS ?
 I mean between the mua's  (the clients) and the box that houses
 your mta and assp ?

 Just wondering.

 s.


 --
 Want fast and easy access to all the code in your enterprise? Index and
 search up to 200,000 lines of code with a free copy of Black Duck
 Code Sight - the same software that powers the world's largest code
 search on Ohloh, the Black Duck Open Hub! Try it now.
 http://p.sf.net/sfu/bds
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test




 DISCLAIMER:
 ***
 This email and any files transmitted with it may be confidential, legally
 privileged and protected in law and are intended solely for the use of the

 individual to whom it is addressed.
 This email was multiple times scanned for viruses. There should be no
 known virus

Re: [Assp-test] Timeout issues

[Colin]

Hi Spyros,

ASSP still does TLS for incoming connections. The only thing that I have 
disabled is the MTA (Exim) receiving inbound TLS connections - it will 
still send outbound emails via TLS. This means that the only affected 
connections are between ASSP and Exim and this occurs on the local 
loopback address. Exim does not even listen on any external interfaces. 
The only security risk is someone with access to the box being able to 
run tcpdump and by that point we're in serious trouble anyway!

I looked into the tcpdump again yesterday and got further baffled. It 
seems that disabling TLS has changed the issue somewhat. The original 
issue was that ASSP would receive the message and deliver it to the MTA. 
Something would happen and the connection would go idle at the end of 
DATA until the MTA timed it out at 400s.

Now I get the whole session through to the end of DATA and the trailing 
. within a few seconds. The MTA sends the OK and queue id number back to 
ASSP, ASSP logs it and then leaves the connection open but does nothing 
with it. The MTA then times out the connection after 14400s. The message 
has long since been delivered. Interestingly, it seems that Amazon Web 
Services IP addresses are responsible for the majority of these odd 
sessions in the logs.

The only changes I have made are to disable TLS between ASSP and Exim as 
above and to increase the timeout from 400s to 14400s.

All the best,

Colin Waring.



On 15/07/2014 09:25, Spyros Tsiolis wrote:
 
 On Sat, 12/7/14, Colin colin.war...@gmail.com wrote:

   Subject: Re: [Assp-test] Timeout issues
   To: assp-test@lists.sourceforge.net
   Date: Saturday, 12 July, 2014, 21:32
   
   Hi All,
   
   Good news. Disabling TLS on
   the mta has resolved the issue completely.
   There isn't any idle time on the
   connections any more and I've observed
   a previously affected server (unable to deliver
   a message to us for a
   couple of days) send
   through on its first retry attempt.
   
   I'm not sure what the issue is as I am
   using the same Exim config as I
   have always
   used. It could be Exim, it could be ASSP but I'm happy
   with
   TLS off as both are on the same box
   communicating over the loopback
   interface.
   
   All
   the best,
   Coin Waring

 hi Colin,

 but now isn't your system open to attacks ?
 Since disabling TLS ?
 I mean between the mua's  (the clients) and the box that houses
 your mta and assp ?

 Just wondering.

 s.



--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Timeout issues

[Colin]

Hi All,

Bad news. Disabling TLS on the mta did not resolve the issue. I'm now 
seeing timeouts after half an hour instead of 400s after increasing the 
timeout.

Time to figure up tcpdump again!

All the best,
Colin Waring

On 12/07/2014 19:32, Colin wrote:
 Hi All,

 Good news. Disabling TLS on the mta has resolved the issue completely. 
 There isn't any idle time on the connections any more and I've 
 observed a previously affected server (unable to deliver a message to 
 us for a couple of days) send through on its first retry attempt.

 I'm not sure what the issue is as I am using the same Exim config as I 
 have always used. It could be Exim, it could be ASSP but I'm happy 
 with TLS off as both are on the same box communicating over the 
 loopback interface.

 All the best,
 Coin Waring

 On 12/07/2014 16:38, Colin wrote:
 Further to this, I've finally managed to capture the communication 
 between ASSP and Exim with tcpdump. Unfortunately I forgot to disable 
 SSL so I can't see the actual SMTP commands but I can see the behaviour.

 ASSP Takes a total of 1.9 seconds to send the message to Exim. The 
 last packet is an ACK from Exim to ASSP at 1.905104s.

 There is then nothing for 400 seconds until at 401.906762 when Exim 
 starts talking again and tells ASSP that it has timed out.

 I've turned off TLS in Exim and will rerun the capture to see if I 
 can get the unencrypted content of the message. I can't seem to get 
 wireshark to decrypt the TLS stream even though I've given it a copy 
 of the key.

 I've also increased the timeout in Exim on the off chance that ASSP 
 is for some reason taking a long time to do something before 
 finishing off the conversation.

 Now I just need to wait for an affected connection to come back in 
 again!

 On 11/07/2014 20:31, Colin wrote:
 Hi again,

 Further on this - the issue doesn't seem to be there when the server 
 first boots. After I while I start seeing these in the logs, not 
 sure how related they are.

 2014-07-11 20:14:24 [Worker_1] Warning: got unexpected signal 
 CONT in Worker_1: package - main, file - sub 
 main::ThreadMaintMain, line - 457!

 The problem creeps in and always seems to affect the same senders 
 but the messages do eventually get through.

 ASSP will not cleanly shut down. If I kill the process and remove 
 the pid file ASSP will start back up, listen on the ports and appear 
 to accept connections but nothing happens or appears in the log. I 
 have to reboot to get the ports back.

 All the best,
 Colin Waring.



 On 10/07/2014 17:13, Nigel Kukard wrote:
 On 07/10/2014 09:18 AM, Thomas Eckardt wrote:
 pipelining
 The pipelining extension has nothing to do with the behavior to 
 simply
 deliver multiple emails within one connection.

 pipelining is related to SMTP commands not mails - this SMTP 
 extension
 makes it possible to send multiple SMTP commands without waiting 
 for a
 reply after each one
 You're right, I clearly misunderstood the extension :)


 --
  

 Open source business process management suite built on Java and 
 Eclipse
 Turn processes into business applications with Bonita BPM Community 
 Edition
 Quickly connect people, data, and systems into organized workflows
 Winner of BOSSIE, CODIE, OW2 and Gartner awards
 http://p.sf.net/sfu/Bonitasoft
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test





--
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck#174;
Code Sight#153; - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Timeout issues

[Colin]

Further to this, I've finally managed to capture the communication 
between ASSP and Exim with tcpdump. Unfortunately I forgot to disable 
SSL so I can't see the actual SMTP commands but I can see the behaviour.

ASSP Takes a total of 1.9 seconds to send the message to Exim. The last 
packet is an ACK from Exim to ASSP at 1.905104s.

There is then nothing for 400 seconds until at 401.906762 when Exim 
starts talking again and tells ASSP that it has timed out.

I've turned off TLS in Exim and will rerun the capture to see if I can 
get the unencrypted content of the message. I can't seem to get 
wireshark to decrypt the TLS stream even though I've given it a copy of 
the key.

I've also increased the timeout in Exim on the off chance that ASSP is 
for some reason taking a long time to do something before finishing off 
the conversation.

Now I just need to wait for an affected connection to come back in again!

On 11/07/2014 20:31, Colin wrote:
 Hi again,

 Further on this - the issue doesn't seem to be there when the server 
 first boots. After I while I start seeing these in the logs, not sure 
 how related they are.

 2014-07-11 20:14:24 [Worker_1] Warning: got unexpected signal CONT 
 in Worker_1: package - main, file - sub main::ThreadMaintMain, 
 line - 457!

 The problem creeps in and always seems to affect the same senders but 
 the messages do eventually get through.

 ASSP will not cleanly shut down. If I kill the process and remove the 
 pid file ASSP will start back up, listen on the ports and appear to 
 accept connections but nothing happens or appears in the log. I have 
 to reboot to get the ports back.

 All the best,
 Colin Waring.



 On 10/07/2014 17:13, Nigel Kukard wrote:
 On 07/10/2014 09:18 AM, Thomas Eckardt wrote:
 pipelining
 The pipelining extension has nothing to do with the behavior to simply
 deliver multiple emails within one connection.

 pipelining is related to SMTP commands not mails - this SMTP extension
 makes it possible to send multiple SMTP commands without waiting for a
 reply after each one
 You're right, I clearly misunderstood the extension :)


 --
  

 Open source business process management suite built on Java and Eclipse
 Turn processes into business applications with Bonita BPM Community 
 Edition
 Quickly connect people, data, and systems into organized workflows
 Winner of BOSSIE, CODIE, OW2 and Gartner awards
 http://p.sf.net/sfu/Bonitasoft
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test



--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Timeout issues

[Colin]

Hi All,

Good news. Disabling TLS on the mta has resolved the issue completely. 
There isn't any idle time on the connections any more and I've observed 
a previously affected server (unable to deliver a message to us for a 
couple of days) send through on its first retry attempt.

I'm not sure what the issue is as I am using the same Exim config as I 
have always used. It could be Exim, it could be ASSP but I'm happy with 
TLS off as both are on the same box communicating over the loopback 
interface.

All the best,
Coin Waring

On 12/07/2014 16:38, Colin wrote:
 Further to this, I've finally managed to capture the communication 
 between ASSP and Exim with tcpdump. Unfortunately I forgot to disable 
 SSL so I can't see the actual SMTP commands but I can see the behaviour.

 ASSP Takes a total of 1.9 seconds to send the message to Exim. The 
 last packet is an ACK from Exim to ASSP at 1.905104s.

 There is then nothing for 400 seconds until at 401.906762 when Exim 
 starts talking again and tells ASSP that it has timed out.

 I've turned off TLS in Exim and will rerun the capture to see if I can 
 get the unencrypted content of the message. I can't seem to get 
 wireshark to decrypt the TLS stream even though I've given it a copy 
 of the key.

 I've also increased the timeout in Exim on the off chance that ASSP is 
 for some reason taking a long time to do something before finishing 
 off the conversation.

 Now I just need to wait for an affected connection to come back in again!

 On 11/07/2014 20:31, Colin wrote:
 Hi again,

 Further on this - the issue doesn't seem to be there when the server 
 first boots. After I while I start seeing these in the logs, not sure 
 how related they are.

 2014-07-11 20:14:24 [Worker_1] Warning: got unexpected signal 
 CONT in Worker_1: package - main, file - sub 
 main::ThreadMaintMain, line - 457!

 The problem creeps in and always seems to affect the same senders but 
 the messages do eventually get through.

 ASSP will not cleanly shut down. If I kill the process and remove the 
 pid file ASSP will start back up, listen on the ports and appear to 
 accept connections but nothing happens or appears in the log. I have 
 to reboot to get the ports back.

 All the best,
 Colin Waring.



 On 10/07/2014 17:13, Nigel Kukard wrote:
 On 07/10/2014 09:18 AM, Thomas Eckardt wrote:
 pipelining
 The pipelining extension has nothing to do with the behavior to simply
 deliver multiple emails within one connection.

 pipelining is related to SMTP commands not mails - this SMTP extension
 makes it possible to send multiple SMTP commands without waiting for a
 reply after each one
 You're right, I clearly misunderstood the extension :)


 --
  

 Open source business process management suite built on Java and Eclipse
 Turn processes into business applications with Bonita BPM Community 
 Edition
 Quickly connect people, data, and systems into organized workflows
 Winner of BOSSIE, CODIE, OW2 and Gartner awards
 http://p.sf.net/sfu/Bonitasoft
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test




--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Timeout issues

[Colin]

Hi again,

Further on this - the issue doesn't seem to be there when the server 
first boots. After I while I start seeing these in the logs, not sure 
how related they are.

2014-07-11 20:14:24 [Worker_1] Warning: got unexpected signal CONT 
in Worker_1: package - main, file - sub main::ThreadMaintMain, line 
- 457!

The problem creeps in and always seems to affect the same senders but 
the messages do eventually get through.

ASSP will not cleanly shut down. If I kill the process and remove the 
pid file ASSP will start back up, listen on the ports and appear to 
accept connections but nothing happens or appears in the log. I have to 
reboot to get the ports back.

All the best,
Colin Waring.



On 10/07/2014 17:13, Nigel Kukard wrote:
 On 07/10/2014 09:18 AM, Thomas Eckardt wrote:
 pipelining
 The pipelining extension has nothing to do with the behavior to simply
 deliver multiple emails within one connection.

 pipelining is related to SMTP commands not mails - this SMTP extension
 makes it possible to send multiple SMTP commands without waiting for a
 reply after each one
 You're right, I clearly misunderstood the extension :)


 --
 Open source business process management suite built on Java and Eclipse
 Turn processes into business applications with Bonita BPM Community Edition
 Quickly connect people, data, and systems into organized workflows
 Winner of BOSSIE, CODIE, OW2 and Gartner awards
 http://p.sf.net/sfu/Bonitasoft
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: Re: Timeout issues

[Colin]

Hi,

I think the debug logs have thrown me. The full connection debug log 
shows the receipt of the message and the delivery of the message to the 
MTA sequentially which is why I initially thought it was receiving two 
messages.

I've enabled Exim to write a copy of selected messages to a folder to 
see what Exim says. As far as I can tell it does not capture affected 
messages as they do not get to the end of the delivery and exim doesn't 
have a way to capture a full debug the way ASSP does. I've added a 300GB 
drive and am running tcpdump -i lo to it. Maybe that will tell me 
exactly what is going on if I can find an affected session in amongst 
the general chatter!

All the best,
Colin Waring.


On 10/07/2014 09:03, Nigel Kukard wrote:
 On 07/10/2014 06:27 AM, Thomas Eckardt wrote:
 I don't believe ASSP is currently able to modify the capability list to
 remove the PIPELINING capability.


 This is done as long as I know assp.
 You are indeed right.

 believe  ?
 You don't have to believe anything - you can check this by comparing the
 reply from the MTA with the reply of assp.


 ASSP removes all not supported SMTP offers from EHLO an HELP replies.
 Simple mistake, I telnet'ed to the wrong port. Sorry.

 It still doesn't explain why Colin is seeing multiple delivery attempts
 over the same connection if pipelining is not being listed as a
 capability. Is the correct behavior not to reject the user of a feature
 which is not listed as supported?

 Have you tried turning off pipelining in your MTA? you will at then get
 it rejected by the MTA.



 --
 Open source business process management suite built on Java and Eclipse
 Turn processes into business applications with Bonita BPM Community Edition
 Quickly connect people, data, and systems into organized workflows
 Winner of BOSSIE, CODIE, OW2 and Gartner awards
 http://p.sf.net/sfu/Bonitasoft
 ___
 Assp-test mailing list
 Assp-test@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/assp-test


--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Timeout issues

[Colin]

Thanks Thomas,

I should know by now to check the debug options! It has already caught 
four since I turned it on and they all say:

2014-07-09 10:30:36 client Timeout after 360 secs
2014-07-09 10:30:36 client was readable
2014-07-09 10:30:36 client was not writable

I've set connection logging to diagnostic to see if it pulls any more 
specific information on it.

All the best,
Colin Waring.

On 09/07/2014 10:21, Thomas Eckardt wrote:
 enable 'ConTimeOutDebug'

 assp will write the connection states in to files in the debug folder.

 Thomas





 Von:Colin colin.war...@gmail.com
 An: Assp-test@lists.sourceforge.net
 Datum:  08.07.2014 17:04
 Betreff:Re: [Assp-test] Timeout issues



 Apologies for replying to self, turns out the log lines are out of sync
 by a second so my greps weren't showing it up.

 The MTA is indeeed showing SMTP data timeout (message abandoned) on
 connection from so I'll change my report. The connection is being
 opened to the MTA but it looks like ASSP is not passing the message to
 the MTA after scanning it and getting message ok. The result is then
 either the MTA or ASSP timing out, I think the timeouts are set to the
 same so either one could kick in first - that explains difference
 between the last two logs.

 On 08/07/2014 15:50, Colin wrote:
 Hi Folks,

 I've given up on the list accepting messages from my main account. SF
 didn't respond to my request for help either.

 I have had an ongoing issue for a good few weeks now where its seems
 like ASSP accepts a message but does not pass it on to the MTA. It
 talks to the remote host and goes through the motions. There is no
 evidence at all in the MTA logs that ASSP has even opened a connection
 to it for the message concerned. When I originally spotted the
 problem, ASSP didn't output anything for the message past message ok.

 One of the recent revisions changed that and now ASSP reports a
 timeout, but I have seen it reported in two different ways - logs below.

 Example 1 - no timeout reported by ASSP

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld info: found
 message size announcement: 13.54 kByte

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld [SMTP Reply] 250
 OK
 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 250
 Accepted

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 354
 Enter message, ending with . on a line by itself

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld info: found known
 good HELO 'mout1.freenet.de' - weight is -2

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -40 for KnownGoodHelo, total score for this message is now -40

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -15 (pbwValencePB) for In Penalty White Box, total score for
 this message is now -55

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -5 for 195.4.92 in griplist (0.00), total score for this message
 is now -60

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [scoring] SPF: none
 ip=1.1.1.1 mailfrom=sen...@domain.tld
 mailto:mailfrom=sen...@domain.tld helo=mout1.freenet.de

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Whitelisted sender
 address: sen...@domain.tld mailto:sen...@domain.tld for recipient
 recipi...@domain.tld mailto:recipi...@domain.tld

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [Plugin] calling
 plugin ASSP_AFC

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld whitelisted (no bad
 attachments)

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld

Re: [Assp-test] Timeout issues

[Colin]

Hi Thomas,

I think I have spotted something. I'm seeing loads being written to the 
debug files now. In all cases that I have checked the affected messages 
are from a connection where multiple emails have been delivered.

The first email comes through absolutely fine, then the connection is 
kept open for the second email. The second email is received but times 
out. This may explain why I could find the opening of the connection for 
affected emails and why not all messages from particular senders are 
affected.

Is there a setting I can set in ASSP to have it accept one message and 
disconnect, forcing the remote MTA to open a new connection for each 
message? It isn't ideal but would allow me to establish that the issue 
is definitely limited to these kinds of connections.

All the best,
Colin Waring.


On 09/07/2014 10:21, Thomas Eckardt wrote:
 enable 'ConTimeOutDebug'

 assp will write the connection states in to files in the debug folder.

 Thomas





 Von:Colin colin.war...@gmail.com
 An: Assp-test@lists.sourceforge.net
 Datum:  08.07.2014 17:04
 Betreff:Re: [Assp-test] Timeout issues



 Apologies for replying to self, turns out the log lines are out of sync
 by a second so my greps weren't showing it up.

 The MTA is indeeed showing SMTP data timeout (message abandoned) on
 connection from so I'll change my report. The connection is being
 opened to the MTA but it looks like ASSP is not passing the message to
 the MTA after scanning it and getting message ok. The result is then
 either the MTA or ASSP timing out, I think the timeouts are set to the
 same so either one could kick in first - that explains difference
 between the last two logs.

 On 08/07/2014 15:50, Colin wrote:
 Hi Folks,

 I've given up on the list accepting messages from my main account. SF
 didn't respond to my request for help either.

 I have had an ongoing issue for a good few weeks now where its seems
 like ASSP accepts a message but does not pass it on to the MTA. It
 talks to the remote host and goes through the motions. There is no
 evidence at all in the MTA logs that ASSP has even opened a connection
 to it for the message concerned. When I originally spotted the
 problem, ASSP didn't output anything for the message past message ok.

 One of the recent revisions changed that and now ASSP reports a
 timeout, but I have seen it reported in two different ways - logs below.

 Example 1 - no timeout reported by ASSP

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld info: found
 message size announcement: 13.54 kByte

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld [SMTP Reply] 250
 OK
 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 250
 Accepted

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 354
 Enter message, ending with . on a line by itself

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld info: found known
 good HELO 'mout1.freenet.de' - weight is -2

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -40 for KnownGoodHelo, total score for this message is now -40

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -15 (pbwValencePB) for In Penalty White Box, total score for
 this message is now -55

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -5 for 195.4.92 in griplist (0.00), total score for this message
 is now -60

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [scoring] SPF: none
 ip=1.1.1.1 mailfrom=sen...@domain.tld
 mailto:mailfrom=sen...@domain.tld helo=mout1.freenet.de

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Whitelisted sender
 address: sen...@domain.tld mailto:sen...@domain.tld for recipient
 recipi...@domain.tld mailto:recipi...@domain.tld

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi

Re: [Assp-test] Timeout issues

[Colin]

Hi Thomas,

Further to this I set a 1 message limit on the MTA. The number of 
affected connections seems to have reduced but I am still seeing 
timeouts being logged on individual messages. The only extra thing the 
full connection debug shows after the end of receiving the data is 
server unpoll.

2014-07-09 10:47:23 server unpoll from main sub main::ThreadMain 165
2014-07-09 10:53:24 client Timeout after 360 secs^M
2014-07-09 10:53:24 client was readable^M
2014-07-09 10:53:24 client was not writable^M

All the best,
Colin Waring.

On 09/07/2014 10:21, Thomas Eckardt wrote:
 enable 'ConTimeOutDebug'

 assp will write the connection states in to files in the debug folder.

 Thomas





 Von:Colin colin.war...@gmail.com
 An: Assp-test@lists.sourceforge.net
 Datum:  08.07.2014 17:04
 Betreff:Re: [Assp-test] Timeout issues



 Apologies for replying to self, turns out the log lines are out of sync
 by a second so my greps weren't showing it up.

 The MTA is indeeed showing SMTP data timeout (message abandoned) on
 connection from so I'll change my report. The connection is being
 opened to the MTA but it looks like ASSP is not passing the message to
 the MTA after scanning it and getting message ok. The result is then
 either the MTA or ASSP timing out, I think the timeouts are set to the
 same so either one could kick in first - that explains difference
 between the last two logs.

 On 08/07/2014 15:50, Colin wrote:
 Hi Folks,

 I've given up on the list accepting messages from my main account. SF
 didn't respond to my request for help either.

 I have had an ongoing issue for a good few weeks now where its seems
 like ASSP accepts a message but does not pass it on to the MTA. It
 talks to the remote host and goes through the motions. There is no
 evidence at all in the MTA logs that ASSP has even opened a connection
 to it for the message concerned. When I originally spotted the
 problem, ASSP didn't output anything for the message past message ok.

 One of the recent revisions changed that and now ASSP reports a
 timeout, but I have seen it reported in two different ways - logs below.

 Example 1 - no timeout reported by ASSP

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld info: found
 message size announcement: 13.54 kByte

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld [SMTP Reply] 250
 OK
 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 250
 Accepted

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 354
 Enter message, ending with . on a line by itself

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld info: found known
 good HELO 'mout1.freenet.de' - weight is -2

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -40 for KnownGoodHelo, total score for this message is now -40

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -15 (pbwValencePB) for In Penalty White Box, total score for
 this message is now -55

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -5 for 195.4.92 in griplist (0.00), total score for this message
 is now -60

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [scoring] SPF: none
 ip=1.1.1.1 mailfrom=sen...@domain.tld
 mailto:mailfrom=sen...@domain.tld helo=mout1.freenet.de

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Whitelisted sender
 address: sen...@domain.tld mailto:sen...@domain.tld for recipient
 recipi...@domain.tld mailto:recipi...@domain.tld

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [Plugin] calling
 plugin ASSP_AFC

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld whitelisted (no bad
 attachments)

 2014

Re: [Assp-test] Timeout issues

[Colin]

Hi Thomas,

It does not appear to be, in fact I have found one message that was 
written to the discard folder then the AFC plugin called before running 
into the timeout.

I can bundle up a sample of the connection debugs and send them over if 
they will help.

All the best,
Colin Waring.

On 09/07/2014 11:00, Thomas Eckardt wrote:
 Is there a setting I can set in ASSP to have it accept one message and
 disconnect

 No - there is no such setting.

 Colin - please check if the issue is anyhow related to 'noprocessing'
 and/or 'whitelisted'.

 Thomas



 Von:Colin colin.war...@gmail.com
 An: assp-test@lists.sourceforge.net
 Datum:  09.07.2014 11:46
 Betreff:Re: [Assp-test] Timeout issues



 Hi Thomas,

 I think I have spotted something. I'm seeing loads being written to the
 debug files now. In all cases that I have checked the affected messages
 are from a connection where multiple emails have been delivered.

 The first email comes through absolutely fine, then the connection is
 kept open for the second email. The second email is received but times
 out. This may explain why I could find the opening of the connection for
 affected emails and why not all messages from particular senders are
 affected.

 Is there a setting I can set in ASSP to have it accept one message and
 disconnect, forcing the remote MTA to open a new connection for each
 message? It isn't ideal but would allow me to establish that the issue
 is definitely limited to these kinds of connections.

 All the best,
 Colin Waring.


 On 09/07/2014 10:21, Thomas Eckardt wrote:
 enable 'ConTimeOutDebug'

 assp will write the connection states in to files in the debug folder.

 Thomas





 Von:Colin colin.war...@gmail.com
 An: Assp-test@lists.sourceforge.net
 Datum:  08.07.2014 17:04
 Betreff:Re: [Assp-test] Timeout issues



 Apologies for replying to self, turns out the log lines are out of sync
 by a second so my greps weren't showing it up.

 The MTA is indeeed showing SMTP data timeout (message abandoned) on
 connection from so I'll change my report. The connection is being
 opened to the MTA but it looks like ASSP is not passing the message to
 the MTA after scanning it and getting message ok. The result is then
 either the MTA or ASSP timing out, I think the timeouts are set to the
 same so either one could kick in first - that explains difference
 between the last two logs.

 On 08/07/2014 15:50, Colin wrote:
 Hi Folks,

 I've given up on the list accepting messages from my main account. SF
 didn't respond to my request for help either.

 I have had an ongoing issue for a good few weeks now where its seems
 like ASSP accepts a message but does not pass it on to the MTA. It
 talks to the remote host and goes through the motions. There is no
 evidence at all in the MTA logs that ASSP has even opened a connection
 to it for the message concerned. When I originally spotted the
 problem, ASSP didn't output anything for the message past message ok.

 One of the recent revisions changed that and now ASSP reports a
 timeout, but I have seen it reported in two different ways - logs
 below.
 Example 1 - no timeout reported by ASSP

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld info: found
 message size announcement: 13.54 kByte

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld [SMTP Reply] 250
 OK
 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 250
 Accepted

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 354
 Enter message, ending with . on a line by itself

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld info: found known
 good HELO 'mout1.freenet.de' - weight is -2

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -40 for KnownGoodHelo, total score for this message is now -40

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -15 (pbwValencePB) for In Penalty White Box, total score for
 this message is now -55

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -5 for 195.4.92 in griplist (0.00), total score for this message
 is now -60

 2014-06-25 11:14

Re: [Assp-test] Timeout issues

[Colin]

Hi Thomas,

I think my logs are getting confused - having the MTA only accept 1 
message per connection is causing ASSP to write debug logs for the ones 
it doesn't accept. Oddly, ASSP still accepts the message from the remote 
sender when the MTA has rejected the message at the MAIL FROM: stage as 
the debug file shows both complete messages.

I will remove the MTA restriction, empty the debug folder and restart 
ASSP to make sure only fresh connections get written to the debug folder 
before sending anything over.

Thanks for the help,
Colin.

On 09/07/2014 11:14, Thomas Eckardt wrote:
 2014-07-09 10:47:23 server unpoll from main sub main::ThreadMain 165
 This signals that assp has written all received data to the MTA's
 connection. Please could you anyhow check, if the trailing CRLF.CRLF was
 sent to the MTA in DATA part of the mail - or better first, if the client
 has sent the CRLF.CRLF (should be in the logged .eml file).

 Thomas





 Von:Colin colin.war...@gmail.com
 An: assp-test@lists.sourceforge.net
 Datum:  09.07.2014 12:02
 Betreff:Re: [Assp-test] Timeout issues



 Hi Thomas,

 Further to this I set a 1 message limit on the MTA. The number of
 affected connections seems to have reduced but I am still seeing
 timeouts being logged on individual messages. The only extra thing the
 full connection debug shows after the end of receiving the data is
 server unpoll.

 2014-07-09 10:47:23 server unpoll from main sub main::ThreadMain 165
 2014-07-09 10:53:24 client Timeout after 360 secs^M
 2014-07-09 10:53:24 client was readable^M
 2014-07-09 10:53:24 client was not writable^M

 All the best,
 Colin Waring.

 On 09/07/2014 10:21, Thomas Eckardt wrote:
 enable 'ConTimeOutDebug'

 assp will write the connection states in to files in the debug folder.

 Thomas





 Von:Colin colin.war...@gmail.com
 An: Assp-test@lists.sourceforge.net
 Datum:  08.07.2014 17:04
 Betreff:Re: [Assp-test] Timeout issues



 Apologies for replying to self, turns out the log lines are out of sync
 by a second so my greps weren't showing it up.

 The MTA is indeeed showing SMTP data timeout (message abandoned) on
 connection from so I'll change my report. The connection is being
 opened to the MTA but it looks like ASSP is not passing the message to
 the MTA after scanning it and getting message ok. The result is then
 either the MTA or ASSP timing out, I think the timeouts are set to the
 same so either one could kick in first - that explains difference
 between the last two logs.

 On 08/07/2014 15:50, Colin wrote:
 Hi Folks,

 I've given up on the list accepting messages from my main account. SF
 didn't respond to my request for help either.

 I have had an ongoing issue for a good few weeks now where its seems
 like ASSP accepts a message but does not pass it on to the MTA. It
 talks to the remote host and goes through the motions. There is no
 evidence at all in the MTA logs that ASSP has even opened a connection
 to it for the message concerned. When I originally spotted the
 problem, ASSP didn't output anything for the message past message ok.

 One of the recent revisions changed that and now ASSP reports a
 timeout, but I have seen it reported in two different ways - logs
 below.
 Example 1 - no timeout reported by ASSP

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld info: found
 message size announcement: 13.54 kByte

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld [SMTP Reply] 250
 OK
 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 250
 Accepted

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 354
 Enter message, ending with . on a line by itself

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld info: found known
 good HELO 'mout1.freenet.de' - weight is -2

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -40 for KnownGoodHelo, total score for this message is now -40

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score:
 added -15 (pbwValencePB) for In Penalty White Box, total score for
 this message is now -55

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out]
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to:
 recipi...@domain.tld mailto:recipi

Re: [Assp-test] Timeout issues

[Colin]

Apologies for replying to self, turns out the log lines are out of sync 
by a second so my greps weren't showing it up.

The MTA is indeeed showing SMTP data timeout (message abandoned) on 
connection from so I'll change my report. The connection is being 
opened to the MTA but it looks like ASSP is not passing the message to 
the MTA after scanning it and getting message ok. The result is then 
either the MTA or ASSP timing out, I think the timeouts are set to the 
same so either one could kick in first - that explains difference 
between the last two logs.

On 08/07/2014 15:50, Colin wrote:
 Hi Folks,

 I've given up on the list accepting messages from my main account. SF 
 didn't respond to my request for help either.

 I have had an ongoing issue for a good few weeks now where its seems 
 like ASSP accepts a message but does not pass it on to the MTA. It 
 talks to the remote host and goes through the motions. There is no 
 evidence at all in the MTA logs that ASSP has even opened a connection 
 to it for the message concerned. When I originally spotted the 
 problem, ASSP didn't output anything for the message past message ok.

 One of the recent revisions changed that and now ASSP reports a 
 timeout, but I have seen it reported in two different ways - logs below.

 Example 1 - no timeout reported by ASSP

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld info: found 
 message size announcement: 13.54 kByte

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld [SMTP Reply] 250 OK

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 250 
 Accepted

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 354 
 Enter message, ending with . on a line by itself

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld info: found known 
 good HELO 'mout1.freenet.de' - weight is -2

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score: 
 added -40 for KnownGoodHelo, total score for this message is now -40

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score: 
 added -15 (pbwValencePB) for In Penalty White Box, total score for 
 this message is now -55

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld Message-Score: 
 added -5 for 195.4.92 in griplist (0.00), total score for this message 
 is now -60

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld [scoring] SPF: none 
 ip=1.1.1.1 mailfrom=sen...@domain.tld 
 mailto:mailfrom=sen...@domain.tld helo=mout1.freenet.de

 2014-06-25 11:14:28 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld Whitelisted sender 
 address: sen...@domain.tld mailto:sen...@domain.tld for recipient 
 recipi...@domain.tld mailto:recipi...@domain.tld

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld [Plugin] calling 
 plugin ASSP_AFC

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld whitelisted (no bad 
 attachments)

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld [Plugin] calling 
 plugin ASSP_DCC

 2014-06-25 11:14:29 m1-91268-01851 [Worker_5] [TLS-in] [TLS-out] 
 [MessageOK] 1.1.1.1 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld message ok [Fwd Re 
 Test]


 Example 2 - ASSP reported its own timeout

 2014-06-30 17:42:49 m1-46569-09929 [Worker_4] [TLS-out] 1.1.1.1 
 sen...@domain.tld mailto:sen...@domain.tld [SMTP Reply] 250 OK

 2014-06-30 17:42:50 m1-46569-09929 [Worker_4] [TLS-out] 1.1.1.1 
 sen...@domain.tld mailto:sen...@domain.tld to: 
 recipi...@domain.tld mailto:recipi...@domain.tld [SMTP Reply] 250 
 Accepted

 2014-06-30 17:42:50 m1

Re: [Assp-test] Unsupported bDat

[Colin Waring]

Hi Thomas,

If you have a test version please feel free to send it over. I'm starting to 
get a lot of complaints on this one - I thought it was just one sender at first 
but it looks like we're going to get a lot of grief over this one!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 30 June 2014 16:03
To: ASSP development mailing list
Subject: Re: [Assp-test] Unsupported bDat

Had the same trouble - but I think I found the BUG - just testing. The Problem 
is only related to whitelisted and or noprocessing mails.

Thomas




Von:Daniel Miller dmil...@amfes.com
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  30.06.2014 16:23
Betreff:[Assp-test] Unsupported bDat



Having trouble sending an attachment - never seen this error before:

Jun-30-1407:14:3837678-11881[Worker_1][TLS-in][TLS-out]*98.167.72.49**dmil...@amfes.com*info:foundmessagesizeannouncement:3.26MByte
Jun-30-1407:14:3837678-11881[Worker_1][TLS-in][TLS-out]*98.167.72.49**dmil...@amfes.com*messageproxiedwithoutprocessing-messagesize(3416095)isabove50(npSizeOut
 

http://bubba.amfes.lan:5/#npSizeOut).
Jun-30-1407:14:4237681-13661[Worker_1][TLS-in][TLS-out][unsupported_bDAt]*98.167.72.49*bDAtnotallowed
 



--
Daniel

--
Open source business process management suite built on Java and Eclipse Quickly 
connect people, data, and systems into organized workflows Winner of BOSSIE, 
CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft 
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known 
virus in this email!
***




--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Unsupported bDat

[Colin Waring]

Sorry I though you said you had fixed it and were just testing. Can we 
downgrade to an earlier version to get away from this bug? I have one client 
that is affected massively by this for some reason.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 30 June 2014 17:07
To: ASSP development mailing list
Subject: Re: [Assp-test] Unsupported bDat

If you have a test version please feel free to send it over.

This makes currently no sense - there is a BUG that I need to find and to fix. 
If I think I have fixed it, I'll release the code.

Thomas





Von:Colin Waring co...@dolphinict.co.uk
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  30.06.2014 17:46
Betreff:Re: [Assp-test] Unsupported bDat



Hi Thomas,

If you have a test version please feel free to send it over. I'm starting to 
get a lot of complaints on this one - I thought it was just one sender at first 
but it looks like we're going to get a lot of grief over this one!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 30 June 2014 16:03
To: ASSP development mailing list
Subject: Re: [Assp-test] Unsupported bDat

Had the same trouble - but I think I found the BUG - just testing. The Problem 
is only related to whitelisted and or noprocessing mails.

Thomas




Von:Daniel Miller dmil...@amfes.com
An: ASSP development mailing list assp-test@lists.sourceforge.net
Datum:  30.06.2014 16:23
Betreff:[Assp-test] Unsupported bDat



Having trouble sending an attachment - never seen this error before:

Jun-30-1407:14:3837678-11881[Worker_1][TLS-in][TLS-out]*98.167.72.49**dmil...@amfes.com*info:foundmessagesizeannouncement:3.26MByte
Jun-30-1407:14:3837678-11881[Worker_1][TLS-in][TLS-out]*98.167.72.49**dmil...@amfes.com*messageproxiedwithoutprocessing-messagesize(3416095)isabove50(npSizeOut
 


http://bubba.amfes.lan:5/#npSizeOut).
Jun-30-1407:14:4237681-13661[Worker_1][TLS-in][TLS-out][unsupported_bDAt]*98.167.72.49*bDAtnotallowed
 




--
Daniel

--
Open source business process management suite built on Java and Eclipse Quickly 
connect people, data, and systems into organized workflows Winner of BOSSIE, 
CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft 
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 


individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
Open source business process management suite built on Java and Eclipse
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Open source business process management suite built on Java and Eclipse
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Open source business process management suite

Re: [Assp-test] fixes in assp 2.4.2 build 14181

[Colin Waring]

Thank you so much for getting the fix out quickly on this one. 

I'm going to forward an email I sent earlier this month - I'm hoping that your 
fix may have resolved that issue too but I wanted to make sure now that I 
appear to be able to send to the SF lists again!

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 30 June 2014 19:33
To: ASSP List
Subject: [Assp-test] fixes in assp 2.4.2 build 14181

Hi all,

fixed in assp 2.4.2 build 14181:

- mails archived by ASSP_ARC.pm had an additionaly trailing '.'

- for some mails an exception 'Odd number of elements in hash assignment at
  ...Perl/site/lib/Mail/SPF/Server.pm line 210.' was thrown

- with an installed version 1.994 of IO::Socket::SSL , the SMTP-SSL
listener(s) was only working in plain text

- message scoring was not working for local and outgoing mails, read the 
'changed' section

changed:

- message scoring was switched off in the code for local and outgoing mails, it 
is now enabled and configurable -
  read the 'added' section

- on very slow IP connections to the Web-Interface, it was possible that the 
transfered data were incomplete
  because of a hardcoded content-transfer-timeout of 30 seconds
  This timeout value is now controlled with the hidden configuration variable 
'WebTrafficTimeout', which has
  a default value of 60 seconds
 
added:

'DoLocalPenaltyMessage','Message Scoring Mode for Local and Outgoing Mails', 
'If this feature is selected, the total score for all checks during a local or 
outgoing message is used to  determine if the email is Spam. If the combined 
score is greater than the Local Low MessageLimit
 (LocalPenaltyMessageLow) and less than or equal the Local High MessageLimit 
(LocalPenaltyMessageLimit)  the message will not be blocked but tagged. If the 
combined score is greater than the Local High MessageLimit  
(LocalPenaltyMessageLimit), the message will be blocked.


'LocalPenaltyMessageLow','Low MessageLimit for Local and Outgoing Mails'
'MessageMode will not block local and outgoing messages whose score exceeds 
this threshold during the message but  will tag them.  For example: 40'


'LocalPenaltyMessageLimit','High MessageLimit for Local and Outgoing Mails'
'MessageMode will block local and outgoing messages whose score exceeds this 
threshold during the message.
 For example: 50'

Thomas


DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known 
virus in this email!
***




--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

--
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Crashes today

[Colin Waring]

Anyone else seeing ASSP crashing a lot today?

 

Each time is preceeded by emails from quotes@somethingorother like this:

 

2014-06-02 12:10:00 m1-07354-06606 [Worker_5] [SSL-out] 109.228.10.136
quo...@professional-crm.co.uk to: recipi...@domain.tld info: start damping
on closing connection (12)

2014-06-02 12:11:30 m1-07490-12287 [Worker_2] [TLS-out] 109.228.22.120
quo...@letscompare-seo.co.uk [SMTP Reply] 250 OK

2014-06-02 12:11:33 m1-07490-12287 [Worker_2] [TLS-out] 109.228.22.120
quo...@letscompare-seo.co.uk to: recipi...@domain.tld [SMTP Reply] 250
Accepted

2014-06-02 12:11:33 m1-07490-12287 [Worker_2] [TLS-out] 109.228.22.120
quo...@letscompare-seo.co.uk to: recipi...@domain.tld recipient delayed:
recipi...@domain.tld

2014-06-02 12:11:33 m1-07490-12287 [Worker_2] [TLS-out] 109.228.22.120
quo...@letscompare-seo.co.uk to: recipi...@domain.tld [SMTP Status] 451
4.7.1 Greylisting, Please try again after 1 minute

2014-06-02 12:11:56 m1-07516-07642 [Worker_2] [TLS-out] 109.228.30.116
quo...@compare-frankingmachines.co.uk [SMTP Reply] 250 OK

2014-06-02 12:12:05 m1-07516-07642 [Worker_2] [TLS-out] 109.228.30.116
quo...@compare-frankingmachines.co.uk to: recipi...@domain.tld [SMTP
Reply] 250 Accepted

2014-06-02 12:12:05 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk [SMTP Reply] 250 OK

2014-06-02 12:12:08 m1-07516-07642 [Worker_2] [TLS-out] 109.228.30.116
quo...@compare-frankingmachines.co.uk to: recipi...@domain.tld recipient
delayed: recipi...@domain.tld

2014-06-02 12:12:08 m1-07516-07642 [Worker_2] [TLS-out] 109.228.30.116
quo...@compare-frankingmachines.co.uk to: recipi...@domain.tld [SMTP
Status] 451 4.7.1 Greylisting, Please try again after 1 minute

2014-06-02 12:12:15 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk to: recipi...@domain.tld [SMTP Reply]
250 Accepted

2014-06-02 12:12:21 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk to: recipi...@domain.tld [SMTP Reply]
354 Enter message, ending with . on a line by itself

2014-06-02 12:12:25 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk to: recipi...@domain.tld
DomainKey-Signature found

2014-06-02 12:12:25 m1-07544-12617 [Worker_1] [TLS-out] 109.228.4.25
quo...@compare-webdesign.co.uk [SMTP Reply] 250 OK

2014-06-02 12:12:26 m1-07524-03885 [Worker_2] [TLS-out] 109.228.2.102
quo...@communication-systems.co.uk to: recipi...@domain.tld Message-Score:
added 25 for DNSBL: neutral, 109.228.2.102 listed in
bb.barracudacentral.org, total score for this message is now 25

 

I'm snowed under as we're moving out entire infrastructure to a new platform
at the moment, has anyone else crafted any rules to stop these? Source
addresses are similar but I'm not sure about blocking an entire /16 

--
Learn Graph Databases - Download FREE O'Reilly Book
Graph Databases is the definitive new guide to graph databases and their 
applications. Written by three acclaimed leaders in the field, 
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Email::MIME problem

[Colin Waring]

Strangely, downgrading to 1.911 and upgrading to the latest ASSP did not
work for me. I have already had some overnight reports of corrupted mail.

I am just upgrading to 14144 now so hopefully that will resolve the problem.
With 14144 should we use the latest Email::MIME?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 23 May 2014 16:39
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

clean the PTRCache after upgrade

Thomas




Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 17:27
Betreff:Re: [Assp-test] Email::MIME problem



I've gone through my servers and replaces Email::MIME 1.926 with 1.911.

I'll get the latest version of ASSP running shortly. PTR cache is turned off
already, presumably I noticed problems with the cache at some point.

I have ASSP_AFC enabled but charset conversation not enabled.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 10:23
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

Colin,

so you may wish to check the code introduced after 14097

I'm doing this for several days now.

With Email::MIME 1.926 I (and others) get wired behavior - even if I use the
same mail multiple times.

There are two places where assp uses Email::MIME to modify an  email, the
charset conversion and the ASSP_AFC plugin (in case of spam/bad attachment
only).
Both could be disabled having the same result.

I have an idea what could happen - but I hope I'm wrong. I'll have to look
in to Perl and Carp source code - eval and exception handling.

workaround - use Email::MIME 1.911

Colin , btw. 14097 has a big issue with the PTR resolving/caching - switch
this check off!

Thomas

 





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 09:56
Betreff:Re: [Assp-test] Email::MIME problem



Hi Thomas,

This sounds like exactly the issue I reported on the 6th.

I found that the issue was not present in version 14097 and earlier so you
may wish to check the code introduced after 14097.

I've been busy with other things so haven't been able to do any more
troubleshooting on it and am still running 14097 myself without the issue.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 06:52
To: development mailing list ASSP
Subject: [Assp-test] Email::MIME problem

Hi all,

I got a report from Philipp about the problem were mails are delivered
incomplete/destroyed.

Philipp wrote:

...
I recently had the problem that some mails were forwarded incorrectly by
ASSP
2.4.2 build 14141 (and 14130, too).

That is, the mail was received and saved to file correctly, but when it was
forwarded to the destination MTA, it started in the midst of the content,
removed ALL header lines and added the ASSP-Headers at the end of the mail.
It was the same for noProcessing-mails, too, thus I excluded problems by
spam processing.

After long and painful debugging, I concluded that the problem must be the 



Header-Parsing of some multipart mails, but not all of them and I still
don't know which of those exactly, sorry!

Since I have another server with more or less the same configuration (at
least no differences that would influence noProcessing-mails) and there is
no sign of this problem, I concentrated on differences between those two
servers.

The only real differences were on some perl module version numbers. While
the faulty server had Email::MIME in version 1.926, the other one in version
1.911. Thus I downgraded this module on the faulty server and it seems as it
solved the problem.


I've released Email::MIME 1.911 as ZIP in the /lib folder on SF and SF-CVS.
To install it, copy the extracted ZIP in to the assp/lib folder and restart
assp. 
I currently don't know why and where the problem in assp is. It will take a
while to analyze the problem.

Thomas



DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 



individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***





--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform 
available
Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu

Re: [Assp-test] Email::MIME problem

[Colin Waring]

Thanks Thomas, muchly appreciated.

I have put a few tests through with 14144 and the latest Email::MIME and
they seem fine.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 24 May 2014 12:13
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

I am just upgrading to 14144 now so hopefully that will resolve the 
problem.

It will !

With 14144 should we use the latest Email::MIME?

Yes, the latest.

IMHO Email::MIME was not directly involved - only the bad 'bondary' was 
possibly - but is no longer a problem with any version of this module

The reason was a  'not common' but nice coding (by me) some years ago, 
used to call Perl's sv_grow for large mails (eg. noprocessing by size). 
This was no longer working with the new permanent opened UDP-DNS sockets - 
very strange and very hard to find - even the Perl souce code does not 
clearly explain what happens.
The assp install script also installs the Convert::Scalar module (since 
years), which consumes more memory, but is commonly used to do the 
sv_grow. If installed, it is used now by assp for this function. 

Thomas



Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  24.05.2014 12:15
Betreff:Re: [Assp-test] Email::MIME problem



Strangely, downgrading to 1.911 and upgrading to the latest ASSP did not
work for me. I have already had some overnight reports of corrupted mail.

I am just upgrading to 14144 now so hopefully that will resolve the 
problem.
With 14144 should we use the latest Email::MIME?

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 23 May 2014 16:39
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

clean the PTRCache after upgrade

Thomas




Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 17:27
Betreff:Re: [Assp-test] Email::MIME problem



I've gone through my servers and replaces Email::MIME 1.926 with 1.911.

I'll get the latest version of ASSP running shortly. PTR cache is turned 
off
already, presumably I noticed problems with the cache at some point.

I have ASSP_AFC enabled but charset conversation not enabled.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 10:23
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

Colin,

so you may wish to check the code introduced after 14097

I'm doing this for several days now.

With Email::MIME 1.926 I (and others) get wired behavior - even if I use 
the
same mail multiple times.

There are two places where assp uses Email::MIME to modify an  email, the
charset conversion and the ASSP_AFC plugin (in case of spam/bad attachment
only).
Both could be disabled having the same result.

I have an idea what could happen - but I hope I'm wrong. I'll have to look
in to Perl and Carp source code - eval and exception handling.

workaround - use Email::MIME 1.911

Colin , btw. 14097 has a big issue with the PTR resolving/caching - switch
this check off!

Thomas

 





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 09:56
Betreff:Re: [Assp-test] Email::MIME problem



Hi Thomas,

This sounds like exactly the issue I reported on the 6th.

I found that the issue was not present in version 14097 and earlier so you
may wish to check the code introduced after 14097.

I've been busy with other things so haven't been able to do any more
troubleshooting on it and am still running 14097 myself without the issue.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 06:52
To: development mailing list ASSP
Subject: [Assp-test] Email::MIME problem

Hi all,

I got a report from Philipp about the problem were mails are delivered
incomplete/destroyed.

Philipp wrote:

...
I recently had the problem that some mails were forwarded incorrectly by
ASSP
2.4.2 build 14141 (and 14130, too).

That is, the mail was received and saved to file correctly, but when it 
was
forwarded to the destination MTA, it started in the midst of the content,
removed ALL header lines and added the ASSP-Headers at the end of the 
mail.
It was the same for noProcessing-mails, too, thus I excluded problems by
spam processing.

After long and painful debugging, I concluded that the problem must be the 




Header-Parsing of some multipart mails, but not all of them and I still
don't know which of those exactly, sorry!

Since I have another server with more or less the same configuration (at
least no differences that would influence noProcessing-mails) and there is
no sign

Re: [Assp-test] Email::MIME problem

[Colin Waring]

I've gone through my servers and replaces Email::MIME 1.926 with 1.911.

I'll get the latest version of ASSP running shortly. PTR cache is turned off
already, presumably I noticed problems with the cache at some point.

I have ASSP_AFC enabled but charset conversation not enabled.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com] 
Sent: 23 May 2014 10:23
To: ASSP development mailing list
Subject: Re: [Assp-test] Email::MIME problem

Colin,

so you may wish to check the code introduced after 14097

I'm doing this for several days now.

With Email::MIME 1.926 I (and others) get wired behavior - even if I use the
same mail multiple times.

There are two places where assp uses Email::MIME to modify an  email, the
charset conversion and the ASSP_AFC plugin (in case of spam/bad attachment
only).
Both could be disabled having the same result.

I have an idea what could happen - but I hope I'm wrong. I'll have to look
in to Perl and Carp source code - eval and exception handling.

workaround - use Email::MIME 1.911

Colin , btw. 14097 has a big issue with the PTR resolving/caching - switch
this check off!

Thomas

 





Von:Colin Waring co...@lanternhosting.co.uk
An: 'ASSP development mailing list' 
assp-test@lists.sourceforge.net
Datum:  23.05.2014 09:56
Betreff:Re: [Assp-test] Email::MIME problem



Hi Thomas,

This sounds like exactly the issue I reported on the 6th.

I found that the issue was not present in version 14097 and earlier so you
may wish to check the code introduced after 14097.

I've been busy with other things so haven't been able to do any more
troubleshooting on it and am still running 14097 myself without the issue.

All the best,
Colin Waring.

-Original Message-
From: Thomas Eckardt [mailto:thomas.ecka...@thockar.com]
Sent: 23 May 2014 06:52
To: development mailing list ASSP
Subject: [Assp-test] Email::MIME problem

Hi all,

I got a report from Philipp about the problem were mails are delivered
incomplete/destroyed.

Philipp wrote:

...
I recently had the problem that some mails were forwarded incorrectly by
ASSP
2.4.2 build 14141 (and 14130, too).

That is, the mail was received and saved to file correctly, but when it was
forwarded to the destination MTA, it started in the midst of the content,
removed ALL header lines and added the ASSP-Headers at the end of the mail.
It was the same for noProcessing-mails, too, thus I excluded problems by
spam processing.

After long and painful debugging, I concluded that the problem must be the 


Header-Parsing of some multipart mails, but not all of them and I still
don't know which of those exactly, sorry!

Since I have another server with more or less the same configuration (at
least no differences that would influence noProcessing-mails) and there is
no sign of this problem, I concentrated on differences between those two
servers.

The only real differences were on some perl module version numbers. While
the faulty server had Email::MIME in version 1.926, the other one in version
1.911. Thus I downgraded this module on the faulty server and it seems as it
solved the problem.


I've released Email::MIME 1.911 as ZIP in the /lib folder on SF and SF-CVS.
To install it, copy the extracted ZIP in to the assp/lib folder and restart
assp. 
I currently don't know why and where the problem in assp is. It will take a
while to analyze the problem.

Thomas



DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the 


individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no known
virus in this email!
***





--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform 
available
Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu/SauceLabs
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***




--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run

Re: [Assp-test] rebuildspamdb error ASSP version 2.4.2(14132)

[Colin Waring]

You need to make sure that all the files in your lib and plugins folders are up 
to date. Wordstem will be one of them and no doubt others will be out of date.

All the best,
Colin Waring

On 17 May 2014 18:02, Daniel K. Du Vall dduv...@1peter4-10.org wrote:

 I have run rebuildspamdb as suggested but still getting this error. Have I 
 missed updating something else somewhere or maybe something else? 

 May-17-14 11:54:50 [init] Spamdb has 103,090 records 
 May-17-14 11:54:50 [init] Warning: the current Spamdb is possibly 
 incompatible to this version of ASSP. Please run a rebuildspamdb. current: 
 2_14094_5.014002_UAX#29_WordStem1.23 - required: 
 2_14094_5.014002_UAX#29_WordStem1.27 
 May-17-14 11:54:50 [init] Start analyze whitelist 
 May-17-14 11:54:50 [init] Whitelist has 4,049 records 
 May-17-14 11:54:50 [init] The Hidden-Markov-Model-DB has 887,742 records. 
 May-17-14 11:54:50 [init] Warning: the current HMMdb is possibly incompatible 
 to this version of ASSP. Please run a rebuildspamdb. current: 
 2_14094_5.014002_UAX#29_WordStem1.23 - required: 
 2_14094_5.014002_UAX#29_WordStem1.27 
 May-17-14 11:54:50 [init] Info: saving Stats in file asspstats.sav 
 May-17-14 11:54:50 [init] Info: saving ScoreStats in file asspscorestats.sav 


 Daniel Du Vall 

 --
  
 Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE 
 Instantly run your Selenium tests across 300+ browser/OS combos. 
 Get unparalleled scalability from the best Selenium testing platform 
 available 
 Simple to use. Nothing to install. Get started now for free. 
 http://p.sf.net/sfu/SauceLabs 
 ___ 
 Assp-test mailing list 
 Assp-test@lists.sourceforge.net 
 https://lists.sourceforge.net/lists/listinfo/assp-test 
--
Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free.
http://p.sf.net/sfu/SauceLabs
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test