[Assp-test] Antwort: Re: Speeding up MySQL

[Thomas Eckardt]

forgot my speed results

Sep-13-12 04:15:43 Generating weighted Bayesian tuplets

Sep-13-12 04:15:59 start populating Spamdb with 172,557 records - Bayesian 
check is now disabled!
Sep-13-12 04:16:43 Finished populating Spamdb with 172,557 records - 
Bayesian check is now enabled!

44 seconds for 172.557 spamdb records (include a fixed delay of 5 seconds 
- so it takes really 39 seconds)

Sep-13-12 04:18:44 start populating Hidden Markov Model with 774,061 
records!
Sep-13-12 04:20:43 Finished populating Hidden Markov Model with 774,061 
records!

2 miniutes (119 seconds) for 774.061 hmmdb records

The performance would be better if this system were not a allrounder. In 
german we say "an oviparous wolled sow that gives milk". 

Thomas
 




Von:Thomas Eckardt 
An: ASSP development mailing list , 
Datum:  13.09.2012 07:52
Betreff:[Assp-test] Antwort: Re:  Speeding up MySQL



>SELECT COUNT(*) FROM hmmdb takes 22 seconds.

Looks like you don't use INNODB for the tables or your INNODB cache is too 

small.


hmm use 100MB and apmdb 15MB on my system

the relevant settings for mysql and INNODB are

max_allowed_packet=128M
net_buffer_length=32M
sql-mode=STRICT_ALL_TABLES,STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
max_connections=800
query_cache_size=84M
table_cache=1520
tmp_table_size=30M
innodb_additional_mem_pool_size=6M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=30M
innodb_buffer_pool_size=250M
innodb_log_file_size=512M
innodb_thread_concurrency=8
innodb_file_io_threads=4

Thomas


Von:Colin 
An: assp-test@lists.sourceforge.net, 
Datum:  13.09.2012 01:41
Betreff:Re: [Assp-test] Speeding up MySQL



Thanks for the tips, I have turned on the slow query logging and it only 
shows two queries. DELETE FROM hmmdb takes just under two minutes and 
SELECT COUNT(*) FROM hmmdb takes 22 seconds.

Nothing else is logged.

I'll investigate mysqltuner further tomorrow - it thinks that mysql is 
requiring 13GB at its biggest which is rather big!

I'm not sure fragmentation will make much of a difference as the HMM db 
is emptied before it is repopulated

Thanks for the suggestions, I've never really run any optimisation on 
mySQL so any tips are appreciated.

All the best,
Colin Waring.

On 12/09/2012 22:15, Rusty Nejdl wrote:
> Here's some generic mysql tuning info:
>
> Turn on logging of slow queries and figure out which queries are 
> taking a while.  Run them through an analyzer to see if any indexes 
> are missing.  mysqltuner and mysqlfragfinder are two scripts you can 
> run to help tune the database.
>
> (If I am telling you stuff you already know, don't take it personally.)
>
> Rusty Nejdl
>
> On 2012-09-12 15:37, Colin wrote:
>> I think this came up a long time ago, but the "populating Hidden Markov
>> Model" part of rebuildspamdb takes a very long time on my system (over
>> an hour).
>>
>> This time I have been keeping an eye on it and I have no idea why.
>>
>> For the majority of that hour, there is no CPU activity, no network
>> activity and a very small amount of disk write activity. For the last
>> minute, there was a burst of network and CPU activity.
>>
>> Has anyone experience with how to tune up mysql so it works a bit
>> quicker during the rebuild but doesn't negatively affect the regular
>> operations during the day? This server does nothing except the ASSP
>> database..
>>
>> All the best,
>> Colin Waring.
>>
>>
>> 
-- 


>>
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. 
>> Discussions
>> will include endpoint security, mobile security and the latest in 
>> malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
> 
--
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. 
Discussions
> will include endpoint security, mobile security and the latest in 
malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04

[Assp-test] fixes in assp 2.2.2 build 12257

[Thomas Eckardt]

Hi all,

fixed in assp 2.2.2 build 12257:

- The rebuild task has possibly tried to correct the corpus by deleting 
files, even this was not required or
  useless


Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Antwort: Re: Speeding up MySQL

[Thomas Eckardt]

>SELECT COUNT(*) FROM hmmdb takes 22 seconds.

Looks like you don't use INNODB for the tables or your INNODB cache is too 
small.


hmm use 100MB and apmdb 15MB on my system

the relevant settings for mysql and INNODB are

max_allowed_packet=128M
net_buffer_length=32M
sql-mode=STRICT_ALL_TABLES,STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
max_connections=800
query_cache_size=84M
table_cache=1520
tmp_table_size=30M
innodb_additional_mem_pool_size=6M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=30M
innodb_buffer_pool_size=250M
innodb_log_file_size=512M
innodb_thread_concurrency=8
innodb_file_io_threads=4

Thomas


Von:Colin 
An: assp-test@lists.sourceforge.net, 
Datum:  13.09.2012 01:41
Betreff:Re: [Assp-test] Speeding up MySQL



Thanks for the tips, I have turned on the slow query logging and it only 
shows two queries. DELETE FROM hmmdb takes just under two minutes and 
SELECT COUNT(*) FROM hmmdb takes 22 seconds.

Nothing else is logged.

I'll investigate mysqltuner further tomorrow - it thinks that mysql is 
requiring 13GB at its biggest which is rather big!

I'm not sure fragmentation will make much of a difference as the HMM db 
is emptied before it is repopulated

Thanks for the suggestions, I've never really run any optimisation on 
mySQL so any tips are appreciated.

All the best,
Colin Waring.

On 12/09/2012 22:15, Rusty Nejdl wrote:
> Here's some generic mysql tuning info:
>
> Turn on logging of slow queries and figure out which queries are 
> taking a while.  Run them through an analyzer to see if any indexes 
> are missing.  mysqltuner and mysqlfragfinder are two scripts you can 
> run to help tune the database.
>
> (If I am telling you stuff you already know, don't take it personally.)
>
> Rusty Nejdl
>
> On 2012-09-12 15:37, Colin wrote:
>> I think this came up a long time ago, but the "populating Hidden Markov
>> Model" part of rebuildspamdb takes a very long time on my system (over
>> an hour).
>>
>> This time I have been keeping an eye on it and I have no idea why.
>>
>> For the majority of that hour, there is no CPU activity, no network
>> activity and a very small amount of disk write activity. For the last
>> minute, there was a burst of network and CPU activity.
>>
>> Has anyone experience with how to tune up mysql so it works a bit
>> quicker during the rebuild but doesn't negatively affect the regular
>> operations during the day? This server does nothing except the ASSP
>> database..
>>
>> All the best,
>> Colin Waring.
>>
>>
>> 
-- 

>>
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. 
>> Discussions
>> will include endpoint security, mobile security and the latest in 
>> malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
> 
--
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. 
Discussions
> will include endpoint security, mobile security and the latest in 
malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the la

Re: [Assp-test] Speeding up MySQL

[Colin]

Thanks for the tips, I have turned on the slow query logging and it only 
shows two queries. DELETE FROM hmmdb takes just under two minutes and 
SELECT COUNT(*) FROM hmmdb takes 22 seconds.

Nothing else is logged.

I'll investigate mysqltuner further tomorrow - it thinks that mysql is 
requiring 13GB at its biggest which is rather big!

I'm not sure fragmentation will make much of a difference as the HMM db 
is emptied before it is repopulated

Thanks for the suggestions, I've never really run any optimisation on 
mySQL so any tips are appreciated.

All the best,
Colin Waring.

On 12/09/2012 22:15, Rusty Nejdl wrote:
> Here's some generic mysql tuning info:
>
> Turn on logging of slow queries and figure out which queries are 
> taking a while.  Run them through an analyzer to see if any indexes 
> are missing.  mysqltuner and mysqlfragfinder are two scripts you can 
> run to help tune the database.
>
> (If I am telling you stuff you already know, don't take it personally.)
>
> Rusty Nejdl
>
> On 2012-09-12 15:37, Colin wrote:
>> I think this came up a long time ago, but the "populating Hidden Markov
>> Model" part of rebuildspamdb takes a very long time on my system (over
>> an hour).
>>
>> This time I have been keeping an eye on it and I have no idea why.
>>
>> For the majority of that hour, there is no CPU activity, no network
>> activity and a very small amount of disk write activity. For the last
>> minute, there was a burst of network and CPU activity.
>>
>> Has anyone experience with how to tune up mysql so it works a bit
>> quicker during the rebuild but doesn't negatively affect the regular
>> operations during the day? This server does nothing except the ASSP
>> database..
>>
>> All the best,
>> Colin Waring.
>>
>>
>> --
>>  
>>
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. 
>> Discussions
>> will include endpoint security, mobile security and the latest in 
>> malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
> --
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm

[Colin]

2012-09-12 22:57:41 generating Spamdb.helo records from 7,207 collected HELO's
2012-09-12 22:57:59 cleaning old Spamdb.helo records
2012-09-12 22:57:59 done - cleaning old Spamdb.helo records

2012-09-12 22:57:59 HELO Blacklist: 1 new, 427 now in list

I can't see any issues with HELO records/blacklist here.

On 12/09/2012 23:41, Daniel L. Miller wrote:
> On 9/12/2012 12:15 PM, Colin wrote:
>> I'm not sure there is a need to go to old backups.
>>
>> Run the steps that were suggested:
>>
>> Upgrade
>> Enable Test Mode
>> Run rebuildspamdb
>> Disable Test Mdoe
>> Rerun rebuildspamdb
>>
>> I have done this and am now getting:
>>
>> 2012-09-12 20:09:44 Corpus norm: 1. - (very good - balanced)
>> 2012-09-12 20:09:44 Corpus confidence: 1.
>> 2012-09-12 20:09:44 Recommendation: RebuildSpamDB will limit the number
>> of used messages in your corpus. Excess files will be ingored.
>>
> I also now have an apparent 'perfect' corpus after rebuild - but why
> does it indicate 0 HELO's in my list?  Shouldn't I have at least a few?!
>


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm

[Daniel L. Miller]

On 9/12/2012 12:15 PM, Colin wrote:
> I'm not sure there is a need to go to old backups.
>
> Run the steps that were suggested:
>
> Upgrade
> Enable Test Mode
> Run rebuildspamdb
> Disable Test Mdoe
> Rerun rebuildspamdb
>
> I have done this and am now getting:
>
> 2012-09-12 20:09:44 Corpus norm: 1. - (very good - balanced)
> 2012-09-12 20:09:44 Corpus confidence: 1.
> 2012-09-12 20:09:44 Recommendation: RebuildSpamDB will limit the number
> of used messages in your corpus. Excess files will be ingored.
>

I also now have an apparent 'perfect' corpus after rebuild - but why 
does it indicate 0 HELO's in my list?  Shouldn't I have at least a few?!

-- 
Daniel

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Speeding up MySQL

[Rusty Nejdl]

Here's some generic mysql tuning info:

Turn on logging of slow queries and figure out which queries are taking 
a while.  Run them through an analyzer to see if any indexes are 
missing.  mysqltuner and mysqlfragfinder are two scripts you can run to 
help tune the database.


(If I am telling you stuff you already know, don't take it personally.)

Rusty Nejdl

On 2012-09-12 15:37, Colin wrote:
I think this came up a long time ago, but the "populating Hidden 
Markov
Model" part of rebuildspamdb takes a very long time on my system 
(over

an hour).

This time I have been keeping an eye on it and I have no idea why.

For the majority of that hour, there is no CPU activity, no network
activity and a very small amount of disk write activity. For the last
minute, there was a burst of network and CPU activity.

Has anyone experience with how to tune up mysql so it works a bit
quicker during the rebuild but doesn't negatively affect the regular
operations during the day? This server does nothing except the ASSP
database..

All the best,
Colin Waring.


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. 
Discussions
will include endpoint security, mobile security and the latest in 
malware

threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Speeding up MySQL

[Colin]

I think this came up a long time ago, but the "populating Hidden Markov 
Model" part of rebuildspamdb takes a very long time on my system (over 
an hour).

This time I have been keeping an eye on it and I have no idea why.

For the majority of that hour, there is no CPU activity, no network 
activity and a very small amount of disk write activity. For the last 
minute, there was a burst of network and CPU activity.

Has anyone experience with how to tune up mysql so it works a bit 
quicker during the rebuild but doesn't negatively affect the regular 
operations during the day? This server does nothing except the ASSP 
database..

All the best,
Colin Waring.

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm

[Colin]

I'm not sure there is a need to go to old backups.

Run the steps that were suggested:

Upgrade
Enable Test Mode
Run rebuildspamdb
Disable Test Mdoe
Rerun rebuildspamdb

I have done this and am now getting:

2012-09-12 20:09:44 Corpus norm: 1. - (very good - balanced)
2012-09-12 20:09:44 Corpus confidence: 1.
2012-09-12 20:09:44 Recommendation: RebuildSpamDB will limit the number 
of used messages in your corpus. Excess files will be ingored.

Perfect!

On 12/09/2012 19:31, Steve Moffat wrote:
> I have a vm from last month. I'll just restore it that way.
>
> Steve Moffat
> Operations Director
> Optimum IT Solutions
> Tel 441-292-8849
>
> On Sep 12, 2012, at 2:50 PM, "Hill, Brett"  wrote:
>
>> I learned that a long time ago when my corpus was massacred due to some bad 
>> code in rebuildspamdb.  Been backing up ever since.  I've just got a batch 
>> file that backs it up into .7z and keeps a rolling history.
>>
>>> -Original Message-
>>> From: Steve Moffat [mailto:st...@optimum.bm]
>>> Sent: Wednesday, September 12, 2012 1:02 PM
>>> To: ASSP development mailing list
>>> Cc: ASSP development mailing list
>>> Subject: Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm
>>>
>>> Nope. 😄 but I will be from now on.
>>>
>>> Steve Moffat
>>> Operations Director
>>> Optimum IT Solutions
>>> Tel 441-292-8849
>> --
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
> --
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm

[Steve Moffat]

I have a vm from last month. I'll just restore it that way. 

Steve Moffat
Operations Director
Optimum IT Solutions
Tel 441-292-8849

On Sep 12, 2012, at 2:50 PM, "Hill, Brett"  wrote:

> I learned that a long time ago when my corpus was massacred due to some bad 
> code in rebuildspamdb.  Been backing up ever since.  I've just got a batch 
> file that backs it up into .7z and keeps a rolling history.
> 
>> -Original Message-
>> From: Steve Moffat [mailto:st...@optimum.bm]
>> Sent: Wednesday, September 12, 2012 1:02 PM
>> To: ASSP development mailing list
>> Cc: ASSP development mailing list
>> Subject: Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm
>> 
>> Nope. 😄 but I will be from now on.
>> 
>> Steve Moffat
>> Operations Director
>> Optimum IT Solutions
>> Tel 441-292-8849
> 
> --
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Antwort: FW: RebuildSpamDB - report from assp.isp.bm

[Colin]

After upgrading I turned on test mode and ran a rebuild which produced 
the following so back to normal.

2012-09-12 18:53:19 Corpus norm:0.9942 - (very good - balanced)
2012-09-12 18:53:19 Corpus confidence:  1.
2012-09-12 18:53:19 Recommendation: RebuildSpamDB will limit the number 
of used messages in your corpus. Excess files will be ingored.

I've turned test mode off and will run a proper rebuild now.
Thanks Thomas.


On 12/09/2012 18:06, Thomas Eckardt wrote:
> - restore the corpus
> - remove the file 'normfile'
> - set 'RebuildTestMode' to on
> - run a rebuild
> - run a rebuild
>
> Thomas
>
>
>
>
> Von:Steve Moffat 
> An: "'assp-test@lists.sourceforge.net'"
> ,
> Datum:  12.09.2012 18:51
> Betreff:[Assp-test] FW: RebuildSpamDB - report from assp.isp.bm
>
>
>
> Hi, Just ran rebuildspamdb with the new release. The results are even
> worse.before this I had a perfect corpus
>
> Steve
>
> -Original Message-
> From: assp@assp.local [mailto:assp@assp.local]
> Sent: Wednesday, September 12, 2012 1:50 PM
> To: Steve Moffat
> Subject: RebuildSpamDB - report from assp.isp.bm
>
> File rebuildrun.txt follows:
>
>
>
> Sep-12-12 13:18:25 RebuildSpamDB-thread rebuildspamdb-version 6.02 started
> in ASSP version 2.2.2(12256)
>
> Sep-12-12 13:18:25 RebuildSpamDB will create a Hidden Markov Model!
>
> Sep-12-12 13:18:25 RebuildSpamDB will include attachment-database-entries
> in to spamdb!
>
> Sep-12-12 13:18:25 RebuildSpamDB will create unicode enabled databases.
>
> Sep-12-12 13:18:25 RebuildSpamDB process all words as Sequence of UAX #29
> Grapheme Clusters.
>
> Sep-12-12 13:18:25 RebuildSpamDB will use the ASSP_WordStem engine.
>
> Sep-12-12 13:18:25 ---ASSP Settings---
> Sep-12-12 13:18:25 Do Not Collect RedRe Messages: Enabled **Messages
> matching the RedRe will be removed from the corpus!**
>
> Sep-12-12 13:18:25 Use Subject as Maillog Names: True
> Sep-12-12 13:18:25 Maxbytes: 4000
> Sep-12-12 13:18:25 RebuildFileTimeLimit: 1 5
> Sep-12-12 13:18:25 RebuildFileTimeLimit: files will be moved away from the
> corpus, if there processing takes longer than 5 second(s)
>
> Sep-12-12 13:18:25 C:/assp/errors/spam
> Sep-12-12 13:18:25 File Count:   319
> Sep-12-12 13:18:25 Processing... errors/spam with 319 files
> Sep-12-12 13:18:25 ignore and remove files older than Dec-17-09 12:18:25
> in folder errors/spam
> Sep-12-12 13:18:33 1 attachment/image entries processed
> Sep-12-12 13:18:33 Imported Files:   317
> Sep-12-12 13:18:33 Finished in 8 second(s)
>
> Sep-12-12 13:18:33 C:/assp/errors/notspam
> Sep-12-12 13:18:33 File Count:   113
> Sep-12-12 13:18:33 Processing... errors/notspam with 113 files
> Sep-12-12 13:18:33 ignore and remove files older than Dec-17-09 12:18:33
> in folder errors/notspam
> Sep-12-12 13:18:40 26 attachment/image entries processed
> Sep-12-12 13:18:40 Imported Files:   111
> Sep-12-12 13:18:40 Finished in 7 second(s)
> Sep-12-12 13:18:40 warning: missing information for automatic corpus
> correction in file C:/assp/normfile - rerun the rebuild, if you see this
> warning the first time!
>
> Sep-12-12 13:18:40 C:/assp/spam
> Sep-12-12 13:18:40 File Count:   4,363
> Sep-12-12 13:18:40 Processing... spam with 4,363 files
> Sep-12-12 13:19:27 remove
> C:/assp/spam/Confirmation_of_changes_to_Boo--140013.eml WhiteList:
> 'ba.custs...@contact.britishairways.com'
> Sep-12-12 13:19:27 remove
> C:/assp/spam/Confirmation_of_changes_to_Boo--144011.eml WhiteList:
> 'ba.custs...@contact.britishairways.com'
> Sep-12-12 13:19:27 remove
> C:/assp/spam/Confirmation_of_changes_to_Boo--145936.eml WhiteList:
> 'ba.custs...@contact.britishairways.com'
> Sep-12-12 13:19:27 remove
> C:/assp/spam/Confirmation_of_changes_to_Boo--172792.eml WhiteList:
> 'ba.custs...@contact.britishairways.com'
> Sep-12-12 13:20:07 remove
> C:/assp/spam/FW_Time_Clarification_Walk_the--81794.eml WhiteList:
> 'busbysu...@hotmail.com'
> Sep-12-12 13:22:50 Removed White:5
> Sep-12-12 13:22:50 481 attachment/image entries processed
> Sep-12-12 13:22:50 Imported Files:   4,356
> Sep-12-12 13:22:50 Finished in 250 second(s)
>
> Sep-12-12 13:22:50 C:/assp/notspam
> Sep-12-12 13:22:50 File Count:   12,640
> Sep-12-12 13:22:50 Processing... notspam with 12,000 files
> Sep-12-12 13:42:28 2,022 attachment/image entries processed
> Sep-12-12 13:42:28 Imported Files:   12,001
> Sep-12-12 13:42:28 Folder contents exceeded 'MaxFiles'(12000).
> Sep-12-12 13:42:28 Finished in 1,178 second(s)
>
> Sep-12-12 13:42:28 Rebuild processed 11.63 files per second.
>
> Sep-12-12 13:42:28 Generating weighted Bayesian tuplets
> Sep-12-12 13:42:38 start populating Spamdb with 175,796 records - Bayesian
> check is now disabled!
> Sep-12-12 13:43:45 Finished populating Spamdb with 175,796 records -
> Bayesian check is now enabled!
> Sep-12-12 13:43:45 done - Generating weighted Bayesian tuplets
>
> Sep-12-12 13:43:45 Bayesian

Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm

[Hill, Brett]

I learned that a long time ago when my corpus was massacred due to some bad 
code in rebuildspamdb.  Been backing up ever since.  I've just got a batch file 
that backs it up into .7z and keeps a rolling history.

> -Original Message-
> From: Steve Moffat [mailto:st...@optimum.bm]
> Sent: Wednesday, September 12, 2012 1:02 PM
> To: ASSP development mailing list
> Cc: ASSP development mailing list
> Subject: Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm
> 
> Nope. 😄 but I will be from now on.
> 
> Steve Moffat
> Operations Director
> Optimum IT Solutions
> Tel 441-292-8849

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Antwort: FW: RebuildSpamDB - report from assp.isp.bm

[Thomas Eckardt]

- restore the corpus
- remove the file 'normfile'
- set 'RebuildTestMode' to on
- run a rebuild
- run a rebuild

Thomas




Von:Steve Moffat 
An: "'assp-test@lists.sourceforge.net'" 
, 
Datum:  12.09.2012 18:51
Betreff:[Assp-test] FW: RebuildSpamDB - report from assp.isp.bm



Hi, Just ran rebuildspamdb with the new release. The results are even 
worse.before this I had a perfect corpus

Steve

-Original Message-
From: assp@assp.local [mailto:assp@assp.local] 
Sent: Wednesday, September 12, 2012 1:50 PM
To: Steve Moffat
Subject: RebuildSpamDB - report from assp.isp.bm

File rebuildrun.txt follows:



Sep-12-12 13:18:25 RebuildSpamDB-thread rebuildspamdb-version 6.02 started 
in ASSP version 2.2.2(12256)

Sep-12-12 13:18:25 RebuildSpamDB will create a Hidden Markov Model!

Sep-12-12 13:18:25 RebuildSpamDB will include attachment-database-entries 
in to spamdb!

Sep-12-12 13:18:25 RebuildSpamDB will create unicode enabled databases.

Sep-12-12 13:18:25 RebuildSpamDB process all words as Sequence of UAX #29 
Grapheme Clusters.

Sep-12-12 13:18:25 RebuildSpamDB will use the ASSP_WordStem engine.

Sep-12-12 13:18:25 ---ASSP Settings---
Sep-12-12 13:18:25 Do Not Collect RedRe Messages: Enabled **Messages 
matching the RedRe will be removed from the corpus!**

Sep-12-12 13:18:25 Use Subject as Maillog Names: True
Sep-12-12 13:18:25 Maxbytes: 4000
Sep-12-12 13:18:25 RebuildFileTimeLimit: 1 5
Sep-12-12 13:18:25 RebuildFileTimeLimit: files will be moved away from the 
corpus, if there processing takes longer than 5 second(s) 

Sep-12-12 13:18:25 C:/assp/errors/spam
Sep-12-12 13:18:25 File Count:   319
Sep-12-12 13:18:25 Processing... errors/spam with 319 files
Sep-12-12 13:18:25 ignore and remove files older than Dec-17-09 12:18:25 
in folder errors/spam
Sep-12-12 13:18:33 1 attachment/image entries processed
Sep-12-12 13:18:33 Imported Files:   317
Sep-12-12 13:18:33 Finished in 8 second(s)

Sep-12-12 13:18:33 C:/assp/errors/notspam
Sep-12-12 13:18:33 File Count:   113
Sep-12-12 13:18:33 Processing... errors/notspam with 113 files
Sep-12-12 13:18:33 ignore and remove files older than Dec-17-09 12:18:33 
in folder errors/notspam
Sep-12-12 13:18:40 26 attachment/image entries processed
Sep-12-12 13:18:40 Imported Files:   111
Sep-12-12 13:18:40 Finished in 7 second(s)
Sep-12-12 13:18:40 warning: missing information for automatic corpus 
correction in file C:/assp/normfile - rerun the rebuild, if you see this 
warning the first time!

Sep-12-12 13:18:40 C:/assp/spam
Sep-12-12 13:18:40 File Count:   4,363
Sep-12-12 13:18:40 Processing... spam with 4,363 files
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--140013.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--144011.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--145936.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--172792.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:20:07 remove 
C:/assp/spam/FW_Time_Clarification_Walk_the--81794.eml WhiteList: 
'busbysu...@hotmail.com'
Sep-12-12 13:22:50 Removed White:5
Sep-12-12 13:22:50 481 attachment/image entries processed
Sep-12-12 13:22:50 Imported Files:   4,356
Sep-12-12 13:22:50 Finished in 250 second(s)

Sep-12-12 13:22:50 C:/assp/notspam
Sep-12-12 13:22:50 File Count:   12,640
Sep-12-12 13:22:50 Processing... notspam with 12,000 files
Sep-12-12 13:42:28 2,022 attachment/image entries processed
Sep-12-12 13:42:28 Imported Files:   12,001
Sep-12-12 13:42:28 Folder contents exceeded 'MaxFiles'(12000). 
Sep-12-12 13:42:28 Finished in 1,178 second(s)

Sep-12-12 13:42:28 Rebuild processed 11.63 files per second.

Sep-12-12 13:42:28 Generating weighted Bayesian tuplets
Sep-12-12 13:42:38 start populating Spamdb with 175,796 records - Bayesian 
check is now disabled!
Sep-12-12 13:43:45 Finished populating Spamdb with 175,796 records - 
Bayesian check is now enabled!
Sep-12-12 13:43:45 done - Generating weighted Bayesian tuplets

Sep-12-12 13:43:45 Bayesian Pairs: 175,796 now in list

Sep-12-12 13:43:45 Generating consolidated Hidden-Markov-Model database 
from 1,634,405 record model
Sep-12-12 13:45:16 HMM sequences: 800,876 now in list

Sep-12-12 13:45:16 generating Spamdb.helo records from 3,664 collected 
HELO's
Sep-12-12 13:45:16 cleaning old Spamdb.helo records
Sep-12-12 13:45:17 done - cleaning old Spamdb.helo records

Sep-12-12 13:45:17 HELO Blacklist: 3 new, 94 now in list

Sep-12-12 13:45:17 Spam Weight: 1,598,969
Sep-12-12 13:45:17 Not-Spam Weight:   4,554,517

Sep-12-12 13:45:17 Corpus norm:  0.3511 - (warning: extremely ham 
heavy)
Sep-12-12 13:45:17 Corpus confidence:0.1352

[Assp-test] Antwort: FW: RebuildSpamDB - report from assp.isp.bm

[Thomas Eckardt]

>Sep-12-12 13:18:40 warning: missing information for automatic corpus 
correction in file C:/assp/normfile - rerun the rebuild, if you see this 
warning the first time!


changelog:

...
- The automatic corpus correction in the rebuildspamdb task was not 
working as expected in every case.
  There is one rebuild task required to enable this feature in future. 


Steve - the last run was a default - without any correction.

>before this I had a perfect corpus

I don't beleve that -. because

Sep-12-12 13:18:40 File Count:   4,363
Sep-12-12 13:18:40 Processing... spam with 4,363 files

Sep-12-12 13:22:50 C:/assp/notspam
Sep-12-12 13:42:28 Imported Files:   12,001



Thomas




Von:Steve Moffat 
An: "'assp-test@lists.sourceforge.net'" 
, 
Datum:  12.09.2012 18:51
Betreff:[Assp-test] FW: RebuildSpamDB - report from assp.isp.bm



Hi, Just ran rebuildspamdb with the new release. The results are even 
worse.before this I had a perfect corpus

Steve

-Original Message-
From: assp@assp.local [mailto:assp@assp.local] 
Sent: Wednesday, September 12, 2012 1:50 PM
To: Steve Moffat
Subject: RebuildSpamDB - report from assp.isp.bm

File rebuildrun.txt follows:



Sep-12-12 13:18:25 RebuildSpamDB-thread rebuildspamdb-version 6.02 started 
in ASSP version 2.2.2(12256)

Sep-12-12 13:18:25 RebuildSpamDB will create a Hidden Markov Model!

Sep-12-12 13:18:25 RebuildSpamDB will include attachment-database-entries 
in to spamdb!

Sep-12-12 13:18:25 RebuildSpamDB will create unicode enabled databases.

Sep-12-12 13:18:25 RebuildSpamDB process all words as Sequence of UAX #29 
Grapheme Clusters.

Sep-12-12 13:18:25 RebuildSpamDB will use the ASSP_WordStem engine.

Sep-12-12 13:18:25 ---ASSP Settings---
Sep-12-12 13:18:25 Do Not Collect RedRe Messages: Enabled **Messages 
matching the RedRe will be removed from the corpus!**

Sep-12-12 13:18:25 Use Subject as Maillog Names: True
Sep-12-12 13:18:25 Maxbytes: 4000
Sep-12-12 13:18:25 RebuildFileTimeLimit: 1 5
Sep-12-12 13:18:25 RebuildFileTimeLimit: files will be moved away from the 
corpus, if there processing takes longer than 5 second(s) 

Sep-12-12 13:18:25 C:/assp/errors/spam
Sep-12-12 13:18:25 File Count:   319
Sep-12-12 13:18:25 Processing... errors/spam with 319 files
Sep-12-12 13:18:25 ignore and remove files older than Dec-17-09 12:18:25 
in folder errors/spam
Sep-12-12 13:18:33 1 attachment/image entries processed
Sep-12-12 13:18:33 Imported Files:   317
Sep-12-12 13:18:33 Finished in 8 second(s)

Sep-12-12 13:18:33 C:/assp/errors/notspam
Sep-12-12 13:18:33 File Count:   113
Sep-12-12 13:18:33 Processing... errors/notspam with 113 files
Sep-12-12 13:18:33 ignore and remove files older than Dec-17-09 12:18:33 
in folder errors/notspam
Sep-12-12 13:18:40 26 attachment/image entries processed
Sep-12-12 13:18:40 Imported Files:   111
Sep-12-12 13:18:40 Finished in 7 second(s)
Sep-12-12 13:18:40 warning: missing information for automatic corpus 
correction in file C:/assp/normfile - rerun the rebuild, if you see this 
warning the first time!

Sep-12-12 13:18:40 C:/assp/spam
Sep-12-12 13:18:40 File Count:   4,363
Sep-12-12 13:18:40 Processing... spam with 4,363 files
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--140013.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--144011.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--145936.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--172792.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:20:07 remove 
C:/assp/spam/FW_Time_Clarification_Walk_the--81794.eml WhiteList: 
'busbysu...@hotmail.com'
Sep-12-12 13:22:50 Removed White:5
Sep-12-12 13:22:50 481 attachment/image entries processed
Sep-12-12 13:22:50 Imported Files:   4,356
Sep-12-12 13:22:50 Finished in 250 second(s)

Sep-12-12 13:22:50 C:/assp/notspam
Sep-12-12 13:22:50 File Count:   12,640
Sep-12-12 13:22:50 Processing... notspam with 12,000 files
Sep-12-12 13:42:28 2,022 attachment/image entries processed
Sep-12-12 13:42:28 Imported Files:   12,001
Sep-12-12 13:42:28 Folder contents exceeded 'MaxFiles'(12000). 
Sep-12-12 13:42:28 Finished in 1,178 second(s)

Sep-12-12 13:42:28 Rebuild processed 11.63 files per second.

Sep-12-12 13:42:28 Generating weighted Bayesian tuplets
Sep-12-12 13:42:38 start populating Spamdb with 175,796 records - Bayesian 
check is now disabled!
Sep-12-12 13:43:45 Finished populating Spamdb with 175,796 records - 
Bayesian check is now enabled!
Sep-12-12 13:43:45 done - Generating weighted Bayesian tuplets

Sep-12-12 13:43:45 Bayesian Pairs: 175,796 now in list

Sep-12-12 13:43:45 Generating consolidated Hidden-Markov-Model da

Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm

[Steve Moffat]

Nope. 😄 but I will be from now on. 

Steve Moffat
Operations Director
Optimum IT Solutions
Tel 441-292-8849

On Sep 12, 2012, at 1:56 PM, "Hill, Brett"  wrote:

> Hopefully you keep backups of your corpse to restore it back to near
> perfect...
> 
>> -Original Message-
>> From: Steve Moffat [mailto:st...@optimum.bm]
>> Sent: Wednesday, September 12, 2012 12:51 PM
>> To: 'assp-test@lists.sourceforge.net'
>> Subject: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm
>> 
>> Hi, Just ran rebuildspamdb with the new release. The results are even
>> worse.before this I had a perfect corpus
>> 
>> Steve 
> 
> 
> 
> --
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm

[Hill, Brett]

Hopefully you keep backups of your corpse to restore it back to near
perfect...

> -Original Message-
> From: Steve Moffat [mailto:st...@optimum.bm]
> Sent: Wednesday, September 12, 2012 12:51 PM
> To: 'assp-test@lists.sourceforge.net'
> Subject: [Assp-test] FW: RebuildSpamDB - report from assp.isp.bm
> 
> Hi, Just ran rebuildspamdb with the new release. The results are even
> worse.before this I had a perfect corpus
> 
> Steve 



--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] FW: RebuildSpamDB - report from assp.isp.bm

[Steve Moffat]

Hi, Just ran rebuildspamdb with the new release. The results are even 
worse.before this I had a perfect corpus

Steve

-Original Message-
From: assp@assp.local [mailto:assp@assp.local] 
Sent: Wednesday, September 12, 2012 1:50 PM
To: Steve Moffat
Subject: RebuildSpamDB - report from assp.isp.bm

File rebuildrun.txt follows:



Sep-12-12 13:18:25 RebuildSpamDB-thread rebuildspamdb-version 6.02 started in 
ASSP version 2.2.2(12256)

Sep-12-12 13:18:25 RebuildSpamDB will create a Hidden Markov Model!

Sep-12-12 13:18:25 RebuildSpamDB will include attachment-database-entries in to 
spamdb!

Sep-12-12 13:18:25 RebuildSpamDB will create unicode enabled databases.

Sep-12-12 13:18:25 RebuildSpamDB process all words as Sequence of UAX #29 
Grapheme Clusters.

Sep-12-12 13:18:25 RebuildSpamDB will use the ASSP_WordStem engine.

Sep-12-12 13:18:25 ---ASSP Settings---
Sep-12-12 13:18:25 Do Not Collect RedRe Messages: Enabled **Messages matching 
the RedRe will be removed from the corpus!**

Sep-12-12 13:18:25 Use Subject as Maillog Names: True
Sep-12-12 13:18:25 Maxbytes: 4000
Sep-12-12 13:18:25 RebuildFileTimeLimit: 1 5
Sep-12-12 13:18:25 RebuildFileTimeLimit: files will be moved away from the 
corpus, if there processing takes longer than 5 second(s) 

Sep-12-12 13:18:25 C:/assp/errors/spam
Sep-12-12 13:18:25 File Count:  319
Sep-12-12 13:18:25 Processing... errors/spam with 319 files
Sep-12-12 13:18:25 ignore and remove files older than Dec-17-09 12:18:25 in 
folder errors/spam
Sep-12-12 13:18:33 1 attachment/image entries processed
Sep-12-12 13:18:33 Imported Files:  317
Sep-12-12 13:18:33 Finished in 8 second(s)

Sep-12-12 13:18:33 C:/assp/errors/notspam
Sep-12-12 13:18:33 File Count:  113
Sep-12-12 13:18:33 Processing... errors/notspam with 113 files
Sep-12-12 13:18:33 ignore and remove files older than Dec-17-09 12:18:33 in 
folder errors/notspam
Sep-12-12 13:18:40 26 attachment/image entries processed
Sep-12-12 13:18:40 Imported Files:  111
Sep-12-12 13:18:40 Finished in 7 second(s)
Sep-12-12 13:18:40 warning: missing information for automatic corpus correction 
in file C:/assp/normfile - rerun the rebuild, if you see this warning the first 
time!

Sep-12-12 13:18:40 C:/assp/spam
Sep-12-12 13:18:40 File Count:  4,363
Sep-12-12 13:18:40 Processing... spam with 4,363 files
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--140013.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--144011.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--145936.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:19:27 remove 
C:/assp/spam/Confirmation_of_changes_to_Boo--172792.eml WhiteList: 
'ba.custs...@contact.britishairways.com'
Sep-12-12 13:20:07 remove 
C:/assp/spam/FW_Time_Clarification_Walk_the--81794.eml WhiteList: 
'busbysu...@hotmail.com'
Sep-12-12 13:22:50 Removed White:   5
Sep-12-12 13:22:50 481 attachment/image entries processed
Sep-12-12 13:22:50 Imported Files:  4,356
Sep-12-12 13:22:50 Finished in 250 second(s)

Sep-12-12 13:22:50 C:/assp/notspam
Sep-12-12 13:22:50 File Count:  12,640
Sep-12-12 13:22:50 Processing... notspam with 12,000 files
Sep-12-12 13:42:28 2,022 attachment/image entries processed
Sep-12-12 13:42:28 Imported Files:  12,001
Sep-12-12 13:42:28 Folder contents exceeded 'MaxFiles'(12000). 
Sep-12-12 13:42:28 Finished in 1,178 second(s)

Sep-12-12 13:42:28 Rebuild processed 11.63 files per second.

Sep-12-12 13:42:28 Generating weighted Bayesian tuplets
Sep-12-12 13:42:38 start populating Spamdb with 175,796 records - Bayesian 
check is now disabled!
Sep-12-12 13:43:45 Finished populating Spamdb with 175,796 records - Bayesian 
check is now enabled!
Sep-12-12 13:43:45 done - Generating weighted Bayesian tuplets

Sep-12-12 13:43:45 Bayesian Pairs: 175,796 now in list

Sep-12-12 13:43:45 Generating consolidated Hidden-Markov-Model database from 
1,634,405 record model
Sep-12-12 13:45:16 HMM sequences: 800,876 now in list

Sep-12-12 13:45:16 generating Spamdb.helo records from 3,664 collected HELO's
Sep-12-12 13:45:16 cleaning old Spamdb.helo records
Sep-12-12 13:45:17 done - cleaning old Spamdb.helo records

Sep-12-12 13:45:17 HELO Blacklist: 3 new, 94 now in list

Sep-12-12 13:45:17 Spam Weight:1,598,969
Sep-12-12 13:45:17 Not-Spam Weight:   4,554,517

Sep-12-12 13:45:17 Corpus norm: 0.3511 - (warning: extremely ham heavy)
Sep-12-12 13:45:17 Corpus confidence:   0.13526783
Sep-12-12 13:45:17 Recommendation: RebuildSpamDB will limit the number of used 
messages in your corpus. Excess files will be ingored.
Sep-12-12 13:45:17 Corpus norm should be between 0.6 and 1.4

Sep-12-12 13:45:17 Recommendation: You need more spam messages in the corpus.

Sep-12-12 13:45:17 starting auto correction for corpus - delete old ham files 
from notspam

Sep-12-12 13:45:22 info: start

[Assp-test] fixes in assp 2.2.2 build 12256

[Thomas Eckardt]

Hi all,

fixed in assp 2.2.2 build 12256:

- The automatic corpus correction in the rebuildspamdb task was not 
working as expected in every case.
  There is one rebuild task required to enable this feature in future.
  You may use the 'RebuildTestMode' to speed up this single task.
 

added:

'RebuildTestMode','Run the Rebuild in Test Mode'
'If selected, all rebuildspamdb tasks will not populate the spamdb and 
hmmdb - and no data will be sent 
to the griplist-Server.'

Thomas

DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] no sender email in log => ASSP 1.9.7.5(0.0.02)

[Fritz Borgstedt]

ASSP development mailing list  schrei
bt:
>is it a good/bad idea to show in ASSP log the From: value in case of t
>he envelope sender is not specified/empty ?


empty means bounce

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] New version

[Daniel L. Miller]

Here's mine:



Sep-12-12 01:57:25 RebuildSpamDB-thread rebuildspamdb-version 6.01 
started in ASSP version 2.2.2(12255)

Sep-12-12 01:57:25 RebuildSpamDB will create a Hidden Markov Model!

Sep-12-12 01:57:25 RebuildSpamDB will create unicode enabled databases.

Sep-12-12 01:57:25 RebuildSpamDB process all words as Sequence of UAX 
#29 Grapheme Clusters.

Sep-12-12 01:57:25 RebuildSpamDB will use the ASSP_WordStem engine.

Sep-12-12 01:57:25 ---ASSP Settings---
Sep-12-12 01:57:25 Do Not Collect Messages with RedListed address: Enabled
**Messages with RedListed addresses will be removed from the corpus!**

Sep-12-12 01:57:25 Do Not Collect RedRe Messages: Enabled
**Messages matching the RedRe will be removed from the corpus!**

Sep-12-12 01:57:25 Use Subject as Maillog Names: False
Sep-12-12 01:57:25 Maxbytes: 5000
Sep-12-12 01:57:25 RebuildFileTimeLimit: 1 5
Sep-12-12 01:57:25 RebuildFileTimeLimit: files will be moved away from 
the corpus, if there processing takes longer than 5 second(s)

Sep-12-12 01:57:25 /opt/assp2/errors/spam
Sep-12-12 01:57:25 File Count:646
Sep-12-12 01:57:25 Processing... errors/spam with 646 files
Sep-12-12 01:58:06 Imported Files:644
Sep-12-12 01:58:06 Finished in 41 second(s)

Sep-12-12 01:58:06 /opt/assp2/errors/notspam
Sep-12-12 01:58:06 File Count:130
Sep-12-12 01:58:06 Processing... errors/notspam with 130 files
Sep-12-12 01:58:12 Imported Files:128
Sep-12-12 01:58:12 Finished in 6 second(s)
Sep-12-12 01:58:12 info: corpusnorm after processing errors/spam and 
errors/notspam is spamwords 639358/ hamwords 191376 => 3.34084733717917
Sep-12-12 01:58:12 info: require 1223 files from folder spam to get a 
fine corpusnorm (1)

Sep-12-12 01:58:12 /opt/assp2/spam
Sep-12-12 01:58:12 File Count:4,089
Sep-12-12 01:58:12 Processing... spam with 1,223 files
Sep-12-12 01:58:12 ignore and remove files older than Aug-12-12 01:58:12 
in folder spam
Sep-12-12 01:59:12 Imported Files:1,224
Sep-12-12 01:59:12 Finished in 60 second(s)
Sep-12-12 01:59:12 info: require 4094 files from folder notspam to get a 
fine corpusnorm (1)

Sep-12-12 01:59:12 /opt/assp2/notspam
Sep-12-12 01:59:12 File Count:4,893
Sep-12-12 01:59:12 Processing... notspam with 4,094 files
Sep-12-12 02:02:29 Imported Files:4,095
Sep-12-12 02:02:29 Finished in 197 second(s)

Sep-12-12 02:02:29 Rebuild processed 20.17 files per second.

Sep-12-12 02:02:29 Generating weighted Bayesian tuplets
Sep-12-12 02:02:36 populating Spamdb 110995 records - Bayesian check is 
now disabled
Sep-12-12 02:02:43 done - populating Spamdb records - 110995 - Bayesian 
check is now enabled
Sep-12-12 02:02:43 done - Generating weighted Bayesian tuplets

Sep-12-12 02:02:43 Bayesian Pairs: 110,995 now in list

Sep-12-12 02:02:43 Generating consolidated Hidden-Markov-Model database 
from 1,416,063 record model
Sep-12-12 02:03:23 HMM sequences: 668,556 now in list

Sep-12-12 02:03:23 generating Spamdb.helo records from 2,353 collected 
HELO's
Sep-12-12 02:03:23 cleaning old Spamdb.helo records
Sep-12-12 02:03:23 done - cleaning old Spamdb.helo records

Sep-12-12 02:03:23 HELO Blacklist: 0 new, 0 now in list

Sep-12-12 02:03:23 Spam Weight:   1,085,713
Sep-12-12 02:03:23 Not-Spam Weight:   1,454,191

Sep-12-12 02:03:23 Corpus norm:0.7466 - (ok - slighly ham heavy)
Sep-12-12 02:03:23 Corpus confidence:0.63654233

Sep-12-12 02:03:28 Start populating Hidden Markov Model. HMM-check is 
disabled for this time!
Sep-12-12 02:03:29 start populating Hidden Markov Model with 668,556 
records!
Sep-12-12 02:03:41 Finished populating Hidden Markov Model with 668,556 
records!
Sep-12-12 02:03:41 Finished populating Hidden Markov Model. HMM-check is 
now enabled again!

Sep-12-12 02:03:41 Total processing time: 376 second(s)

Sep-12-12 02:03:41 Total processing data: 39.56 MByte

Sep-12-12 02:03:41 building new GripList records and bounce report
Sep-12-12 02:03:41 processing Logfile /opt/assp2/logs/maillog.txt
Sep-12-12 02:03:41 processing Logfile /opt/assp2/logs/12-09-11.maillog.txt
Sep-12-12 02:03:42 processing Logfile /opt/assp2/logs/12-09-10.maillog.txt
Sep-12-12 02:03:44 processing Logfile /opt/assp2/logs/12-09-09.maillog.txt
Sep-12-12 02:03:45 processing Logfile /opt/assp2/logs/12-09-08.maillog.txt
Sep-12-12 02:03:45 processing Logfile /opt/assp2/logs/12-09-07.maillog.txt

Sep-12-12 02:03:46 skipping bounce report because 'DoNotCollectBounces' 
is switched ON

Sep-12-12 02:03:46 Uploading Griplist via Direct Connection
Sep-12-12 02:03:47 Submitted 5,997 bytes: 0 IPv6 addresses, 665 IPv4 
addresses

Sep-12-12 02:03:47 Trashlist was saved to /opt/assp2/trashlist.db


-- 
Daniel

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.a

Re: [Assp-test] New version

[Scott MacLean]

I'm seeing the same behavior.

Before:

Sep-11-12 06:29:32 Corpus norm: 0.7039 - (ok - slighly ham heavy)
Sep-11-12 06:29:32 Corpus confidence: 0.59527529

After new version installed:

Sep-11-12 18:29:13 Corpus norm: 0.4919 - (warning: extremely ham heavy)
Sep-11-12 18:29:13 Corpus confidence: 0.19333149



At 03:49 PM 9/11/2012, Steve Moffat wrote:

>Hi
>I updated to the new release today and rebuildspamdb has ruined my 
>corpus confidence. Not too happy with that
>
>Sep-11-12 16:26:55 Spam Weight:3,904,196
>Sep-11-12 16:26:55 Not-Spam Weight:   1,950,092
>
>Sep-11-12 16:26:55 Corpus norm: 2.0021 - (warning: 
>extremely spam heavy)
>Sep-11-12 16:26:55 Corpus confidence:  0.06224349
>Sep-11-12 16:26:55 Recommendation: RebuildSpamDB will limit the 
>number of used messages in your corpus. Excess files will be ingored.
>Sep-11-12 16:26:55 Corpus norm should be between 0.6 and 1.4
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] FW: RebuildSpamDB - report from assp.isp.bm

[Steve Moffat]

As requested. This is last nights..

Steve

-Original Message-
From: assp@assp.local [mailto:assp@assp.local] 
Sent: Wednesday, September 12, 2012 12:20 AM
To: Steve Moffat
Subject: RebuildSpamDB - report from assp.isp.bm

File rebuildrun.txt follows:



Sep-12-12 00:00:02 RebuildSpamDB-thread rebuildspamdb-version 6.01 started in 
ASSP version 2.2.2(12255)

Sep-12-12 00:00:02 RebuildSpamDB will create a Hidden Markov Model!

Sep-12-12 00:00:02 RebuildSpamDB will include attachment-database-entries in to 
spamdb!

Sep-12-12 00:00:02 RebuildSpamDB will create unicode enabled databases.

Sep-12-12 00:00:02 RebuildSpamDB process all words as Sequence of UAX #29 
Grapheme Clusters.

Sep-12-12 00:00:02 RebuildSpamDB will use the ASSP_WordStem engine.

Sep-12-12 00:00:02 ---ASSP Settings---
Sep-12-12 00:00:02 Do Not Collect RedRe Messages: Enabled **Messages matching 
the RedRe will be removed from the corpus!**

Sep-12-12 00:00:02 Use Subject as Maillog Names: True
Sep-12-12 00:00:02 Maxbytes: 4000
Sep-12-12 00:00:02 RebuildFileTimeLimit: 1 5
Sep-12-12 00:00:02 RebuildFileTimeLimit: files will be moved away from the 
corpus, if there processing takes longer than 5 second(s) 

Sep-12-12 00:00:02 C:/assp/errors/spam
Sep-12-12 00:00:02 File Count:  318
Sep-12-12 00:00:02 Processing... errors/spam with 318 files
Sep-12-12 00:00:02 ignore and remove files older than Dec-16-09 23:00:02 in 
folder errors/spam
Sep-12-12 00:00:11 1 attachment/image entries processed
Sep-12-12 00:00:11 Imported Files:  316
Sep-12-12 00:00:11 Finished in 9 second(s)

Sep-12-12 00:00:11 C:/assp/errors/notspam
Sep-12-12 00:00:11 File Count:  113
Sep-12-12 00:00:11 Processing... errors/notspam with 113 files
Sep-12-12 00:00:11 ignore and remove files older than Dec-16-09 23:00:11 in 
folder errors/notspam
Sep-12-12 00:00:25 26 attachment/image entries processed
Sep-12-12 00:00:25 Imported Files:  111
Sep-12-12 00:00:25 Finished in 14 second(s)
Sep-12-12 00:00:25 info: corpusnorm after processing errors/spam and 
errors/notspam is spamwords 124138/ hamwords 271248 => 0.457654987317879
Sep-12-12 00:00:26 info: require 7230 files from folder spam to get a fine 
corpusnorm (1)

Sep-12-12 00:00:26 C:/assp/spam
Sep-12-12 00:00:26 File Count:  7,230
Sep-12-12 00:00:26 Processing... spam with 7,230 files
Sep-12-12 00:03:58 remove 
C:/assp/spam/Catherine_West_Hidden_in_the_H--172884.eml WhiteList: 
'cathw...@logic.bm'
Sep-12-12 00:08:18 Removed White:   1
Sep-12-12 00:08:18 707 attachment/image entries processed
Sep-12-12 00:08:18 Imported Files:  7,227
Sep-12-12 00:08:18 Finished in 472 second(s)
Sep-12-12 00:08:18 info: require 3384 files from folder notspam to get a fine 
corpusnorm (1)

Sep-12-12 00:08:18 C:/assp/notspam
Sep-12-12 00:08:18 File Count:  13,161
Sep-12-12 00:08:18 Processing... notspam with 3,384 files
Sep-12-12 00:11:29 2,395 attachment/image entries processed
Sep-12-12 00:11:29 Imported Files:  3,385
Sep-12-12 00:11:29 Folder contents exceeded 'MaxFiles'(12000). 
Sep-12-12 00:11:29 Finished in 191 second(s)

Sep-12-12 00:11:29 Rebuild processed 12.92 files per second.

Sep-12-12 00:11:29 Generating weighted Bayesian tuplets
Sep-12-12 00:11:41 start populating Spamdb with 115,501 records - Bayesian 
check is now disabled!
Sep-12-12 00:12:11 Finished populating Spamdb with 115,501 records - Bayesian 
check is now enabled!
Sep-12-12 00:12:11 done - Generating weighted Bayesian tuplets

Sep-12-12 00:12:11 Bayesian Pairs: 115,501 now in list

Sep-12-12 00:12:11 Generating consolidated Hidden-Markov-Model database from 
1,416,031 record model
Sep-12-12 00:13:26 HMM sequences: 697,084 now in list

Sep-12-12 00:13:26 generating Spamdb.helo records from 4,107 collected HELO's
Sep-12-12 00:13:27 cleaning old Spamdb.helo records
Sep-12-12 00:13:29 done - cleaning old Spamdb.helo records

Sep-12-12 00:13:29 HELO Blacklist: 4 new, 142 now in list

Sep-12-12 00:13:29 Spam Weight:2,442,496
Sep-12-12 00:13:29 Not-Spam Weight:   1,218,995

Sep-12-12 00:13:29 Corpus norm: 2.0037 - (warning: extremely spam heavy)
Sep-12-12 00:13:29 Corpus confidence:   0.06204007
Sep-12-12 00:13:29 Recommendation: RebuildSpamDB will limit the number of used 
messages in your corpus. Excess files will be ingored.
Sep-12-12 00:13:29 Corpus norm should be between 0.6 and 1.4

Sep-12-12 00:13:29 Recommendation: You need more not-spam messages in the 
corpus.

Sep-12-12 00:13:29 starting auto correction for corpus - delete old spam files 
from spam

Sep-12-12 00:13:33 info: starting cleanup for to much (old) files in folder 
C:/assp/spam - will try to remove 40% of the files - will keep at least 4000 
files - will keep files younger than 14 days
info: deleted 2890 old files from folder C:/assp/spam

Sep-12-12 00:13:33 Recommendation: You should increase now MaxBytes to 6000!  

Sep-12-12 00:13:38 Start populating Hidden Markov Model. HMM-check is disabled 
for this time!
Sep-12-12 00:13:38 start populating Hidden Markov Model with 697,

Re: [Assp-test] Antwort: Re: New version

[Colin]

Apologies,

I had the right bits copy/pasted into a notepad window but for some 
reason ended up with only half of it in the message. Full log below 
minus any lines containing filenames with subjects in them. Both my spam 
and notspam folders currently have just short of 15,000 messages in them.


2012-09-11 22:00:00 RebuildSpamDB-thread rebuildspamdb-version 6.01 
started in ASSP version 2.2.2(12255)

2012-09-11 22:00:00 RebuildSpamDB will create a Hidden Markov Model!

2012-09-11 22:00:00 RebuildSpamDB will create unicode enabled databases.

2012-09-11 22:00:00 RebuildSpamDB process all words as Sequence of UAX 
#29 Grapheme Clusters.

2012-09-11 22:00:00 RebuildSpamDB will use the ASSP_WordStem engine.

2012-09-11 22:00:00 ---ASSP Settings---
2012-09-11 22:00:00 Do Not Collect Messages with RedListed address: Enabled
**Messages with RedListed addresses will be removed from the corpus!**

2012-09-11 22:00:00 Do Not Collect RedRe Messages: Enabled
**Messages matching the RedRe will be removed from the corpus!**

2012-09-11 22:00:00 Use Subject as Maillog Names: True
2012-09-11 22:00:00 Maxbytes: 4000
2012-09-11 22:00:00 RebuildFileTimeLimit: 1 5
2012-09-11 22:00:00 RebuildFileTimeLimit: files will be moved away from 
the corpus, if there processing takes longer than 5 second(s)

2012-09-11 22:00:00 Trashlist cleaning finished, 30 of 106 files deleted

2012-09-11 22:00:00 /usr/local/assp/store/errors/spam
2012-09-11 22:00:00 File Count:1,422
2012-09-11 22:00:00 Processing... store/errors/spam with 1,422 files
2012-09-11 22:00:00 ignore and remove files older than 2009-12-16 
21:00:00 in folder store/errors/spam
2012-09-11 22:02:22 Imported Files:1,420
2012-09-11 22:02:22 Finished in 142 second(s)

2012-09-11 22:02:22 /usr/local/assp/store/errors/notspam
2012-09-11 22:02:22 File Count:1,392
2012-09-11 22:02:22 Processing... store/errors/notspam with 1,392 files
2012-09-11 22:02:22 ignore and remove files older than 2009-12-16 
21:02:22 in folder store/errors/notspam
2012-09-11 22:04:43 Imported Files:1,390
2012-09-11 22:04:43 Finished in 141 second(s)
2012-09-11 22:04:43 info: corpusnorm after processing store/errors/spam 
and store/errors/notspam is spamwords 996168/ hamwords 1886016 => 
0.528186399267026
2012-09-11 22:04:43 info: require 15163 files from folder store/spam to 
get a fine corpusnorm (1)

2012-09-11 22:04:43 /usr/local/assp/store/spam
2012-09-11 22:04:43 File Count:15,163
2012-09-11 22:04:43 Processing... store/spam with 14,000 files
2012-09-11 22:35:28 Removed White:2
2012-09-11 22:35:28 Imported Files:14,001
2012-09-11 22:35:28 Folder contents exceeded 'MaxFiles'(14000).
2012-09-11 22:35:28 Finished in 1,845 second(s)
2012-09-11 22:35:28 info: require 6506 files from folder store/notspam 
to get a fine corpusnorm (1)

2012-09-11 22:35:28 /usr/local/assp/store/notspam
2012-09-11 22:35:28 File Count:17,475
2012-09-11 22:35:28 Processing... store/notspam with 6,506 files
2012-09-11 22:48:45 Imported Files:6,507
2012-09-11 22:48:45 Folder contents exceeded 'MaxFiles'(14000).
2012-09-11 22:48:45 Finished in 797 second(s)

2012-09-11 22:48:45 Rebuild processed 7.97 files per second. Good values 
are 12 files per second and higher. You can speed up the rebuild 
process, using a cached (>=128MB) IO-controller or a RAM-disk with at 
least 1.04 GBbyte for the folder '/usr/local/assp/tmpDB'.

2012-09-11 22:48:45 Generating weighted Bayesian tuplets
2012-09-11 22:49:22 start populating Spamdb with 360,515 records - 
Bayesian check is now disabled!
2012-09-11 22:51:16 Finished populating Spamdb with 360,515 records - 
Bayesian check is now enabled!
2012-09-11 22:51:16 done - Generating weighted Bayesian tuplets

2012-09-11 22:51:16 Bayesian Pairs: 360,515 now in list

2012-09-11 22:51:18 Generating consolidated Hidden-Markov-Model database 
from 5,252,212 record model
2012-09-11 22:56:39 HMM sequences: 2,553,143 now in list

2012-09-11 22:56:39 generating Spamdb.helo records from 7,212 collected 
HELO's
2012-09-11 22:56:59 cleaning old Spamdb.helo records
2012-09-11 22:57:01 done - cleaning old Spamdb.helo records

2012-09-11 22:57:01 HELO Blacklist: 52 new, 547 now in list

2012-09-11 22:57:01 Spam Weight:   7,281,890
2012-09-11 22:57:01 Not-Spam Weight:   4,460,847

2012-09-11 22:57:01 Corpus norm:1.6324 - (warning: extremely spam heavy)
2012-09-11 22:57:01 Corpus confidence:0.14082925
2012-09-11 22:57:01 Recommendation: RebuildSpamDB will limit the number 
of used messages in your corpus. Excess files will be ingored.
2012-09-11 22:57:01 Corpus norm should be between 0.6 and 1.4

2012-09-11 22:57:01 Recommendation: You need more not-spam messages in 
the corpus.

2012-09-11 22:57:01 starting auto correction for corpus - delete old 
spam files from store/spam

2012-09-11 22:57:02 info: starting cleanup for to much (old) files in 
folder /usr/local/assp/store/spam - will try to remove 40% of the files 
- will keep at least 4000 files - will keep files y

[Assp-test] Antwort: Re: New version

[Thomas Eckardt]

Same to you Colin,

to verify how the new code is working I need at least all the output about 
all the folders and the the resulting corpusnorm.

Thomas




Von:Colin 
An: assp-test@lists.sourceforge.net, 
Datum:  12.09.2012 10:28
Betreff:Re: [Assp-test] New version



I am seeing the same although not quite as bad. It seems to be related 
to the notspam folder and less than half of the files in it being 
processed.

Yesterday I had:

2012-09-10 23:15:39 Corpus norm: 0.9749 - (very good - 
balanced)
2012-09-10 23:15:39 Corpus confidence:   1.

Now I have:

2012-09-11 22:04:43 /usr/local/assp/store/spam
2012-09-11 22:04:43 File Count:  15,163
2012-09-11 22:04:43 Processing... store/spam with 14,000 files
2012-09-11 22:35:28 Imported Files:  14,001
2012-09-11 22:35:28 Folder contents exceeded 'MaxFiles'(14000).
2012-09-11 22:35:28 Finished in 1,845 second(s)
2012-09-11 22:35:28 info: require 6506 files from folder store/notspam to 
get a fine corpusnorm (1)

2012-09-11 22:35:28 /usr/local/assp/store/notspam
2012-09-11 22:35:28 File Count:  17,475
2012-09-11 22:35:28 Processing... store/notspam with 6,506 files
2012-09-11 22:47:45 remove/usr/local/assp/store/notspam/--522681.eml 
corrected spam
2012-09-11 22:48:45 Imported Files:  6,507
2012-09-11 22:48:45 Folder contents exceeded 'MaxFiles'(14000).
2012-09-11 22:48:45 Finished in 797 second(s)

All the best,
Colin Waring.

On 11/09/2012 20:49, Steve Moffat wrote:
> Hi
> I updated to the new release today and rebuildspamdb has ruined my 
corpus confidence. Not too happy with that
>
> Sep-11-12 16:26:55 Spam Weight:3,904,196
> Sep-11-12 16:26:55 Not-Spam Weight:   1,950,092
>
> Sep-11-12 16:26:55 Corpus norm: 2.0021 - (warning: extremely 
spam heavy)
> Sep-11-12 16:26:55 Corpus confidence:  0.06224349
> Sep-11-12 16:26:55 Recommendation: RebuildSpamDB will limit the number 
of used messages in your corpus. Excess files will be ingored.
> Sep-11-12 16:26:55 Corpus norm should be between 0.6 and 1.4
>
> Thanks
> Steve
> Steve Moffat
> Operations Director
> Optimum IT Solutions
> Desk:   441 292 8849
> Mobile: 441 292 8849
> MSN IM:st...@optimum.bm
> Web:http://www.optimum.bm
>
> 
--
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. 
Discussions
> will include endpoint security, mobile security and the latest in 
malware
> threats.http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] New version

[Colin]

I am seeing the same although not quite as bad. It seems to be related 
to the notspam folder and less than half of the files in it being 
processed.

Yesterday I had:

2012-09-10 23:15:39 Corpus norm:0.9749 - (very good - balanced)
2012-09-10 23:15:39 Corpus confidence:  1.

Now I have:

2012-09-11 22:04:43 /usr/local/assp/store/spam
2012-09-11 22:04:43 File Count: 15,163
2012-09-11 22:04:43 Processing... store/spam with 14,000 files
2012-09-11 22:35:28 Imported Files: 14,001
2012-09-11 22:35:28 Folder contents exceeded 'MaxFiles'(14000).
2012-09-11 22:35:28 Finished in 1,845 second(s)
2012-09-11 22:35:28 info: require 6506 files from folder store/notspam to get a 
fine corpusnorm (1)

2012-09-11 22:35:28 /usr/local/assp/store/notspam
2012-09-11 22:35:28 File Count: 17,475
2012-09-11 22:35:28 Processing... store/notspam with 6,506 files
2012-09-11 22:47:45 remove/usr/local/assp/store/notspam/--522681.eml corrected 
spam
2012-09-11 22:48:45 Imported Files: 6,507
2012-09-11 22:48:45 Folder contents exceeded 'MaxFiles'(14000).
2012-09-11 22:48:45 Finished in 797 second(s)

All the best,
Colin Waring.

On 11/09/2012 20:49, Steve Moffat wrote:
> Hi
> I updated to the new release today and rebuildspamdb has ruined my corpus 
> confidence. Not too happy with that
>
> Sep-11-12 16:26:55 Spam Weight:3,904,196
> Sep-11-12 16:26:55 Not-Spam Weight:   1,950,092
>
> Sep-11-12 16:26:55 Corpus norm: 2.0021 - (warning: extremely spam 
> heavy)
> Sep-11-12 16:26:55 Corpus confidence:  0.06224349
> Sep-11-12 16:26:55 Recommendation: RebuildSpamDB will limit the number of 
> used messages in your corpus. Excess files will be ingored.
> Sep-11-12 16:26:55 Corpus norm should be between 0.6 and 1.4
>
> Thanks
> Steve
> Steve Moffat
> Operations Director
> Optimum IT Solutions
> Desk:   441 292 8849
> Mobile: 441 292 8849
> MSN IM:st...@optimum.bm
> Web:http://www.optimum.bm
>
> --
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats.http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] no sender email in log => ASSP 1.9.7.5(0.0.02)

[Graziano]

indeed X-Assp-Envelope-From: is empty

is it a good/bad idea to show in ASSP log the From: value in case of the 
envelope sender is not specified/empty ?

Graziano

> ASSP development mailing list  schrei
> bt:
>> From: "Email Marketing" 
>
>
> ASSP shows the envelope sender in the logfile not the "From:".
>
> --
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> ___
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

[Assp-test] Antwort: New version

[Thomas Eckardt]

Steve, please post the complete rebuild log.

Thomas



Von:Steve Moffat 
An: "'assp-test@lists.sourceforge.net'" 
, 
Datum:  11.09.2012 21:50
Betreff:[Assp-test] New version



Hi
I updated to the new release today and rebuildspamdb has ruined my corpus 
confidence. Not too happy with that

Sep-11-12 16:26:55 Spam Weight:3,904,196
Sep-11-12 16:26:55 Not-Spam Weight:   1,950,092

Sep-11-12 16:26:55 Corpus norm: 2.0021 - (warning: extremely 
spam heavy)
Sep-11-12 16:26:55 Corpus confidence:  0.06224349
Sep-11-12 16:26:55 Recommendation: RebuildSpamDB will limit the number of 
used messages in your corpus. Excess files will be ingored.
Sep-11-12 16:26:55 Corpus norm should be between 0.6 and 1.4

Thanks
Steve
Steve Moffat
Operations Director
Optimum IT Solutions
Desk:   441 292 8849
Mobile: 441 292 8849
MSN IM: st...@optimum.bm
Web: http://www.optimum.bm

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***


--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test