Good Morning Thomas,

thanks for your reply.
I am sure to have enough files in the corpus (see rebuildrun.txt below).

For testing purposes I set backupDBInterval to 0 and HMM remains  
enabled until a rebuildspamdb.
Now I get "Spamdb has version: n/a - required version:  
2_13218_5.014002_UAX#29_WordStem ! Run a rebuildspamdb to correct  
this!" ONLY on statuspage and "HMMdb has version: n/a - required  
version: 2_13218_5.014002_UAX#29_WordStem !" ONLY on the analyze page.

Maillog after rebuildspamdb shows for example this:

Okt-31-13 09:53:04 id-09582-00045 [Worker_1] 209.167.231.214  
<[email protected]> to: [email protected] HMM is not available  
- hmmdb is empty
Okt-31-13 09:53:04 id-09582-00045 [Worker_1] 209.167.231.214  
<[email protected]> to: [email protected] info: Bayesian-Check  
has taken 0 seconds
Okt-31-13 09:53:04 id-09582-00045 [Worker_1] 209.167.231.214  
<[email protected]> to: [email protected] Bayesian Check  -  
Prob: 1.00000 => spam
Okt-31-13 09:53:04 id-09582-00045 [Worker_1] 209.167.231.214  
<[email protected]> to: [email protected] Message-Score: added  
49 for Bayesian Probability: 1.00000, total score for this message is  
now 44


Right BEFORE rebuildspamdb maillog shows this:

Okt-31-13 09:34:30 id-08470-13116 [Worker_3] 176.9.212.115  
<[email protected]> to:  
[email protected] info: HMM-Check has taken 0 seconds and has given 214  
results
Okt-31-13 09:34:30 id-08470-13116 [Worker_3] 176.9.212.115  
<[email protected]> to:  
[email protected] HMM Check  - Prob: 1.00000 => spam
Okt-31-13 09:34:30 id-08470-13116 [Worker_3] 176.9.212.115  
<[email protected]> to:  
[email protected] Message-Score: added 49 for HMM Probability: 1.0000, total  
score for this message is now 29
Okt-31-13 09:34:30 id-08470-13116 [Worker_3] 176.9.212.115  
<[email protected]> to:  
[email protected] PB-IP-Score for '176.9.212.115' is 49, added 49 for HMM


So I started a new rebuildspamdb as instructed. Now spamdb is okay but  
statuspage shows "HMMdb has version: - required version:  
2_13218_5.014002_UAX#29_WordStem ! Run a rebuildspamdb to correct  
this!" and maillog still shows "HMMdb is empty". Filesize of  
/usr/share/assp/HMMdb.bdb is 173.035.520 bytes.
Do I have a problem with BerkeleyDB, because of this and because the  
whitelist is empty too if configured to use DB:.
Installed is:
BerkeleyDB             0.51
BerkeleyDB_DBEngine    5.1

Thanks for your support in advance!

Michael


Okt-31-13 09:26:40 RebuildSpamDB-thread rebuildspamdb-version 6.30  
started in ASSP version 2.3.4(13303)
Okt-31-13 09:26:40 RebuildSpamDB will create a Hidden Markov Model!
Okt-31-13 09:26:40 RebuildSpamDB will include  
attachment-database-entries in to spamdb!
Okt-31-13 09:26:40 RebuildSpamDB will create unicode enabled databases.
Okt-31-13 09:26:40 RebuildSpamDB will process all words as Sequence of  
UAX #29 Grapheme Clusters.
Okt-31-13 09:26:40 RebuildSpamDB will use the ASSP_WordStem engine.

Okt-31-13 09:26:40 ---ASSP Settings---
Okt-31-13 09:26:40 Do Not Collect RedRe Messages: Enabled **Messages  
matching the RedRe will be removed from the corpus!**

Okt-31-13 09:26:40 Use Subject as Maillog Names: False
Okt-31-13 09:26:40 Maxbytes: 10000
Okt-31-13 09:26:40 RebuildFileTimeLimit: 1 5
Okt-31-13 09:26:40 RebuildFileTimeLimit: files will be moved away from  
the corpus, if there processing takes longer than 5 second(s)

Okt-31-13 09:26:40 Trashlist cleaning finished, 0 of 6 files deleted

Okt-31-13 09:26:40 /usr/share/assp/errors/spam
Okt-31-13 09:26:40 File Count:  2,549
Okt-31-13 09:26:40 Processing... errors/spam with 2,549 files
Okt-31-13 09:26:40 ignore files older than Jun-18-12 10:26:40 in  
folder errors/spam
Okt-31-13 09:28:14 Ignored:     1,022
Okt-31-13 09:28:14 0 attachment/image entries processed
Okt-31-13 09:28:14 Imported Files for HeloBlackList:    1,526
Okt-31-13 09:28:14 Imported Files for Bayes/HMM:        2,548
Okt-31-13 09:28:14 Finished in 94 second(s)

Okt-31-13 09:28:14 /usr/share/assp/errors/notspam
Okt-31-13 09:28:14 File Count:  2,286
Okt-31-13 09:28:14 Processing... errors/notspam with 2,286 files
Okt-31-13 09:28:14 ignore files older than Jun-18-12 10:28:14 in  
folder errors/notspam
Okt-31-13 09:29:41 Ignored:     799
Okt-31-13 09:29:41 0 attachment/image entries processed
Okt-31-13 09:29:41 Imported Files for HeloBlackList:    1,486
Okt-31-13 09:29:41 Imported Files for Bayes/HMM:        2,285
Okt-31-13 09:29:41 Finished in 87 second(s)
Okt-31-13 09:29:41 info: corpusnorm after processing errors/spam and  
errors/notspam is Spam Weight: 1132318 / Not-Spam Weight: 2239052 =>  
norm: 0.506
Okt-31-13 09:29:41 info: require apx. 13395 files (1353198 words) from  
folder spam to get the wanted corpusnorm (1.000)

Okt-31-13 09:29:41 /usr/share/assp/spam
Okt-31-13 09:29:41 File Count:  13,398
Okt-31-13 09:29:41 Processing... spam with 13,398 files
Okt-31-13 09:29:41 ignore files older than Aug-02-13 10:29:41 in folder spam
Okt-31-13 09:33:53 Ignored:     7,808
Okt-31-13 09:33:53 0 attachment/image entries processed
Okt-31-13 09:33:53 Imported Files for HeloBlackList:    5,590
Okt-31-13 09:33:53 Imported Files for Bayes/HMM:        10,464
Okt-31-13 09:33:53 Finished in 252 second(s)
Okt-31-13 09:33:53 info: require apx. 2625 files (246818 words) from  
folder notspam to get the wanted corpusnorm (1.000)

Okt-31-13 09:33:53 /usr/share/assp/notspam
Okt-31-13 09:33:53 File Count:  13,878
Okt-31-13 09:33:53 Processing... notspam with 13,878 files
Okt-31-13 09:33:53 ignore files older than Aug-02-13 10:33:53 in  
folder notspam
Okt-31-13 09:35:10 Ignored:     2,184
Okt-31-13 09:35:10 0 attachment/image entries processed
Okt-31-13 09:35:10 Imported Files for HeloBlackList:    11,694
Okt-31-13 09:35:10 Imported Files for Bayes/HMM:        2,814
Okt-31-13 09:35:10 Finished in 77 second(s)

Okt-31-13 09:35:10 Rebuild processed 39.80 files per second.

Okt-31-13 09:35:10 Generating weighted Bayesian tuplets
Okt-31-13 09:35:18 populating Spamdb 255601 records - Bayesian check  
is now disabled
Okt-31-13 09:35:27 done - populating Spamdb records - 255601 -  
Bayesian check is now enabled
Okt-31-13 09:35:27 done - Generating weighted Bayesian tuplets

Okt-31-13 09:35:27 Bayesian Pairs: 255,601 now in list

Okt-31-13 09:35:27 Generating consolidated Hidden-Markov-Model  
database from 2,093,274 record model
Okt-31-13 09:36:17 HMM sequences: 1,002,045 now in list





Zitat von Thomas Eckardt <[email protected]>:

>> Okt-29-13 23:39:36 [Worker_10000] Backup: 3948 records of database
> table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb
>
> 3948 records for HMMdb is simply much too less to get good HMM-results.
>
> Oct-31-13 04:00:11 Maxbytes: 8000
>
> Oct-31-13 04:00:11 C:/assp/errors/spam
> Oct-31-13 04:00:11 File Count:           1,765
>
> Oct-31-13 04:06:20 C:/assp/errors/notspam
> Oct-31-13 04:06:20 File Count:           847
>
> Oct-31-13 04:09:15 C:/assp/spam
> Oct-31-13 04:09:15 File Count:           1,959
>
> Oct-31-13 04:13:21 C:/assp/notspam
> Oct-31-13 04:13:21 File Count:           583
>
> Oct-31-13 04:15:13 start populating Spamdb with 259,569 records - Bayesian
> check is now disabled!
> Oct-31-13 04:15:49 Finished populating Spamdb with 259,569 records -
> Bayesian check is now enabled!
>
> Oct-31-13 04:15:50 Generating consolidated Hidden-Markov-Model database
> from 1,748,314 record model
> Oct-31-13 04:17:48 HMM sequences: 859,419 now in list
>
> You need more files in the corpus.
>
> Thomas.
>
>
>
>
> Von:    Michael <[email protected]>
> An:     assp-test <[email protected]>,
> Datum:  30.10.2013 12:26
> Betreff:        [Assp-test] HMMdb not working after backup
>
>
>
> Hello @all,
>
> I have noticed since several versions of ASSP that my HMMdb gave no
> results AFTER a database backup and it is reproducable after a
> logfile-check (see example below).
> In addition to that my whitelist is always empty if I use DB. After
> changing to file it works again as expected.
> Rebuildspam does not show any errors. tmpDB is a directory on RAM-disk
> with 5 GB of space, current usage: 501 MB. All rights are set to 777
> Does anybody have similar problems or tips where to start searching?
>
> Thanks you in advance!
>
> Michael
>
> My config for HMM and DB:
> DoBayesian:=1
> DoHMM:=1
> ignoreDBVersionMissMatch:=0
> HMMusesBDB:=
> DoPrivatSpamdb:=0
> BayesMaxProcessTime:=15
> BayesWL:=
> BayesNP:=
> noBayesian:=
> noBayesian_local:=
> Bayesian_localOnly:=
> maxBayesValues:=60
> baysProbability:=0.6
> baysConf:=0
> baysConfidenceHalfScore:=1
> NoTagInTestmode:=
> AddSpamProbHeader:=1
> AddConfidenceHeader:=1
>
> DBdriver:=BerkeleyDB
>
> importDBDir:=mysql/dbbackup
> preventBulkImport:=
> fillUpImportDBDir:=
> ImportMysqlDB:=
> exportDBDir:=mysql/dbexport
> ExportMysqlDB:=
> backupDBDir:=mysql/dbbackup
> backupDBInterval:=2
>
> RebuildSchedule:=0 6 * * *
> useDB4Rebuild:=1
> ReplaceOldSpamdb:=
> doMove2Num:=
> newReportedInterval:=10 5
> MaxKeepDeleted:=7
> autoCorrectCorpus:=0.6-1.4-4000-14
> RebuildFileTimeLimit:=1 5
> RebuildNotify:[email protected]
> RebuildTestMode:=
>
>
>
>
> Okt-29-13 19:59:00 [Main_Thread] MainThread started
> Okt-29-13 19:59:00 [Main_Thread] Admin connection from user root on
> host 192.169.210.6:47382; page:/analyze;
> session-ID:c76031876186ed5f6464f72cb3e1a1c0;
> Okt-29-13 19:59:01 [Main_Thread] Info: HMM-Check has taken 0 seconds
> and has given 423 results
> Okt-29-13 19:59:01 [Main_Thread] HMM Check  - Prob: 1.00000 => spam
>
>
> Okt-29-13 20:40:08 [Main_Thread] Info: HMM-Check has taken 0 seconds
> and has given 268 results
> Okt-29-13 20:40:08 [Main_Thread] HMM Check  - Prob: 1.00000 => spam
>
>
> Okt-29-13 21:39:18 [Worker_10000] Backup: starting backup database
> table PersBlack to file /usr/share/assp/mysql/dbbackup/persblack
> ...
> Okt-29-13 21:39:26 [Worker_10000] Backup: starting backup database
> table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb
> Okt-29-13 21:39:26 [Worker_10000] Backup: 3948 records of database
> table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb
>
>
> Okt-29-13 21:45:34 id-79533-02470 [Worker_1] 212.227.17.22
> <[email protected]> to: [email protected] HMM-Check has given less than 21 results
> - using soring mode only
> Okt-29-13 21:45:34 id-79533-02470 [Worker_1] [HMM][scoring]
> 212.227.17.22 <[email protected]> to: [email protected] HMM-Check has given less
> than 6 results - using monitoring mode only
> Okt-29-13 21:45:34 id-79533-02470 [Worker_1] [HMM][monitoring]
> 212.227.17.22 <[email protected]> to: [email protected] info: HMM-Check has taken 0
> seconds and has given 0 results
>
>
> Okt-29-13 22:04:15 id-80654-02390 [Worker_1] [HMM][scoring]
> 199.187.172.79 <[email protected]> to: [email protected] HMM-Check has
> given less than 6 results - using monitoring mode only
> Okt-29-13 22:04:15 id-80654-02390 [Worker_1] [HMM][monitoring]
> 199.187.172.79 <[email protected]> to: [email protected] info: HMM-Check
> has taken 0 seconds and has given 0 results
> Okt-29-13 22:04:15 id-80654-02390 [Worker_1] [HMM][monitoring]
> 199.187.172.79 <[email protected]> to: [email protected] info:
> Bayesian-Check has taken 0 seconds
>
> *** ASSP RESTART ***
>
> Okt-29-13 22:06:29 [Main_Thread] MainThread started
> Okt-29-13 22:06:29 [Main_Thread] Admin connection from user root on
> host 192.169.210.6:49984; page:/analyze;
> session-ID:c76031876186ed5f6464f72cb3e1a1c0;
> Okt-29-13 22:06:29 [Main_Thread] [scoring] SPF: none (cache)
> ip=109.170.242.23 [email protected]
> helo=mx-relay-03.vmx.sdc.uk.murphx.net
> Okt-29-13 22:06:29 [Main_Thread] Info: HMM-Check has taken 0 seconds
> and has given 266 results
> Okt-29-13 22:06:29 [Main_Thread] HMM Check  - Prob: 1.00000 => spam
>
>
> Okt-29-13 22:57:51 id-83869-12808 [Worker_1] 103.14.120.29
> <[email protected]> to: [email protected] info: HMM-Check has taken 0
> seconds and has given 43 results
> Okt-29-13 22:57:51 id-83869-12808 [Worker_1] 103.14.120.29
> <[email protected]> to: [email protected] HMM Check  - Prob: 1.00000 => spam
> Okt-29-13 22:57:51 id-83869-12808 [Worker_1] 103.14.120.29
> <[email protected]> to: [email protected] Message-Score: added 49 for HMM
> Probability: 1.0000, total score for this message is now 64
>
> Okt-29-13 23:37:16 id-86235-03435 [Worker_1] 74.114.9.196
> <[email protected]> to: [email protected] info:
> HMM-Check has taken 0 seconds and has given 138 results
> Okt-29-13 23:37:16 id-86235-03435 [Worker_1] 74.114.9.196
> <[email protected]> to: [email protected] HMM Check  -
> Prob: 1.00000 => spam
> Okt-29-13 23:37:16 id-86235-03435 [Worker_1] 74.114.9.196
> <[email protected]> to: [email protected] Message-Score:
> added 49 for HMM Probability: 1.0000, total score for this message is
> now 69
>
>
> Okt-29-13 23:39:28 [Worker_10000] Backup: starting backup database
> table PersBlack to file /usr/share/assp/mysql/dbbackup/persblack
> ...
> Okt-29-13 23:39:36 [Worker_10000] Backup: starting backup database
> table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb
> Okt-29-13 23:39:36 [Worker_10000] Backup: 3948 records of database
> table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform
> that
> developers love is also attractive to malware creators. Download this
> white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
>
> _______________________________________________
> Assp-test mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************



------------------------------------------------------------------------------
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
_______________________________________________
Assp-test mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/assp-test

Reply via email to