Good Morning Thomas, thanks for your reply. I am sure to have enough files in the corpus (see rebuildrun.txt below).
For testing purposes I set backupDBInterval to 0 and HMM remains enabled until a rebuildspamdb. Now I get "Spamdb has version: n/a - required version: 2_13218_5.014002_UAX#29_WordStem ! Run a rebuildspamdb to correct this!" ONLY on statuspage and "HMMdb has version: n/a - required version: 2_13218_5.014002_UAX#29_WordStem !" ONLY on the analyze page. Maillog after rebuildspamdb shows for example this: Okt-31-13 09:53:04 id-09582-00045 [Worker_1] 209.167.231.214 <[email protected]> to: [email protected] HMM is not available - hmmdb is empty Okt-31-13 09:53:04 id-09582-00045 [Worker_1] 209.167.231.214 <[email protected]> to: [email protected] info: Bayesian-Check has taken 0 seconds Okt-31-13 09:53:04 id-09582-00045 [Worker_1] 209.167.231.214 <[email protected]> to: [email protected] Bayesian Check - Prob: 1.00000 => spam Okt-31-13 09:53:04 id-09582-00045 [Worker_1] 209.167.231.214 <[email protected]> to: [email protected] Message-Score: added 49 for Bayesian Probability: 1.00000, total score for this message is now 44 Right BEFORE rebuildspamdb maillog shows this: Okt-31-13 09:34:30 id-08470-13116 [Worker_3] 176.9.212.115 <[email protected]> to: [email protected] info: HMM-Check has taken 0 seconds and has given 214 results Okt-31-13 09:34:30 id-08470-13116 [Worker_3] 176.9.212.115 <[email protected]> to: [email protected] HMM Check - Prob: 1.00000 => spam Okt-31-13 09:34:30 id-08470-13116 [Worker_3] 176.9.212.115 <[email protected]> to: [email protected] Message-Score: added 49 for HMM Probability: 1.0000, total score for this message is now 29 Okt-31-13 09:34:30 id-08470-13116 [Worker_3] 176.9.212.115 <[email protected]> to: [email protected] PB-IP-Score for '176.9.212.115' is 49, added 49 for HMM So I started a new rebuildspamdb as instructed. Now spamdb is okay but statuspage shows "HMMdb has version: - required version: 2_13218_5.014002_UAX#29_WordStem ! Run a rebuildspamdb to correct this!" and maillog still shows "HMMdb is empty". Filesize of /usr/share/assp/HMMdb.bdb is 173.035.520 bytes. Do I have a problem with BerkeleyDB, because of this and because the whitelist is empty too if configured to use DB:. Installed is: BerkeleyDB 0.51 BerkeleyDB_DBEngine 5.1 Thanks for your support in advance! Michael Okt-31-13 09:26:40 RebuildSpamDB-thread rebuildspamdb-version 6.30 started in ASSP version 2.3.4(13303) Okt-31-13 09:26:40 RebuildSpamDB will create a Hidden Markov Model! Okt-31-13 09:26:40 RebuildSpamDB will include attachment-database-entries in to spamdb! Okt-31-13 09:26:40 RebuildSpamDB will create unicode enabled databases. Okt-31-13 09:26:40 RebuildSpamDB will process all words as Sequence of UAX #29 Grapheme Clusters. Okt-31-13 09:26:40 RebuildSpamDB will use the ASSP_WordStem engine. Okt-31-13 09:26:40 ---ASSP Settings--- Okt-31-13 09:26:40 Do Not Collect RedRe Messages: Enabled **Messages matching the RedRe will be removed from the corpus!** Okt-31-13 09:26:40 Use Subject as Maillog Names: False Okt-31-13 09:26:40 Maxbytes: 10000 Okt-31-13 09:26:40 RebuildFileTimeLimit: 1 5 Okt-31-13 09:26:40 RebuildFileTimeLimit: files will be moved away from the corpus, if there processing takes longer than 5 second(s) Okt-31-13 09:26:40 Trashlist cleaning finished, 0 of 6 files deleted Okt-31-13 09:26:40 /usr/share/assp/errors/spam Okt-31-13 09:26:40 File Count: 2,549 Okt-31-13 09:26:40 Processing... errors/spam with 2,549 files Okt-31-13 09:26:40 ignore files older than Jun-18-12 10:26:40 in folder errors/spam Okt-31-13 09:28:14 Ignored: 1,022 Okt-31-13 09:28:14 0 attachment/image entries processed Okt-31-13 09:28:14 Imported Files for HeloBlackList: 1,526 Okt-31-13 09:28:14 Imported Files for Bayes/HMM: 2,548 Okt-31-13 09:28:14 Finished in 94 second(s) Okt-31-13 09:28:14 /usr/share/assp/errors/notspam Okt-31-13 09:28:14 File Count: 2,286 Okt-31-13 09:28:14 Processing... errors/notspam with 2,286 files Okt-31-13 09:28:14 ignore files older than Jun-18-12 10:28:14 in folder errors/notspam Okt-31-13 09:29:41 Ignored: 799 Okt-31-13 09:29:41 0 attachment/image entries processed Okt-31-13 09:29:41 Imported Files for HeloBlackList: 1,486 Okt-31-13 09:29:41 Imported Files for Bayes/HMM: 2,285 Okt-31-13 09:29:41 Finished in 87 second(s) Okt-31-13 09:29:41 info: corpusnorm after processing errors/spam and errors/notspam is Spam Weight: 1132318 / Not-Spam Weight: 2239052 => norm: 0.506 Okt-31-13 09:29:41 info: require apx. 13395 files (1353198 words) from folder spam to get the wanted corpusnorm (1.000) Okt-31-13 09:29:41 /usr/share/assp/spam Okt-31-13 09:29:41 File Count: 13,398 Okt-31-13 09:29:41 Processing... spam with 13,398 files Okt-31-13 09:29:41 ignore files older than Aug-02-13 10:29:41 in folder spam Okt-31-13 09:33:53 Ignored: 7,808 Okt-31-13 09:33:53 0 attachment/image entries processed Okt-31-13 09:33:53 Imported Files for HeloBlackList: 5,590 Okt-31-13 09:33:53 Imported Files for Bayes/HMM: 10,464 Okt-31-13 09:33:53 Finished in 252 second(s) Okt-31-13 09:33:53 info: require apx. 2625 files (246818 words) from folder notspam to get the wanted corpusnorm (1.000) Okt-31-13 09:33:53 /usr/share/assp/notspam Okt-31-13 09:33:53 File Count: 13,878 Okt-31-13 09:33:53 Processing... notspam with 13,878 files Okt-31-13 09:33:53 ignore files older than Aug-02-13 10:33:53 in folder notspam Okt-31-13 09:35:10 Ignored: 2,184 Okt-31-13 09:35:10 0 attachment/image entries processed Okt-31-13 09:35:10 Imported Files for HeloBlackList: 11,694 Okt-31-13 09:35:10 Imported Files for Bayes/HMM: 2,814 Okt-31-13 09:35:10 Finished in 77 second(s) Okt-31-13 09:35:10 Rebuild processed 39.80 files per second. Okt-31-13 09:35:10 Generating weighted Bayesian tuplets Okt-31-13 09:35:18 populating Spamdb 255601 records - Bayesian check is now disabled Okt-31-13 09:35:27 done - populating Spamdb records - 255601 - Bayesian check is now enabled Okt-31-13 09:35:27 done - Generating weighted Bayesian tuplets Okt-31-13 09:35:27 Bayesian Pairs: 255,601 now in list Okt-31-13 09:35:27 Generating consolidated Hidden-Markov-Model database from 2,093,274 record model Okt-31-13 09:36:17 HMM sequences: 1,002,045 now in list Zitat von Thomas Eckardt <[email protected]>: >> Okt-29-13 23:39:36 [Worker_10000] Backup: 3948 records of database > table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb > > 3948 records for HMMdb is simply much too less to get good HMM-results. > > Oct-31-13 04:00:11 Maxbytes: 8000 > > Oct-31-13 04:00:11 C:/assp/errors/spam > Oct-31-13 04:00:11 File Count: 1,765 > > Oct-31-13 04:06:20 C:/assp/errors/notspam > Oct-31-13 04:06:20 File Count: 847 > > Oct-31-13 04:09:15 C:/assp/spam > Oct-31-13 04:09:15 File Count: 1,959 > > Oct-31-13 04:13:21 C:/assp/notspam > Oct-31-13 04:13:21 File Count: 583 > > Oct-31-13 04:15:13 start populating Spamdb with 259,569 records - Bayesian > check is now disabled! > Oct-31-13 04:15:49 Finished populating Spamdb with 259,569 records - > Bayesian check is now enabled! > > Oct-31-13 04:15:50 Generating consolidated Hidden-Markov-Model database > from 1,748,314 record model > Oct-31-13 04:17:48 HMM sequences: 859,419 now in list > > You need more files in the corpus. > > Thomas. > > > > > Von: Michael <[email protected]> > An: assp-test <[email protected]>, > Datum: 30.10.2013 12:26 > Betreff: [Assp-test] HMMdb not working after backup > > > > Hello @all, > > I have noticed since several versions of ASSP that my HMMdb gave no > results AFTER a database backup and it is reproducable after a > logfile-check (see example below). > In addition to that my whitelist is always empty if I use DB. After > changing to file it works again as expected. > Rebuildspam does not show any errors. tmpDB is a directory on RAM-disk > with 5 GB of space, current usage: 501 MB. All rights are set to 777 > Does anybody have similar problems or tips where to start searching? > > Thanks you in advance! > > Michael > > My config for HMM and DB: > DoBayesian:=1 > DoHMM:=1 > ignoreDBVersionMissMatch:=0 > HMMusesBDB:= > DoPrivatSpamdb:=0 > BayesMaxProcessTime:=15 > BayesWL:= > BayesNP:= > noBayesian:= > noBayesian_local:= > Bayesian_localOnly:= > maxBayesValues:=60 > baysProbability:=0.6 > baysConf:=0 > baysConfidenceHalfScore:=1 > NoTagInTestmode:= > AddSpamProbHeader:=1 > AddConfidenceHeader:=1 > > DBdriver:=BerkeleyDB > > importDBDir:=mysql/dbbackup > preventBulkImport:= > fillUpImportDBDir:= > ImportMysqlDB:= > exportDBDir:=mysql/dbexport > ExportMysqlDB:= > backupDBDir:=mysql/dbbackup > backupDBInterval:=2 > > RebuildSchedule:=0 6 * * * > useDB4Rebuild:=1 > ReplaceOldSpamdb:= > doMove2Num:= > newReportedInterval:=10 5 > MaxKeepDeleted:=7 > autoCorrectCorpus:=0.6-1.4-4000-14 > RebuildFileTimeLimit:=1 5 > RebuildNotify:[email protected] > RebuildTestMode:= > > > > > Okt-29-13 19:59:00 [Main_Thread] MainThread started > Okt-29-13 19:59:00 [Main_Thread] Admin connection from user root on > host 192.169.210.6:47382; page:/analyze; > session-ID:c76031876186ed5f6464f72cb3e1a1c0; > Okt-29-13 19:59:01 [Main_Thread] Info: HMM-Check has taken 0 seconds > and has given 423 results > Okt-29-13 19:59:01 [Main_Thread] HMM Check - Prob: 1.00000 => spam > > > Okt-29-13 20:40:08 [Main_Thread] Info: HMM-Check has taken 0 seconds > and has given 268 results > Okt-29-13 20:40:08 [Main_Thread] HMM Check - Prob: 1.00000 => spam > > > Okt-29-13 21:39:18 [Worker_10000] Backup: starting backup database > table PersBlack to file /usr/share/assp/mysql/dbbackup/persblack > ... > Okt-29-13 21:39:26 [Worker_10000] Backup: starting backup database > table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb > Okt-29-13 21:39:26 [Worker_10000] Backup: 3948 records of database > table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb > > > Okt-29-13 21:45:34 id-79533-02470 [Worker_1] 212.227.17.22 > <[email protected]> to: [email protected] HMM-Check has given less than 21 results > - using soring mode only > Okt-29-13 21:45:34 id-79533-02470 [Worker_1] [HMM][scoring] > 212.227.17.22 <[email protected]> to: [email protected] HMM-Check has given less > than 6 results - using monitoring mode only > Okt-29-13 21:45:34 id-79533-02470 [Worker_1] [HMM][monitoring] > 212.227.17.22 <[email protected]> to: [email protected] info: HMM-Check has taken 0 > seconds and has given 0 results > > > Okt-29-13 22:04:15 id-80654-02390 [Worker_1] [HMM][scoring] > 199.187.172.79 <[email protected]> to: [email protected] HMM-Check has > given less than 6 results - using monitoring mode only > Okt-29-13 22:04:15 id-80654-02390 [Worker_1] [HMM][monitoring] > 199.187.172.79 <[email protected]> to: [email protected] info: HMM-Check > has taken 0 seconds and has given 0 results > Okt-29-13 22:04:15 id-80654-02390 [Worker_1] [HMM][monitoring] > 199.187.172.79 <[email protected]> to: [email protected] info: > Bayesian-Check has taken 0 seconds > > *** ASSP RESTART *** > > Okt-29-13 22:06:29 [Main_Thread] MainThread started > Okt-29-13 22:06:29 [Main_Thread] Admin connection from user root on > host 192.169.210.6:49984; page:/analyze; > session-ID:c76031876186ed5f6464f72cb3e1a1c0; > Okt-29-13 22:06:29 [Main_Thread] [scoring] SPF: none (cache) > ip=109.170.242.23 [email protected] > helo=mx-relay-03.vmx.sdc.uk.murphx.net > Okt-29-13 22:06:29 [Main_Thread] Info: HMM-Check has taken 0 seconds > and has given 266 results > Okt-29-13 22:06:29 [Main_Thread] HMM Check - Prob: 1.00000 => spam > > > Okt-29-13 22:57:51 id-83869-12808 [Worker_1] 103.14.120.29 > <[email protected]> to: [email protected] info: HMM-Check has taken 0 > seconds and has given 43 results > Okt-29-13 22:57:51 id-83869-12808 [Worker_1] 103.14.120.29 > <[email protected]> to: [email protected] HMM Check - Prob: 1.00000 => spam > Okt-29-13 22:57:51 id-83869-12808 [Worker_1] 103.14.120.29 > <[email protected]> to: [email protected] Message-Score: added 49 for HMM > Probability: 1.0000, total score for this message is now 64 > > Okt-29-13 23:37:16 id-86235-03435 [Worker_1] 74.114.9.196 > <[email protected]> to: [email protected] info: > HMM-Check has taken 0 seconds and has given 138 results > Okt-29-13 23:37:16 id-86235-03435 [Worker_1] 74.114.9.196 > <[email protected]> to: [email protected] HMM Check - > Prob: 1.00000 => spam > Okt-29-13 23:37:16 id-86235-03435 [Worker_1] 74.114.9.196 > <[email protected]> to: [email protected] Message-Score: > added 49 for HMM Probability: 1.0000, total score for this message is > now 69 > > > Okt-29-13 23:39:28 [Worker_10000] Backup: starting backup database > table PersBlack to file /usr/share/assp/mysql/dbbackup/persblack > ... > Okt-29-13 23:39:36 [Worker_10000] Backup: starting backup database > table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb > Okt-29-13 23:39:36 [Worker_10000] Backup: 3948 records of database > table HMMdb to file /usr/share/assp/mysql/dbbackup/HMMdb > > ------------------------------------------------------------------------------ > Android is increasing in popularity, but the open development platform > that > developers love is also attractive to malware creators. Download this > white > paper to learn more about secure code signing practices that can help keep > Android apps secure. > http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk > > _______________________________________________ > Assp-test mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > DISCLAIMER: > ******************************************************* > This email and any files transmitted with it may be confidential, legally > privileged and protected in law and are intended solely for the use of the > > individual to whom it is addressed. > This email was multiple times scanned for viruses. There should be no > known virus in this email! > ******************************************************* ------------------------------------------------------------------------------ Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk _______________________________________________ Assp-test mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/assp-test
