On Thu, 17 Dec 2009 12:13:57 +0200
Ibrahim Harrani <[email protected]> wrote:
> Hi Stevan,
>
Hallo Ibrahim,
>
> >
> > And how happy are you with 3.9.0? Any feedback you can give us already? Do
> > you have made a package for FreeBSD? Can you share it with us?
> >
>
> It is definitely faster than 3.6.X with pgsql driver :)
>
Cool! I am happy. The speed benefit is mainly comming from the fact that I
rewrote the MySQL driver to be more faster and then later made +/- the same
changes to PostgreSQL as I did for MySQL. It's great that those changes have
such a noticable impact on the PostgreSQL driver. I still think that there is
more room for improvements in the PostgreSQL driver. After 3.9.0 I will take
the time to get more familiar with the PostgreSQL driver and make additional
improvements there.
> I just compiled
> from source and did not have time for freebsd ports.
>
> I have an error in dspam.log file
> #tail /var/log/dspam/dspam.log
> 60222: [12/17/2009 11:48:36] query error: %s: see sql.errors for more
> details
> 60222: [12/17/2009 11:48:36] query error: %s: see sql.errors for more
> details
> 60222: [12/17/2009 11:48:36] query error: %s: see sql.errors for more
> details
> 60222: [12/17/2009 11:48:36] query error: %s: see sql.errors for more
> details
>
> But I don't see any error in sql.errors. I double checked the log file.
> There is only verbose debug messages not errors like following
>
>
> [12/17/2009 11:48:25] 60222: VERBOSE DEBUG (INFO ONLY - NOT AN ERROR):
> SELECT * FROM lookup_tokens(1001,'{5334511981350707200}')
> [12/17/2009 11:48:25] 60222: VERBOSE DEBUG (INFO ONLY - NOT AN ERROR):
> SELECT * FROM
> lookup_tokens(1001,'{4076021200471482585,4869390869022420400,7676102314961599576,-5313266168790565737,-2623254645427778699,8797909246178839117,4865690555037777920,7681988435473998813,-6071744166436269812,9136583896346990292,6899567405689733120,-2545590970016373126,-7459861531341550285,5456758313658129513,3933609856790568557,-1326816071814857433,5158320329376923648,-8057245368052784757,-2747688497916075701,2403076604502008174,63182620316530877,-8546665041429048915,5844535569253380096,
>
I think the error is comming because how the PostgreSQL driver is working in
DSPAM. Read this here ->
http://sourceforge.net/tracker/?func=detail&aid=2881853&group_id=250683&atid=1126467
<-
I should fix that and capture those warnings. I just had not the time to do so.
> When I train dspam, I can see that numbef or SC value increases as expected!
> Is the log message fake?
>
No. The log messag is not fake. Inside the PostgreSQL driver we caputre
anything that does not return PGRES_TUPLES_OK or in some cases PGRES_COMMAND_OK
and if that is the case then we log that condition. I should however capture
PGRES_NONFATAL_ERROR in _ds_setall_spamrecords() and in some other functions
and not log errors for non fatal conditions. I could do that right now but I
think that will not make it into 3.9.0 final. I need to have a quick chit-chat
with the other admins from the DSPAM project and together we will make the
decision if such a change will flow back into 3.9.0 final or if we wait till
3.9.1?
Give me some time. I will today quickly look at the PostgreSQL driver and try
to fix that. Will post a patch here and if you give me the okay then I will
push it to GIT master and talk with the other admins if we can/should add such
a change to 3.9.0 final or if we wait till 3.9.1.
> # dspam_stats -H
> myglobaluser:
> TP True Positives: 5
> TN True Negatives: 0
> FP False Positives: 0
> FN False Negatives: 0
> SC Spam Corpusfed: 14283
> NC Nonspam Corpusfed: 5034
> TL Training Left: 0
> SHR Spam Hit Rate 100.00%
> HSR Ham Strike Rate: 100.00%
> PPV Positive predictive value: 100.00%
> OCA Overall Accuracy: 100.00%
>
> my configuration parameters:
>
> ./configure --sysconfdir=/usr/local/etc --with-logdir=/var/log/dspam
> --with-dspam-home=/var/db/dspam --with-dspam-home-owner=root
> --with-dspam-home-group=mail --with-dspam-home-mode=0770
> --with-dspam-owner=root --with-dspam-group=mail --disable-syslog
> --with-logfile=/var/log/dspam/dspam.log --enable-debug
> --enable-verbose-debug --enable-bnr-debug --enable-daemon
> --with-pgsql-includes=/usr/local/include
> --with-pgsql-libraries=/usr/local/lib
> --with-sqlite-includes=/usr/local/include
> --with-sqlite-libraries=/usr/local/lib
> --with-mysql-includes=/usr/local/include/mysql
> --with-mysql-libraries=/usr/local/lib/mysql
> --with-storage-driver=pgsql_drv,sqlite3_drv,hash_drv,mysql_drv --enable-ldap
>
Drop that "--enable-ldap" part and add "--enable-external-lookup" since we have
replaced LDAP driver with the new External Lookup module.
> --disable-trusted-user-security --with-dspam-mode=4510 --prefix=/usr/local
> --mandir=/usr/local/man --infodir=/usr/local/info/
> --build=amd64-portbld-freebsd6.3
>
>
> Second issue. If I started dspam from freebsd ports startup script (I just
> copied startup script from old dspam port)
>
> # /usr/local/etc/rc.d/dspam-devel start
> ps output:
> root 61919 0.0 0.0 16528 2964 p7 S 12:03PM 0:00.01
> /usr/local/bin/dspam --daemon --debug
>
> I only see the following debug mesages in /var/log/dspam.debug
>
> [root /var/log/dspam]# more dspam.debug
> 60222: [12/17/2009 11:47:44] attribute StorageDriver =
> /usr/local/lib/dspam/libpgsql_drv.so
> 60222: [12/17/2009 11:47:44] attribute PgSQLServer = /tmp
> 60222: [12/17/2009 11:47:44] attribute PgSQLUser = dspam
> 60222: [12/17/2009 11:47:44] attribute PgSQLPass = changeme
> 60222: [12/17/2009 11:47:44] attribute PgSQLDb = dspam
> 60222: [12/17/2009 11:47:44] attribute PgSQLConnectionCache = 10
> 60222: [12/17/2009 11:47:44] attribute HashRecMax = 98317
> 60222: [12/17/2009 11:47:44] attribute HashAutoExtend = on
> 60222: [12/17/2009 11:47:44] attribute HashMaxExtents = 0
> 60222: [12/17/2009 11:47:44] attribute HashExtentSize = 49157
> 60222: [12/17/20
>
> But I started dspam (/usr/local/bin/dspam --debug --daemon) manually, I can
> see the following messages.
> 61760: [12/17/2009 12:01:07] Graham-Bayesian Probability: 1.000000 Samples:
> 15
> 61760: [12/17/2009 12:01:07] Burton-Bayesian Probability: 1.000000 Samples:
> 27
> 61760: [12/17/2009 12:01:07] MISS: Chi-Square
> 61760: [12/17/2009 12:01:07] Result Confidence: 1.00
> 61760: [12/17/2009 12:01:07] BNR Decision Concurs
> 61760: [12/17/2009 12:01:07] Control: [10 10] [11 10] Delta: [1 0]
> 61760: [12/17/2009 12:01:07] libdspam returned probability of 1.000000
> 61760: [12/17/2009 12:01:07] message result: SPAM
> 61760: [12/17/2009 12:01:07] appending header X-DSPAM-Result: Spam
> 61760: [12/17/2009 12:01:07] appending header X-DSPAM-Confidence: 0.9998
> 61760: [12/17/2009 12:01:07] appending header X-DSPAM-Probability: 1.0000
> 61760: [12/17/2009 12:01:07] assembling component 0
> 61760: [12/17/2009 12:01:07] assembling component 1
> 61760: [12/17/2009 12:01:07] assembling component 2
> 61760: [12/17/2009 12:01:07] assembling component 3
> 61760: [12/17/2009 12:01:07] DSPAM Instance Shutdown. Exit Code: 0
> 61760: [12/17/2009 12:01:07] SEND: 250 2.6.0 <surgate> Message accepted for
> delivery: SPAM
>
> Both of them are same binary and using same config file. I know that this is
> stupid situtation and it should never be happen! any idea?
>
I have to look at that. I am right now not sure if disabling syslog is
affecting the logging. Give me time to check that out.
> > --
> > Kind Regards from Switzerland,
> >
> > Stevan Bajić
> >
> >
> > ------------------------------------------------------------------------------
> > This SF.Net email is sponsored by the Verizon Developer Community
> > Take advantage of Verizon's best-in-class app development support
> > A streamlined, 14 day to market process makes app distribution fast and
> > easy
> > Join now and get one step closer to millions of Verizon customers
> > http://p.sf.net/sfu/verizon-dev2dev
> > _______________________________________________
> > Dspam-user mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/dspam-user
> >
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
Dspam-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspam-user
