Sorry for top posting, but just to warn its quite a long post. You have been warned! =)
Sylvain Garcia wrote: > > > Hugo Monteiro wrote: >> Sylvain Garcia wrote: >>> Hi, I 'm working always on integration of dspam into OBM >>> (http://www.obm.org) >>> >>> I had install Dspam in my production environment to make some test >>> with OBM. And I have some problem, i don't understand. >>> >>> I use dspam git with debian package on kirya.net >>> (http://packages.kirya.net/dspam-community) >>> Dspam use postgresql 8.3 backend on debian Etch. >>> >>> dspam is configured with LMTP and postfix and also amavis ( with no >>> relation with dspam ): >>> >>> Postfix --LMTP--> DSPAM --SMTP--> Postfix --SMTP--> >>> Amavis(SA)--SMTP-->Postfix ---LMTP---Cyrus >>> >>> >>> I configured dspam to check user on ldap: >>> ExtLookup on >>> ExtLookupMode strict >>> ExtLookupDriver ldap >>> ExtLookupServer localhost >>> ExtLookupPort 389 >>> ExtLookupDB "dc=local" >>> ExtLookupQuery >>> "(&(|(objectclass=obmuser)(objectclass=obmMailShare))(mailBox=%u))" >>> ExtLookupLDAPAttribute "mailBox" >>> ExtLookupLDAPScope sub >>> ExtLookupLDAPVersion 3 >>> >>> >>> The problem is that i have some user which are create in dspam, but >>> they don't exist on ldap.... >>> >>> for exemple: >>> dspam_stat: >>> [email protected] TP: 3 TN: 2 FP: 0 FN: 0 SC: >>> 0 NC: 0 >>> >>> there is no user whith this mail adress in my ldap, But i have one >>> user which ave sieve script on cyrus which bounce mail at this adress, >>> and in log we can see: >>> "External lookup verification of [email protected] failed: not >>> adding user" >>> and: >>> SELECT * from dspam_virtual_uids where username = >>> '[email protected]'; return: >>> uid | username >>> -----+------------------------------- >>> 110 | [email protected] >>> >>> >>> [email protected] is an example on my dspam database but >>> i have 30 users which doesn't exist on my ldap, and i have 78 real >>> user on dspam. >> >> >> Hello Sylvain, >> >> First things first. Why would you have a user in your database with >> the domain gmail.com? Are you accepting mail for that domain? > It's my problem... > >> I believe you're using DSPAM in your outbound mail too. Are you sure >> you want to do this? > > Hum.. I go to check this, but if user is not present in ldap, dspam > shoud'nt create this in his database... Before, i was talking from the top of my head since it's been quite a while since i've messed with such a DSPAM install. I took some time to perform some tests and i came to the conclusion that when in strict mode, DSPAM will neither create the user entry in the DSPAM database, nor the respective user dir under ~dspam/data. Example follows: STRICT MODE ------------------------- dspam --stdout --client --feature=noise --deliver=innocent,spam --debug --user [email protected] < 1226447237.P1580Q0M565876.mail\,S\=2535\:2\,ST from the log file we see the following 4596: [04/17/2009 20:58:39] No QuarantineAgent option found. Using standard quarantine. 4596: [04/17/2009 20:58:39] looking up user [email protected] using ldap driver. 4596: [04/17/2009 20:58:39] External Lookup: found 0 LDAP entries 4596: [04/17/2009 20:58:39] External Lookup: Backend search failure: no entries found. 4596: [04/17/2009 20:58:39] DSPAM Instance Startup 4596: [04/17/2009 20:58:39] input args: dspam --stdout --client --feature=noise --deliver=innocent,spam --debug --user [email protected] 4596: [04/17/2009 20:58:39] pass-thru args: 4596: [04/17/2009 20:58:39] processing user [email protected] 4596: [04/17/2009 20:58:39] uid = 1010, euid = 1010, gid = 201, egid = 105 4596: [04/17/2009 20:58:39] loading preferences for user [email protected] 4596: [04/17/2009 20:58:39] _mysql_drv_getpwnam returning NULL for query on name: [email protected] 4596: [04/17/2009 20:58:39] _ds_pref_load: unable to _mysql_drv_getpwnam([email protected]) 4596: [04/17/2009 20:58:39] Loading preferences for uid 0 4596: [04/17/2009 20:58:39] Loading preferences for uid 0 4596: [04/17/2009 20:58:39] loaded default preferences externally 4596: [04/17/2009 20:58:39] using /var/spool/dspam/opt-in/ae.fct.unl.pt/nonexistent.dspam as path 4596: [04/17/2009 20:58:39] using /var/spool/dspam/opt-out/ae.fct.unl.pt/nonexistent.nodspam as path 4596: [04/17/2009 20:58:39] sedation level set to: 5 4596: [04/17/2009 20:58:39] _mysql_drv_getpwnam setting, then returning passed name: [email protected] 4596: [04/17/2009 20:58:39] verified_user is 0 4596: [04/17/2009 20:58:39] External lookup verification of [email protected] failed: not adding user 4596: [04/17/2009 20:58:39] _mysql_drv_get_spamtotals: unable to _mysql_drv_getpwnam([email protected]) 4596: [04/17/2009 20:58:39] unable to load totals. using zero values. 4596: [04/17/2009 20:58:39] _mysql_drv_getpwnam setting, then returning passed name: [email protected] 4596: [04/17/2009 20:58:39] verified_user is 0 4596: [04/17/2009 20:58:39] External lookup verification of [email protected] failed: not adding user 4596: [04/17/2009 20:58:39] _ds_getall_spamrecords: unable to _mysql_drv_getpwnam([email protected]) 4596: [04/17/2009 20:58:39] _ds_getall_spamrecords() failed 4596: [04/17/2009 20:58:39] total processing time: 0.00115s 4596: [04/17/2009 20:58:39] _mysql_drv_getpwnam setting, then returning passed name: [email protected] 4596: [04/17/2009 20:58:39] verified_user is 0 4596: [04/17/2009 20:58:39] External lookup verification of [email protected] failed: not adding user 4596: [04/17/2009 20:58:39] _mysql_drv_set_spamtotals: unable to _mysql_drv_getpwnam([email protected]) 4596: [04/17/2009 20:58:39] delivering message 4596: [04/17/2009 20:58:39] DSPAM Instance Shutdown. Exit Code: 0 stderr output was Apr 17 21:01:19 mail dspam[4734]: bailing on error -2 Apr 17 21:01:19 mail dspam[4734]: received invalid result (! DSR_ISSPAM || DSR_INNOCENT) : -2 Apr 17 21:01:19 mail dspam[4734]: process_message returned error -2. delivering. and it actually delivered the message but NO DSPAM headers were added. Also the dspam database remained the same mysql> select * from dspam_virtual_uids where username = '[email protected]'; Empty set (0.00 sec) dspam_stats shows dspam_stats [email protected] [email protected] TP: 0 TN: 0 FP: 0 FN: 0 SC: 0 NC: 0 MAP MODE -------------------- same command as above debug log shows 4840: [04/17/2009 21:04:12] No QuarantineAgent option found. Using standard quarantine. 4840: [04/17/2009 21:04:12] looking up user [email protected] using ldap driver. 4840: [04/17/2009 21:04:12] External Lookup: found 0 LDAP entries 4840: [04/17/2009 21:04:12] External Lookup: Backend search failure: no entries found. 4840: [04/17/2009 21:04:12] no match for user [email protected] but mode is map. continuing... 4840: [04/17/2009 21:04:12] DSPAM Instance Startup 4840: [04/17/2009 21:04:12] input args: dspam --stdout --client --feature=noise --deliver=innocent,spam --debug --user [email protected] 4840: [04/17/2009 21:04:12] pass-thru args: 4840: [04/17/2009 21:04:12] processing user [email protected] 4840: [04/17/2009 21:04:12] uid = 1010, euid = 1010, gid = 201, egid = 105 4840: [04/17/2009 21:04:12] loading preferences for user [email protected] 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam returning NULL for query on name: [email protected] 4840: [04/17/2009 21:04:12] _ds_pref_load: unable to _mysql_drv_getpwnam([email protected]) 4840: [04/17/2009 21:04:12] Loading preferences for uid 0 4840: [04/17/2009 21:04:12] Loading preferences for uid 0 4840: [04/17/2009 21:04:12] loaded default preferences externally 4840: [04/17/2009 21:04:12] using /var/spool/dspam/opt-in/ae.fct.unl.pt/nonexistent.dspam as path 4840: [04/17/2009 21:04:12] using /var/spool/dspam/opt-out/ae.fct.unl.pt/nonexistent.nodspam as path 4840: [04/17/2009 21:04:12] sedation level set to: 5 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam setting, then returning passed name: [email protected] 4840: [04/17/2009 21:04:12] verified_user is 1 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam successful; returning struct for name: [email protected] 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam returning cached name [email protected]. 4840: [04/17/2009 21:04:12] Loading 7 BNR patterns 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam returning cached name [email protected]. 4840: [04/17/2009 21:04:12] Whitelist threshold: 10 ... skip token debugging ... 4840: [04/17/2009 21:04:12] Graham-Bayesian Probability: 0.002278 Samples: 15 4840: [04/17/2009 21:04:12] Burton-Bayesian Probability: 0.000018 Samples: 27 4840: [04/17/2009 21:04:12] no factors specified; using default 4840: [04/17/2009 21:04:12] Result Confidence: 1.00 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam returning cached name [email protected]. 4840: [04/17/2009 21:04:12] Control: [10 10] [10 11] Delta: [0 1] 4840: [04/17/2009 21:04:12] total processing time: 0.09854s 4840: [04/17/2009 21:04:12] sending firstrun.txt to [email protected] (/var/spool/dspam/data/ae.fct.unl.pt/nonexistent/nonexistent.firstrun): No such file or directory 4840: [04/17/2009 21:04:12] creating directory '/var/spool/dspam/data/ae.fct.unl.pt/nonexistent' 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam returning cached name [email protected]. 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam returning cached name [email protected]. 4840: [04/17/2009 21:04:12] saving signature as 365,49e8e0bc48407822015704 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam returning cached name [email protected]. 4840: [04/17/2009 21:04:12] libdspam returned probability of 0.002278 4840: [04/17/2009 21:04:12] message result: NOT SPAM 4840: [04/17/2009 21:04:12] _mysql_drv_getpwnam returning cached name [email protected]. 4840: [04/17/2009 21:04:12] delivering message 4840: [04/17/2009 21:04:12] DSPAM Instance Shutdown. Exit Code: 0 in stderr output we get Apr 17 21:04:12 mail dspam[4840]: Unable to open file for reading: firstrun.txt: No such file or directory in this case we do get the directory created drwxrwx--- 2 hm dspam 4096 2009-04-17 21:04 /var/spool/dspam/data/ae.fct.unl.pt/nonexistent and we do get an entry in the database mysql> select * from dspam_virtual_uids where username = '[email protected]'; +-----+---------------------------+ | uid | username | +-----+---------------------------+ | 365 | [email protected] | +-----+---------------------------+ 1 row in set (0.00 sec) dspam_stats shows dspam_stats [email protected] [email protected] TP: 0 TN: 1 FP: 0 FN: 0 SC: 0 NC: 0 But you were right about dspam_stats. It does got get info from the database: mail:~# rm -rf /var/spool/dspam/data/ae.fct.unl.pt/nonexistent mail:~# dspam_stats [email protected] [email protected] TP: 0 TN: 1 FP: 0 FN: 0 SC: 0 NC: 0 mysql> delete from dspam_virtual_uids where username = '[email protected]'; Query OK, 1 row affected (0.00 sec) mysql> quit Bye mail:~# dspam_stats [email protected] [email protected] TP: 0 TN: 0 FP: 0 FN: 0 SC: 0 NC: 0 and at last VERIFY MODE ------------------------- debug log... 5230: [04/17/2009 21:13:23] No QuarantineAgent option found. Using standard quarantine. 5230: [04/17/2009 21:13:23] looking up user [email protected] using ldap driver. 5230: [04/17/2009 21:13:23] External Lookup: found 0 LDAP entries 5230: [04/17/2009 21:13:23] External Lookup: Backend search failure: no entries found. 5230: [04/17/2009 21:13:23] DSPAM Instance Startup 5230: [04/17/2009 21:13:23] input args: dspam --stdout --client --feature=noise --deliver=innocent,spam --debug --user [email protected] 5230: [04/17/2009 21:13:23] pass-thru args: 5230: [04/17/2009 21:13:23] processing user [email protected] 5230: [04/17/2009 21:13:23] uid = 1010, euid = 1010, gid = 201, egid = 105 5230: [04/17/2009 21:13:23] loading preferences for user [email protected] 5230: [04/17/2009 21:13:23] _mysql_drv_getpwnam returning NULL for query on name: [email protected] 5230: [04/17/2009 21:13:23] _ds_pref_load: unable to _mysql_drv_getpwnam([email protected]) 5230: [04/17/2009 21:13:23] Loading preferences for uid 0 5230: [04/17/2009 21:13:23] Loading preferences for uid 0 5230: [04/17/2009 21:13:23] loaded default preferences externally 5230: [04/17/2009 21:13:23] using /var/spool/dspam/opt-in/ae.fct.unl.pt/nonexistent.dspam as path 5230: [04/17/2009 21:13:23] using /var/spool/dspam/opt-out/ae.fct.unl.pt/nonexistent.nodspam as path 5230: [04/17/2009 21:13:23] sedation level set to: 5 5230: [04/17/2009 21:13:23] _mysql_drv_getpwnam setting, then returning passed name: [email protected] 5230: [04/17/2009 21:13:23] verified_user is 0 5230: [04/17/2009 21:13:23] External lookup verification of [email protected] failed: not adding user 5230: [04/17/2009 21:13:23] _mysql_drv_get_spamtotals: unable to _mysql_drv_getpwnam([email protected]) 5230: [04/17/2009 21:13:23] unable to load totals. using zero values. 5230: [04/17/2009 21:13:23] _mysql_drv_getpwnam setting, then returning passed name: [email protected] 5230: [04/17/2009 21:13:23] verified_user is 0 5230: [04/17/2009 21:13:23] External lookup verification of [email protected] failed: not adding user 5230: [04/17/2009 21:13:23] _ds_getall_spamrecords: unable to _mysql_drv_getpwnam([email protected]) 5230: [04/17/2009 21:13:23] _ds_getall_spamrecords() failed 5230: [04/17/2009 21:13:23] total processing time: 0.00121s 5230: [04/17/2009 21:13:23] _mysql_drv_getpwnam setting, then returning passed name: [email protected] 5230: [04/17/2009 21:13:23] verified_user is 0 5230: [04/17/2009 21:13:23] External lookup verification of [email protected] failed: not adding user 5230: [04/17/2009 21:13:23] _mysql_drv_set_spamtotals: unable to _mysql_drv_getpwnam([email protected]) 5230: [04/17/2009 21:13:23] delivering message 5230: [04/17/2009 21:13:23] DSPAM Instance Shutdown. Exit Code: 0 stderr output Apr 17 21:13:23 mail dspam[5230]: bailing on error -2 Apr 17 21:13:23 mail dspam[5230]: received invalid result (! DSR_ISSPAM || DSR_INNOCENT) : -2 Apr 17 21:13:23 mail dspam[5230]: process_message returned error -2. delivering. no directory and no database entry is created. dspam_stats shows dspam_stats [email protected] [email protected] TP: 0 TN: 0 FP: 0 FN: 0 SC: 0 NC: 0 >> >> Second, what you describe doesn't mean that it's not working as it >> should. Although the user doesn't exist in LDAP, DSPAM still >> processes the message. It just doesn't polute the dspam virtual user >> table, but it will still write statistical message processing >> information into its data directory (That's not handled in by the >> database yet). dspam_stats only prints out the users statistical data >> that resides in ~dspam/data. >> > I desagree with you rm -rf ~dspam/data.gmail.com/[email protected] && > dspam_stats [email protected] > return: > [email protected] TP: 36 TN: 21 FP: 0 FN: 0 SC: 0 > NC: 0 > > >> Also note that the select you perform isn't quite correct. The user >> that failed to verify is [email protected]. I doubt you'll get any >> entry if you do >> >> SELECT * from dspam_virtual_uids where username = '[email protected]'; > > But i have entry. it's my fault, i cur email adress. So if i do: > SELECT * from dspam_virtual_uids where username = '[email protected]'; > > I have: > uid | username > -----+------------------------------- > 110 | [email protected] > (1 ligne) > > > I don't understand... a bug on external lookup????? > To sum things up: map mode - if user doesn't exist in ldap, it creates all the stuff with the supplied username. If it exists in ldap, creates all the stuff with the translated username. verify mode - if user doesn't exist in ldap, it doesn't create a new one in the dspam database, or dspam data directory, and doesn't actually process the message. If it exists in ldap, creates all the stuff with the supplied username. strict mode - if user doesn't exist in ldap, it doesn't create a new one in the dspam database, or dspam data directory, and doesn't actually process the message. If it exists in ldap, creates all the stuff with the translated username. Basically map+verify. So i would say there is no bug in it and that it's behaving as expected. I've taken a peek at the source packages that you've used. Compile time options look just fine, but i haven't tested those packages myself. In your testing, make sure you're purging dspam user data like i did in my example. Regards, Hugo Monteiro. -- ci.fct.unl.pt:~# cat .signature Hugo Monteiro Email : [email protected] Telefone : +351 212948300 Ext.15307 Web : http://hmonteiro.net Centro de Informática Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa Quinta da Torre 2829-516 Caparica Portugal Telefone: +351 212948596 Fax: +351 212948548 www.ci.fct.unl.pt [email protected] ci.fct.unl.pt:~# _ ------------------------------------------------------------------------------ Stay on top of everything new and different, both inside and around Java (TM) technology - register by April 22, and save $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco. 300 plus technical and hands-on sessions. Register today. Use priority code J9JMT32. http://p.sf.net/sfu/p _______________________________________________ Dspam-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspam-devel
