Re: [policyd-users] a point of clarification

[Cami Sardinha]

Joe Sloan wrote:
> [bump]
> 
> no one has any idea?
> 
> Joe
> 
> Sloan wrote:
>> One thing that is not clear to me despite reading the docs, is this:
>>
>> When a domain or address is "whitelisted", is it exempted from all 
>> tests, or from greylisting only?

In Policyd v1, all tests.

Cami

-
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Policyd and Qmail/Qmail-LDAP

[Cami Sardinha]

Hugo Monteiro wrote:
> 
> I'm sorry if this is going to sound a bit strange, since Policyd is 
> announced as an "an anti-spam plugin for Postfix".
> 
> I have put together a small howto (two actually) on how to set up Qmail 
> and Qmail-LDAP with Policyd and take advantage of all the good stuff 
> Policyd provides.
> 
> Those can be reached here:
> 
> http://hmonteiro.net/howtos:qmail:qmail_envelope_scanning
> 
> and
> 
> http://hmonteiro.net/howtos:qmail-ldap:qmail-ldap_envelope_scanning
> 
> respectively.

Looks like we'll have to state "Postfix & Qmail" ;)
Just wow. Very nice work, much appreciated!

Cami

-
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd running through mail function in php

[Cami Sardinha]

Jordi Moles wrote:
> Hi,
> 
> i've been trying to set up policyd in my system for weeks, but i can't 
> get it working as i want.
> I'm using postfix and freebsd in a local machine. Postfix has both uses: 
> incoming and outgoing mails.
> The thing is that i've set up policyd to work on mysql and if i use this 
> server as "smtp server" from thunderbird, policyd logs its activities in 
> the maillog file and actually works as i want. However, if i run, for 
> example, a perl or php script which calls function mail(), policyd just 
> doesn't do anything, it doesn't even write anything in the logs.

This is because mail() doesn't connect to Postfix on port 25.

> Is that a normal behaviour? can i do anything at all to make postfix use 
> policyd when i run a script from the command line interface?

Unfortunately there isn't and its a Postfix limitation.

> The thing is that when i run a script, postfix logs that the email has 
> been sent but policyd doesn't appear anywhere. The lines written in the 
> log file in this case are exactly the same as the ones when i use the 
> stmp server from thunderbird, but it's just that the policyd lines are 
> omitted.

This is because its sent via /usr/sbin/sendmail so bypasses all
the restrictions.

Cami

-
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] two instances of policyd: greylisting and throttling

[Cami Sardinha]

Nigel Kukard wrote:
>> Bleh. This is what the OP's problem is. He was doing
>> recipient throttling at the end_of_data phase. I'm
>> simply stating that it won't work regardless of what
>> version of Policyd is being used as Postfix does not
>> provide that information to Policyd at that stage.
> 
> Correct, if policyd (no matter what version) is setup only in the
> end_of_data restriction, its not going to work.
> 
>> If a message has multiple recipients and someone has
>> Policyd setup to do recipient throttling, its not
>> going to work as its suppose to.
> 
> Agreed ... v2 however understands states and restrictions, it can be
> added to both recipient & end_of_data to track exact counts for all
> recipients. v1 however doesn't have this functionality.
> 
> We on the same page?  ;)

Indeed.

-
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] two instances of policyd: greylisting and throttling

[Cami Sardinha]

Nigel Kukard wrote:
> v2 overcomes this and tracks the recipients for each smtpd instance. To
> track quotas precisely v2 can be used in smtpd_recipient_restrictions
> and smtpd_end_of_data_restrictions. v2 knows about the various
> restrictions.
 False. You will not be able to use Recipient throttling at
 the smtpd_end_of_data_restrictions. This has nothing to do
 with Policyd but an is Postfix limitation.
>>> I suggest you review the v2 code before making an assumption like that,
>>> furthermore you did not read what I said.
>>>
>>> V2 tracks all the recipients for the message. Therefore in end_of_data
>>> state, this information is available.
>> The information is not available in end_of_data.
> 
> "V2 tracks all the recipients for the message.". This is done in the
> recipient_restrictions and referenced using the Postfix instance.
> 
> Look at table   session_tracking  , there is a column RecipientData
> which stores matched policies and the recipients for the current smtp
> transaction (instance). The unique instance variable is available in
> end_of_data, which then does a query against the DB to get the list of
> recipients.
> 
> So it is available to policyd v2 in end_of_data (I made no mention of
> Postfix providing a recipient list in end_of_data).

Bleh. This is what the OP's problem is. He was doing
recipient throttling at the end_of_data phase. I'm
simply stating that it won't work regardless of what
version of Policyd is being used as Postfix does not
provide that information to Policyd at that stage.

If a message has multiple recipients and someone has
Policyd setup to do recipient throttling, its not
going to work as its suppose to.

-
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] two instances of policyd: greylisting and throttling

[Cami Sardinha]

Nigel Kukard wrote:
>>> v2 overcomes this and tracks the recipients for each smtpd instance. To
>>> track quotas precisely v2 can be used in smtpd_recipient_restrictions
>>> and smtpd_end_of_data_restrictions. v2 knows about the various
>>> restrictions.
>> False. You will not be able to use Recipient throttling at
>> the smtpd_end_of_data_restrictions. This has nothing to do
>> with Policyd but an is Postfix limitation.
> 
> I suggest you review the v2 code before making an assumption like that,
> furthermore you did not read what I said.
> 
> V2 tracks all the recipients for the message. Therefore in end_of_data
> state, this information is available.

The information is not available in end_of_data.

Cami

-
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] spamtrap account

[Cami Sardinha]

[EMAIL PROTECTED] wrote:
> Question
> I set up a "spamtrap" address inside Policyd  about 2 months ago...
> 
>It is my understanding that the 'spamtrap' email address
>after it has been set up does **NOT** receive any futher emails
> 
>there are two(2) spams in that emailaddress INBOX  this morning
> 
> 
>1) my current theory is that they sent in a "double" email  (ie  
> two(2) spams inline as one(1) email)
>[ because the local transfer header is missing in these 
> spamtrap emails ]
> 
>2) my system is compromised
> 
> I just would like strong a confirmation that "spamtrap" email address's 
> receive no new email!!

Spamtrap addresses should not receive mail normally.

If you post the headers of the message that slipped
through, it will allow some further insight/explanation.

Cami

-
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] two instances of policyd: greylisting and throttling

[Cami Sardinha]

Nigel Kukard wrote:
>>> Recipient Throttling can not work at smtpd_end_of_data_restrictions.
>>> What happens when 1 message has multiple recipients?
>>>
>>> Cami
>>>   
>> Yes, the problem arises as soon as there is a message with more than one 
>> recipient. As far as I know, Postfix does not pass multiple recipient 
>> addresses to policyd when it is invoked at 
>> smtpd_end_of_data_restrictions. Therefore recipient throttle cannot work 
>> at this point.
> 
> v2 overcomes this and tracks the recipients for each smtpd instance. To
> track quotas precisely v2 can be used in smtpd_recipient_restrictions
> and smtpd_end_of_data_restrictions. v2 knows about the various
> restrictions.

False. You will not be able to use Recipient throttling at
the smtpd_end_of_data_restrictions. This has nothing to do
with Policyd but an is Postfix limitation.

Cami

-
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] two instances of policyd: greylisting and throttling

[Cami Sardinha]

Edi Füllemann wrote:
> invalid triplet_array[8][2]: (recipient throttle):
> 
> As I have turned off everything except throttling, I do not expect policyd to 
> care about triplets.
> 
> The server is debian etch with policyd version 1.80
> 
> This is from main.cf
> smtpd_recipient_restrictions =
> permit_mynetworks,
> permit_sasl_authenticated,
> reject_unauth_destination,
> ...
> check_policy_service inet:[192.168.1.6]:10031,
> permit
> 
> smtpd_end_of_data_restrictions =
> check_policy_service inet:[192.168.1.7]:10031,
> permit

Recipient Throttling can not work at smtpd_end_of_data_restrictions.
What happens when 1 message has multiple recipients?

Cami

-
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] whitelist_dnsname fails (SOLVED)

[Cami Sardinha]

Dietmar Braun wrote:
> Hi,
> 
> Wednesday, March 19, 2008, 6:06:25 PM, you wrote:
>>> Which test program?
> 
>> I sent it to the list on Mar 10.  You can get it from the list archives,
>> or I can send it to you if you want.
> 
> Hm, I am getting errors while compiling it:
> 
> /tmp/ccYisHFt.o: In function `main':
> bug.c:(.text+0x19): undefined reference to `mysql_init'
> bug.c:(.text+0x63): undefined reference to `mysql_real_connect'
> /tmp/ccYisHFt.o: In function `run_long_query':
> bug.c:(.text+0x165): undefined reference to `mysql_query'
> bug.c:(.text+0x192): undefined reference to `mysql_store_result'
> bug.c:(.text+0x1a0): undefined reference to `mysql_fetch_row'
> /tmp/ccYisHFt.o: In function `run_short_query':
> bug.c:(.text+0x1d9): undefined reference to `mysql_query'
> bug.c:(.text+0x1e6): undefined reference to `mysql_store_result'
> bug.c:(.text+0x203): undefined reference to `mysql_fetch_row'
> collect2: ld returned 1 exit status

Is the MySQL v4/v5 libraries installed?

Cami

-
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup performance optimization

[Cami Sardinha]

Dominique Feyer wrote:
> 
> We use a setup with an InnoDB on our cluster (>10'000 domains, >100'000 
> accounts). We convert MyISAM to InnoDB without problem. The only one 
> chage in Policyd is the INSERT DELAY -> INSERT
> 
> With a policyd database size of 3.4Go on a dual xenon 2.4Ghz 6Go RAM it 
> perform really nice without any trouble during the last 2 month

That's extremely small. We're talking about databases with +50gigs
of data. Trying to recover from that is a pain when it breaks.
InnoDB was found to work really well till it passed the +10gig mark.

Cami

-
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup performance optimization

[Cami Sardinha]

[EMAIL PROTECTED] wrote:
> Hi Cami
> 
> I would be willing to try, though it would require some manpower and
> thus has to be planned in advance because more than one team is affected
> (we do not administer the SQL server, only Postfix and relevant
> processes).

As Nigel has pointed out, changing it for a test is probably not
the best way forward since the code is considered stable. Are you
able to try out v2 instead?

Cami

-
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup performance optimization

[Cami Sardinha]

[EMAIL PROTECTED] wrote:
> PS. which reason had the InnoDB crashes you mentioned, to have several
> eyes on that by our monitoring

It was > 2 years ago, i honestly don't recall.
The crashing wasn't the issue, the problem was when it
crashed, InnoDB goes into a recovery mode which it checks
all data for integrity, and on a system like ours
( > 6 million mails per day easily), the recovery process
could take "extremely" long.

Cami

-
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup performance optimization

[Cami Sardinha]

[EMAIL PROTECTED] wrote:
> 
> He found some MySQL document stating that with MyISAM tables, the complete 
> table gets locked during database operations,
> whereas InnoDB type of table would only lock the particular table row (but be 
> somewhat slower in use).

Correct.

> Could there be some problem, with regards to DB or table locking and policyd 
> accessing the tables at same time?

Well, when LOCK's occur, queries from Policyd -> MySQL will (forcefully)
timeout to prevent any delay. The reason we are using MyISAM instead of
InnoDB is because during testing under really high loads, we found that
InnoDB would not only be slower, but it would actually crash. Perhaps
InnoDB needs to be re-evaluated with the latest release.

Are you able to switch your Policyd database over to InnoDB?
(this will require some work, including a few Policyd changes)

Cami

-
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] how to (not readme)

[Cami Sardinha]

Miguel wrote:
> Hi, whre can i find a good how to about the postfix integration?
> Ok, i read the  README file, but in that file  is explained the options 
> that policyd supports,  but how do i enable them in postfix?

Perhaps you should re-read the README, it is covered in there.

Cami

-
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd don't work

[Cami Sardinha]

Alexandru Constantinescu wrote:

> smtpd_recipient_restrictions =
> permit_mynetworks,
> permit_sasl_authenticated,
> check_recipient_access mysql:/etc/postfix/mysql-recipient.cf,
> reject_unauth_destination,
> permit_mx_backup, 
> permit_auth_destination,
> check_policy_service inet:127.0.0.1:10031,

The problem is you allow sasl users (and people in my networks)
to bypass all your restrictions completely..

smtpd_recipient_restrictions =
  check_policy_service inet:127.0.0.1:10031,
  permit_mynetworks,
  permit_sasl_authenticated,
  check_recipient_access mysql:/etc/postfix/mysql-recipient.cf,
  reject_unauth_destination,
  permit_mx_backup,
  permit_auth_destination,
  [..]

On a side note, you really should re-look at your restrictions
as there appears to be a number of problems / oversights.
(Post your config to the Postfix mailing list for people to
  look over)

Cami

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup performance optimization

[Cami Sardinha]

Roland Rosenfeld wrote:
> On Tue, 18 Mar 2008, Cami Sardinha wrote:
> 
>> Policyd was test/written for MySQL 4.x. This doesn't mean it
>> shouldn't behave the same for v5. Unless i'm mistake (or things have
>> changed between versions), using "DELETE QUICK" on an
>> auto-incrementing row is where holes (/fragmentation) occurs. This
>> was one of the reasons that Policyd has no auto-incrementing
>> fields. Is this no longer the case with MySQL v5?
> 
> As I understand this, the timestamp columns cause the same problem
> that auto-increment columns imply

I've yet to find any documentation to state that the same
occurs for standard INT(or TIMESTAMP) types. We are using
INT's and from what i read in the MySQL documentation, there
is no fragmentation caused by it.

CREATE TABLE triplet (
   _fromchar(60) NOT NULL default '',
   _rcptchar(60) NOT NULL default '',
   _hostchar(15) NOT NULL default '',
   _datenew int(10) unsigned NOT NULL default '0',
   _datelastint(10) unsigned NOT NULL default '0',
   _count   smallint(5) unsigned NOT NULL default '0',
   UNIQUE KEY _host (_host,_from,_rcpt),
   KEY _datelast (_datelast),
   KEY _datenew (_datenew)
) TYPE=MyISAM;

Cami

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup performance optimization

[Cami Sardinha]

Roland Rosenfeld wrote:
> 
> The triplet table currently contains 5.5M entries and every hour ~250k
> entries are expired.  Without maintenance this took some minutes now.
> So I tried a "mysqlcheck -r" on the database (which took only two
> minutes) and after this cleanup runs much faster.

5.5M entries should be no problem and even with 10M entries, your
cleanup should be occurring fairly quickly.

>> You also didn't mention what version of mySQL DB you were running.
> 
> 5.0.51a

Policyd was test/written for MySQL 4.x. This doesn't mean it shouldn't
behave the same for v5. Unless i'm mistake (or things have changed
between versions), using "DELETE QUICK" on an auto-incrementing row
is where holes (/fragmentation) occurs. This was one of the reasons
that Policyd has no auto-incrementing fields. Is this no longer the
case with MySQL v5?

> Also the syslog output is broken, because the timestamps
> are created by syslogd, which writes to disk nearly a minute after the
> event happened (had to patch cleanup, now it writes its own timestamps
> to the log instead of syslog).

You shouldn't have to patch cleanup. What is the output of:
# cat /etc/syslog.conf  | grep mail

Cami

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup performance optimization

[Cami Sardinha]

Roland Rosenfeld wrote:
> 
> After only 3 days now the database extended from 3M to 6M in the
> triplet table while cleanup takes 3-5 minutes now, while policyd
> handles all mails as pass through and accepts very much spam :-(

Policyd database entries can go into the millions, at least 10-20M.

> So I decided to have a look into the cleanup code.  I see that it
> uses "DELETE QUICK" everywhere, which should do the deletion without
> rebalancing the index trees (see
> http://dev.mysql.com/doc/refman/5.0/en/delete.html).
> 
> As far as I understand this documentation, this may cause trouble with
> the _datelast and _datenew indexes, because these contain data, which
> isn't equally distributed over time but contains data that
> continuously increases.  As far as I understand the MySQL
> documentation, this will result in degenerated index trees.
> 
> Is it possible that this is the center of my performance issues?

Not possible. _date(last|new) are not auto-incrementing indexes.

> The MySQL manual suggests to do "OPTIMIZE TABLE" do fix the
> degenerated index structures, but with my database I expect this to
> run for some minutes, while the database is unusable, which I try to
> avoid.

Indeed, it can take quite some time. This should be done ~once
a month, preferably at around 2am when things are quiet.

> Any other idea?  I thought about replacing the cleanup process by a
> little perl script that does simple DELETE (without QUICK) calls with
> a much smaller limit (maybe LIMIT 500 but running every 10 seconds?).
> But for this I need an optimal index, while I also found "DELETE
> QUICK" calls in greylist.c and helo.c, which could degenerate my
> indexes, too...
> 
> Okay, I can also change them, but does it really solve my problems?
> 
> I know, that Cami discourages from using "OPTIMIZE TABLE", because the
> free space is reused by MySQL later, but how else can I fixup the
> degenerated index structures of the _datelast/_datenew indexes?

What makes you believe the indexes are degraded?

Cami



-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Strange behaviour with throttle_sasl

[Cami Sardinha]

Sebastian Tymków wrote:
> Hi,
> 
> Recntly I've been testing new configuration for my policyd. I wanted to 
> limit sending messages using _rcpt_max,_rcpt_cur and SASL.
> Everything is almost all right except ... Some messages in database have 
> status _rcpt_cur=1  and some got _rcpt_cur=2.
> When I check logs I can see that both users send only one message per 
> user, so this values should be 1 instead of 2.

Please provide DEBUG=3 logging reports.

Cami

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Policyd not expiring triplet table any more

[Cami Sardinha]

Dietmar Braun wrote:
> Hi,
> 
> yesterday, I emptied my triplet table to get it re-built. Since then,
> policyd isn't expiring any records any more, the table wich had 4 Mio
> entries in average is nearly at 6 Mio now, and as you can also see in
> the graph attached, the hourly thread peaks (due to expiring) have
> gone away.
> 
> What did I miss, what was the error? Are there any indexes which must
> be delete/rebuilt?

Run your cleanup script in DEBUG=3 mode and read the error messages.

Cami

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Policyd, throttling and SASL

[Cami Sardinha]

Sebastian Tymków wrote:
> Hi,
> 
> Recently I've installed policyd and become test it.
> It's doing good job but I've got few questions.
> 1) Is there any possibility to set policyd not to INSERT incoming emails 
> from
> athor addresses than my users/domains ?
> If I  use this configuration :
> === main.cf  ===
> smtpd_recipient_restrictions =
> check_policy_service inet:127.0.0.1:10031 
> ,
> permit_mynetworks,
> permit_sasl_authenticated,
> permit_mx_backup,
> ===main.cf
> Every mail will be cached, and my database will grow fast.
> 
> 2) I need solution that allow me disbale sending mails per day not per 
> 24 hours.
> In example: client quota has 500 mails/24h , after that time he must 
> wait 24 hours
> before he can start sending another 500 mails. What I need is allowing 
> user send
> 500 mails per day. I.e. quota exceeded 19th of february at 19:00 and on 
> 20th of february at 00:01
> quota is cleared. I now that I can use script which connect to database 
> and update throttle table.

I see you are using SASL. Do you force your users to use SASL?
If so, this could allow you to enable the SASL throttling
feature inside Policyd and not rate limit by domain.

Cami

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Policyd throttle per day

[Cami Sardinha]

Sebastian Tymków wrote:
> Hi,
> 
> I was looking for patch to policyd  that enable throttling per user but 
> per day.
> Now I see it can only block user for given time (ex. 1000 mails per hour 
> or per day).
> What I'm looking for is patch to send email per day , ex:
> 1000 mails per 19th of february and after this time database will be 
> cleaned up and
> another 1000 mails could be sent after midnight. Or maybe I'm wrong ?

Every night at 00:00, "TRUNCATE" your table. That way you will:

* keep your table at absolute minimal
* only keep transactions/records for the last 24 hours as you wish

Cami

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd and throttling

[Cami Sardinha]

Sebastian Tymków wrote:
> 
> Please supply DEBUG=3 & Postfix logging that shows this is occurring.
> 
> OK, it was my mistake . I set _count_max instead of _rcpt_max. When I 
> changed it it works fine.
> 
> Ps. Some documentation would be very helpfull and can save much time ;)

Patches are more than welcome if something is not clear enough for you.

Cami

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd and throttling

[Cami Sardinha]

Sebastian Tymków wrote:
> 
> I wonder if it's normal behaviour  that when  I try to send email  with  
> few addresses policyd counts only last one insted of all
> emails. Maybe my configuration is wrong ?
> Is there any workaround to count all emails ?

Please supply DEBUG=3 & Postfix logging that shows this is occurring.

Cami

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup reports incorrect settings?

[Cami Sardinha]

Tobias Kreidl wrote:
> 
> Why "0 days" and "1h"? These look like the default, rather than the
> actual values in the policyd.conf file.
> Is there a bug in "cleanup" or am I doing something incorrectly?

None. Its strictly a cosmetic DEBUG printf()/logmessage() that
was left behind. Things are working as it should.

Cami

--- cleanup.c   2007-07-11 09:06:23.0 +0200
+++ cleanup.c   2008-02-05 08:12:18.0 +0200
@@ -145,10 +145,6 @@
timenow - SENDER_INACTIVE_EXPIRE);
  if(db_deletequery(0) == -1) exit(-1);

-
-logmessage("expiring throttlesender instances older than 1 hour 
(%d)\n",
-  timenow - 3600);
-
  /* build up & execute query */
  snprintf(mysqlquery_array[0], 512,
"DELETE QUICK FROM throttle_from_instance WHERE _expire < %d",

-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cant get sender throttle blacklisting working

[Cami Sardinha]

Artem Bokhan wrote:
> Quota works but  _abuse_tot doen't change.

It does change.

> Can somebody point me the reason of such behavior?

_abuse_tot counts the number of times a specific
email address / domain has been blacklisted.

This is used for internal Policyd use only.

Cami

-
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] OPTINOUT and OPTINOUTALL

[Cami Sardinha]

Arvinn Løkkebakken wrote:
> 
> What am I trying to do? I am trying to internally in policyd pick 
> certain recipient domains that will not be subjected to greylisting. I 
> want all other recipient domains to be subjected to greylisting. I 
> however do not want to maintain a list of all my recipient domains in 
> the policy table.

# enable opt-in/out
OPTINOUT=1

# by default opt everyone in
OPTINOUTALL=1

# opt out just this domain
INSERT INTO policy (_rcpt,_optin,_priority) VALUES ('@domain.com', 0, 10);

All other domains will be subjected to greylisting. If you wish to
opt-in 1 email address inside that @domain.com, u can do that by
inserting a record with a higher preference.

Cami

-
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] triplets getting tooo big

[Cami Sardinha]

Jan-Frode Myklebust wrote:
> We just had a problem with the triplets database getting too big, causing
> cleanup to take ages, and policyd getting into "bypass" mode frequently.
> What are the best practices for keeping the triplets down.. ?
> 
> We had AUTO_WHITELIST_NUMBER=500, will hosts that are awl'd not go to 
> the triplets, so reducing this to say 50 (?) might be a good idea ? Or
> should we reduce TRIPLET_AUTH_TIMEOUT, which is currently set to the
> default 30 days ?

Lowering it is more than likely a good idea.

> Or are there any other options we should tune ?

Once you have whitelisting sorted out, TRUNCATE is your friend. ;)

Cami

-
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] not daemonizing correctly? SOLVED

[Cami Sardinha]

Arvinn Løkkebakken wrote:
> Arvinn Løkkebakken wrote:
>> Hi. Thanks for a fine product.
>>
>> I have set DAEMON=1 in the config.
>> I have some troubles with running policyd as a daemon though. Every time 
>> after starting policyd in a ssh terminal I am not able to end the ssh 
>> terminal when logging out afterwards. The ssh terminal lives forever 
>> until I kill the policyd process that was started from within it, or 
>> until I kill the sshd process itself.
>> Does anybody else experience the same problem?
>>
>> OS:. CentOS 5
>> Policyd 1.82.
>>
>> Regards,
>> Arvinn
>>   
> A colleague of mine looked over the code and did the following changes. 
> Here's the patch I created of the the changes:
> 
> --- sockets.c.orig  2007-08-20 08:42:05.0 +0200
> +++ sockets.c   2007-12-04 09:55:34.0 +0100
> @@ -452,11 +452,11 @@
>  _exit(0);   /* child 1 terminates */
> 
>/* child 2 continues */
> -  if(nochdir)
> +  if(!nochdir)
>  chdir("/"); /* change working directory */
> 
>/* close off all file descriptors */
> -  if(noclose)
> +  if(!noclose)
>  for(i=0;i<64;i++)
>close(i);
> 
> 
> .. claiming that the code did not seem logically correct before. If 
> noclose is set, why close? Anyway, with these changes my problem is gone :)

Your friend is correct, the patch approach however isn't.

> Cami, I'll try your patch too. Thanks.

Ignore the previous patch, it still doesn't solve the correct problem.
Corrected patch below:

--- generic.c   2007-12-04 11:32:30.0 +0200
+++ generic.c   2007-12-04 11:33:36.0 +0200
@@ -693,7 +693,7 @@
  /* dont let cleanup run in the background */
  if(prog == 0)
  {
-  if(daemonize(0,0) == -1)
+  if(daemonize(1,1) == -1)
{
  fprintf(stderr, "daemon(): %s\n", strerror(errno));
  exit(-1);


Nigel, please apply against dev branch as well.
Arvinn, thanks for the bug report/solution.

Regards,
Cami


-
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] not daemonizing correctly?

[Cami Sardinha]

Arvinn Løkkebakken wrote:
> Michael Brennen wrote:
>> On Monday 03 December 2007, Arvinn Løkkebakken wrote:
>>
>>   
>>> I have set DAEMON=1 in the config.
>>> I have some troubles with running policyd as a daemon though. Every time
>>> after starting policyd in a ssh terminal I am not able to end the ssh
>>> terminal when logging out afterwards. The ssh terminal lives forever
>>> until I kill the policyd process that was started from within it, or
>>> until I kill the sshd process itself.
>> 
>> I've not looked at the code, but this may happen if the forked policyd is 
>> not 
>> closing one or more of stdin, stdout or stderr
>
> Is there a way I'll be able to verify whether this is the case?

Please try this patch:

--- sockets.c   2007-12-04 08:01:07.0 +0200
+++ sockets.c   2007-12-04 08:01:23.0 +0200
@@ -455,16 +455,16 @@
if(nochdir)
  chdir("/"); /* change working directory */

-  /* close off all file descriptors */
-  if(noclose)
-for(i=0;i<64;i++)
-  close(i);
-
/* redirect stdin, stdout and stderr to /dev/null */
open("/dev/null", O_RDONLY);
open("/dev/null", O_RDWR);
open("/dev/null", O_RDWR);

+  /* close off all file descriptors */
+  if(noclose)
+for(i=0;i<64;i++)
+  close(i);
+
return (0);
  }

-
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Policy rejection behaviour

[Cami Sardinha]

Jon Duggan wrote:
> This email can be ignored.  The .conf had SENDERMSGLIMIT=512 and users
> getting caught were the heavy ones.
> 
> I guess the limit in the conf overrides the '0' placed in sql?

Correct.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Redundant MySQL connection

[Cami Sardinha]

Olivier Smedts wrote:
> 
> I now think that policyd doesn't need even a little modification to make 
> it work perfectly with MySQL replication. It seems that there is no need 
> for "IF EXIST" or "IF NOT EXIST" because UPDATE and DELETE can be made 
> on non-existent keys, and INSERT DELAYED doesn't return an error in case 
> of duplicate key.

`INSERT DELAYED` will never return any errors anyways.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Early rejection of sender

[Cami Sardinha]

Tobias Kreidl wrote:
> I am getting some odd results when I try to send mail that contains one 
> or more invalid addresses -- I don't want the mail client to reject 
> sending the message altogether, just report back the errors. It seems as 
> if I get greylisted almost immediately, which shouldn't happen (I 
> thought) unless I exceed a certain number of tries within a certain 
> length of time.

The point of greylisting is precisely that, to immediately reject
the first delivery attempt and force the sending MTA to resend the
message N minutes later, which at that point will be accepted.

> 
> A brief explanation to the above: We want to throttle any senders, 
> excepting specific IP addresses/subdomains in the whitelist, so 
> everything goes first through policyd to "tag" each and every message. 

You are talking about 2 different modules, greylisting and sender
throttling is 2 different beasts altogether. If you're only interested
in doing sender throttling, disable greylisting and your problem goes
away.

Regards,
Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup limits to 100000 entries

[Cami Sardinha]

Nate wrote:
> 
> Sorry Cami, I missed the loop.  This was initially noticed by 
> watching the mysql process monitor and seeing LIMIT 10 at the end 
> of the policyd cleanup queries.  I missed entirely the loop which 
> exists that resends the query if 10 rows were affected.  With 
> that explanation it makes perfect sense.  Thank you for pointing that 
> out.  I feel a little hammered by your reply, I was only trying to 
> contribute and help.

I appologise for the previous mail, the tone was uncalled for.

> We're running 5.0.x version of mysql, I wonder if 5.0 or 5.1 has 
> better delete code than previous versions which may eliminate the 
> issue of large deletes.

I certainly hope so. I personally haven't tested MySQL 5.1 with the
large deletes.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup limits to 100000 entries

[Cami Sardinha]

Tobias Kreidl wrote:
> 
> As to the choice of 100,000 as  the query limit, I would think this 
> would be machine- and database-dependent to some extent, and still think 
> it might be a good idea to at least allow the system administrator to 
> respecify it (but leave the default where it is).  Is 100,000 something 
> that was determined empirically?

This number took quite some time to get to. Very large MySQL deletes
would cause the most horrendous of problems from key corruptions to
weird things like trying to complete the delete query even if MySQL
was restarted (this was dependent on the database type being used
during testing (innodb vs myisam).

With 100,000 records deleted, the queries are running far faster, it
is easier to monitor how far into the delete you are, allows for
clean interruption and has generally made Policyd's interaction
with MySQL far more stable. Making it configurable is handing people
a loaded gun. Experienced people are more than welcome to tweak that
manually and provide feedback.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup limits to 100000 entries

[Cami Sardinha]

Benny Pedersen wrote:
> On Mon, October 8, 2007 08:29, Cami Sardinha wrote:
> if you have to do it
> 
> do
> {
> } while count(bar() > 10)
> 
> 
> did you not learn comal in school ?

No.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup limits to 100000 entries

[Cami Sardinha]

Cami Sardinha wrote:
> Tobias Kreidl wrote:
>> The issue of having the LIMIT set, period, instead of it being an option 
>> is a concern.
>> If one ever gets behind, ti would appear that depending on the load, 
>> you'd never catch up.
>>
>> Your point was that there are times where you might actually deal with 
>> over 100,000 messages within an hour and
>> if that's sustained over a few hours, the cleanup will never catch up 
>> (unless it's aware that it hit the limit and is started
>> up again).  So, I agree with your concern about the LIMIT being 
>> implicitly set and would suggest that this could perhaps at least be 
>> overridden through a command line variable.
> 
> Please, if you're going to read the source code, at least do a thorough
> job of it. Do NOT change the MySQL queries until you understand exactly
> what is happening and why.
> 
> There is NO limit of 100 000, read the source code properly.

To be more specific, follow from function to function:

cleanup.c -> mysql.c -> db_deletequery();

db_deletequery()
..
   count=0;
..
start:
..
   /* fire off query */
   if (w_mysql_query(fd, "db_deletequery") != 0)
..
 /* MySQL does not handle extremely large deletes very well */
 if((int)mysql_affected_rows(mysql) == 10)
 {
   count=count+(int)mysql_affected_rows(mysql);
   goto start;
   ^^^
|
Re-run the delete query if we hit up against the 100 000 limit
as this gives MySQL a small "breather" window. (Keep re-running
the delete query until there is less than 100 000 rows deleted)

Running cleanup in DEBUG=3 mode shows you what is happening
in step-by-step detail.

Cami


-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] cleanup limits to 100000 entries

[Cami Sardinha]

Tobias Kreidl wrote:
> 
> The issue of having the LIMIT set, period, instead of it being an option 
> is a concern.
> If one ever gets behind, ti would appear that depending on the load, 
> you'd never catch up.
> 
> Your point was that there are times where you might actually deal with 
> over 100,000 messages within an hour and
> if that's sustained over a few hours, the cleanup will never catch up 
> (unless it's aware that it hit the limit and is started
> up again).  So, I agree with your concern about the LIMIT being 
> implicitly set and would suggest that this could perhaps at least be 
> overridden through a command line variable.

Please, if you're going to read the source code, at least do a thorough
job of it. Do NOT change the MySQL queries until you understand exactly
what is happening and why.

There is NO limit of 100 000, read the source code properly.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd & memcache

[Cami Sardinha]

Artem Bokhan wrote:
> Hello again.
> 
> Did anybody make any efforts to make policyd or some of its parts to
> work with memcache?

There has been no requirements/requests from anyone for such
functionality.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd under high load and select()

[Cami Sardinha]

Artem Bokhan wrote:
> I will play with ulimit (was 1024 :( ) first and wait until bots will 
> wake up again :)
> May be it's possible to increase the limits by policyd instead of shell?..

There is no guarantee that Policyd's attempt to raise it will work.

I do recommend you do some kernel tuning as well as raising the
ulimit's. I can see from your previous post that Policyd is in fact
built to use 4096 file descriptors, however, if you're limiting
Policyd to 1024 in ulimit, can you really blame Policyd for its
behavior?

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd under high load and select()

[Cami Sardinha]

Artem Bokhan wrote:
> I'll try to change ulimit and possible MAXFD.
> But it's still bad that policyd needs to be restarted after it reaches 
> limit.

Artem,

Please downgrade to v1.80 and check/test if this still occurs.


Nigel, can you dig into this further?

-
r5 | nkukard | 2007-06-20 19:48:08 + (Wed, 20 Jun 2007) | 3 lines

* Better check to see if we've run out of available slots, if we have 
emit a warning and close connection
-

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd under high load and select()

[Cami Sardinha]

Tobias J. Kreidl wrote:
> So, in the case of 5k connections per minute, the server is probably
> saturated... does policyd log that it's reached its limit?

Connections per minute do not mean much. Concurrent connections
is what matters. It should log when its limit has been reached.

> If more connections are needed, one would of course have to 
> increase the file descriptor limit and recompile policyd according to 
> MAXFDS in policyd.h.

Correct.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd under high load and select()

[Cami Sardinha]

Tobias J. Kreidl wrote:
> Where is the maximum number of concurrent policyd processes defined?
> Is it defined/restricted by the limit of file descriptors?

Correct. Depending on your OS, ulimit -n will show the limit and
starting Policyd up in DEBUG=3 mode will show if its managed to
override that limit.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd under high load and select()

[Cami Sardinha]

Artem Bokhan wrote:
> Cami Sardinha ?:
>> What types of load?
> 
> What do you mean? :) policyd begin to eat 100% of one cpu core, even 
> after stopping postfix. The only way to fix that is restarting policyd.

Do you only have 1 machine as your MX? Do you run Policyd on
the same server as your MySQL database? It appears as if
Policyd is only allowed to use 1024 concurrent connections,
please start Policyd up in DEBUG=3 mode and post the output
of the startup.

> 1.5k processes max, I have about 5k connections per minute in peaks.

How many concurrent smtpd processes do you have on your MX?

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd under high load and select()

[Cami Sardinha]

Artem Bokhan wrote:
> Under high load policyd (1.81) stalls with 100% cpu load and excessive 
> select() in strace
> Any suggestions?

What types of load? Why are there so many concurrent connections open
to Policyd? What is the maximum amount of connections you've configured
Policyd to allow?

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] [CLOSED] greylisting and several rcpts

[Cami Sardinha]

Artem Bokhan wrote:
>> The hole comes from the fact that batch mailers / spammers
>> who often send in batches of 25/50/100 no longer have do
>> have greylisting applied to all the intended recipients.
>   
> If the one message from the batch passes through greylisting, then 
> reasonably to assume, that other messages from the batch will pass too.

Thats precisely what you do not want. The whole point is to leave
SPAM messages in the senders queue so that it becomes saturated.
(when bulk messages are being sent)

Legit mail will always come shortly after.

As for spam that is not batched up, you are crippling Policyd
and greylisting's effectiveness. As soon as 1 message accepted,
the spammer has free reign to pump as much garbage into your
valid recipients mailboxes as possible.

> So there is no difference between greylisting on duplet (when recipient 
> is NULL with postfix smtpd_data_restrictions) and triplet (in case of 
> multiply rcpt's per envelope).

Please feel free to post your suggestions to the author who came
up with the Greylisting concept: http://www.greylisting.org

Cami


-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] [CLOSED] greylisting and several rcpts

[Cami Sardinha]

Artem Bokhan wrote:
>> Sorry, this as it opens up a huge hole in how greylisting
>> works and allowing null recipients is as good as not using
>> any greylisting at all.
>>
> 
> There is no hole when MTA checks rctps.

The hole comes from the fact that batch mailers / spammers
who often send in batches of 25/50/100 no longer have do
have greylisting applied to all the intended recipients.

> As for me, I patched the source.  I don't insist on patching the 
> distrib, I just said, that it would be nice )

As you wish, i'd still advise against it. Best of luck.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] [CLOSED] greylisting and several rcpts

[Cami Sardinha]

Artem Bokhan wrote:
> Cami Sardinha ?:
>> Artem Bokhan wrote:
>>   
>>> With one empty field (empty recipient) triplet is still triplet... but 
>>> with one empty field :)
>>
>> No, its not. Run 2 instances of Policyd and the triplet will be
>> COMPLETE and the recipient will NEVER be empty.
>>
>   
> Policyd with recipient_restrictions breaks user verification by another 
> servers.

RFC compliant MTA's will retry again shortly later anyways.

> Empty triplet causes less problems then delays in verification.

Sorry, this as it opens up a huge hole in how greylisting
works and allowing null recipients is as good as not using
any greylisting at all.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] [CLOSED] greylisting and several rcpts

[Cami Sardinha]

Artem Bokhan wrote:
> With one empty field (empty recipient) triplet is still triplet... but 
> with one empty field :)

No, its not. Run 2 instances of Policyd and the triplet will be
COMPLETE and the recipient will NEVER be empty.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] [CLOSED] greylisting and several rcpts

[Cami Sardinha]

Artem Bokhan wrote:
>> You can not use greylisting without *all* the required information
>> needed to make the "triplet". 
>
> Without ignorance of principles I can't ) With ignorance - I can. The 
> result in most sutiations will be the same.

If something is unclear about the last paragraph, feel free to ask
questions to squash ignored ignorance principles ;)

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] [CLOSED] greylisting and several rcpts

[Cami Sardinha]

Artem Bokhan wrote:
> The reason of this behaviour is:
> 
> I use policyd with smtpd_data_restrictions to allow fast sender 
> verification by foreign clients.
> With smtpd_data_restrictions and multiply rcpt's per envelope no 
> recipients are passed to policyd by postfix.
> 
> Uder that conditions this part of code causes an error
> 
> if(triplet_array[fd][2][0] == 0x00) {
>   logmessage("invalid triplet_array[%d][2]: (greylist recipient): 
> %s\n", fd, triplet_array[fd][2]);
>   goto err;
> }
> 
> it would be nice if checking of NULL recipient would be configurable.

You can not use greylisting without *all* the required information
needed to make the "triplet". A better / robust solution is to have
2 instances of Policyd, one that does greylisting which happens at
the recipient_restrictions and one that does throttling at endofdata.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] greylisting vs postfix smtpd_hard_error_limit

[Cami Sardinha]

Geert Hendrickx wrote:
> On Thu, Aug 30, 2007 at 03:27:08PM +0200, Cami Sardinha wrote:
>> Indeed. We had training mode running for about 2 months in
>> order to identify the top senders / email addresses. After
>> that point when people complained, they were whitelisted.
>>
>> There was a 2 -> 3 weeks teething period after going live
>> and then it quietened down.
> 
> After how long do you expire unauthorized triplets, authorized triplets and
> auto-whitelisted hosts?
> 
> We use 1/7/30 days.

Unfortunately we found that 1 day was too short and there was
some really broken MTA's out there.. 2/30/30

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] greylisting vs postfix smtpd_hard_error_limit

[Cami Sardinha]

Geert Hendrickx wrote:
> On Thu, Aug 30, 2007 at 02:32:08PM +0200, Cami Sardinha wrote:
>> As Wietse has pointed out, you should be whitelisting the legitimate bulk
>> senders..
> 
> In an ISP environment, it is not obvious to know all your legitimate smtp
> clients..

Indeed. We had training mode running for about 2 months in
order to identify the top senders / email addresses. After
that point when people complained, they were whitelisted.

There was a 2 -> 3 weeks teething period after going live
and then it quietened down.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] greylisting vs postfix smtpd_hard_error_limit

[Cami Sardinha]

Cami Sardinha wrote:
> Geert Hendrickx wrote:
>> Hi,
>>
>> has anyone experienced bad interaction between greylisting and (by default)
>> low smtpd_hard_error_limit settings in postfix?  Our smtpd_hard_error_limit
>> has always been pretty high so I can't tell from my own experience.  But
>> with greylisting, clients get an error on each RCPT command initially, so I
>> can imagine that (legitimate) bulk senders may get disconnected too easily
>> and will only be able to insert (by default) 20 RCPT's into the greylisting
>> table at a time, causing unintended extra long delays?
>>
>> Any experiences with that?
>>
>> http://www.postfix.org/postconf.5.html#smtpd_hard_error_limit
> 
> Interesting. We have at our large installations a very low limit (of 6)
> and never had any complaints / issues.

As Wietse has pointed out, you should be whitelisting the legitimate
bulk senders..

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] greylisting vs postfix smtpd_hard_error_limit

[Cami Sardinha]

Geert Hendrickx wrote:
> Hi,
> 
> has anyone experienced bad interaction between greylisting and (by default)
> low smtpd_hard_error_limit settings in postfix?  Our smtpd_hard_error_limit
> has always been pretty high so I can't tell from my own experience.  But
> with greylisting, clients get an error on each RCPT command initially, so I
> can imagine that (legitimate) bulk senders may get disconnected too easily
> and will only be able to insert (by default) 20 RCPT's into the greylisting
> table at a time, causing unintended extra long delays?
> 
> Any experiences with that?
> 
> http://www.postfix.org/postconf.5.html#smtpd_hard_error_limit

Interesting. We have at our large installations a very low limit (of 6)
and never had any complaints / issues.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] multiple instances of cleanup

[Cami Sardinha]

John Beaver wrote:
> Cami Sardinha wrote:
>> Tobias Kreidl wrote:
>>> If one is running multiple SMTP servers, is there any harm running 
>>> "cleanup" from each one
>>> at the same time via a cron job, or is it better to stagger the times a bit?
>> If all your SMTP servers are connecting to the SAME / SINGLE Policyd
>> database, why on earth would you want to run the cleanup on them all?
>>
>> Run the cleanup script on the MySQL server itself, once and only once.
> 
> I have a throttling instance and I have a greylisting/blacklisting 
> instance.  In my case, each policyd instance is running with a different 
> configuration.  Running a single cleanup would not "clean" all the 
> tables used.  So each cleanup instance "cleans" the tables used by each 
> instance.

Sure it will, just run the cleanup on the same machine as your MySQL
database and have the same configuration of all your combined setups.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Is this throttle config possible?

[Cami Sardinha]

am.lists wrote:
> Recently, I was emailing a system admin at one of the major ISPs. He
> told me that we were blocked for going over their throttle limits,
> which he described as 10 simultaneous connections per IP and 100
> messages per hour.
> 
> Now. I use policyd 1.8x and know that I can do the #/per hour part,
> but how do I limit the number of inbound connections? Is that more at
> the Postfix level? I run in a distributed environment (one primary
> MySQL db), how do I enforce a connection limit across the environment?

Indeed, this needs to be done at the Postfix level.

/etc/postfix/master.cf
..
smtp  inet  n   -   n   -   10smtpd
..

Thats for limiting inbound connections.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] multiple instances of cleanup

[Cami Sardinha]

Tobias Kreidl wrote:
> If one is running multiple SMTP servers, is there any harm running 
> "cleanup" from each one
> at the same time via a cron job, or is it better to stagger the times a bit?

If all your SMTP servers are connecting to the SAME / SINGLE Policyd
database, why on earth would you want to run the cleanup on them all?

Run the cleanup script on the MySQL server itself, once and only once.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] SenderThrottle failures?

[Cami Sardinha]

Fernando Schubert wrote:
> Hi!
> 
> I have a question about throttle - SENDERTHROTTLE -(one more, sorry). In 
> the readme it states that `pass` meains a failure.
> I`m having thousand messages from pass like this:
> 
> Aug 24 03:27:32 srv-06-lb postfix-policyd-sf: rcpt=3132219, throttle=pass
> 
> And also `bypass`
> Aug 24 03:24:39 srv-03-lb postfix-policyd-sf: rcpt=3174638, throttle=bypass
> 
> About my older question that throttle is failing or behaving strange, I 
> got a postfix log from a client that received a Policy Rejection without 
> limit reached. I found this message strange because the Verification 
> failed statement is on it. Can this have a relation with the pass and 
> bypass messages throttle is giving me?

No idea .. possibly. Ensure your database tables are not corrupt as you
should never be having "thousands" of bypass/pass failures.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Error in bind()

[Cami Sardinha]

Jordi Espasa Clofent wrote:
> 
> #
> # port to bind to:
> #
> #   port which the policy daemon will listen on
> #
> BINDPORT=10031
> 
> I think this is correct and there is not any service in this port:
> 
> [EMAIL PROTECTED] ~]# netstat -punta | grep LISTEN
> tcp0  0 127.0.0.1:10024 0.0.0.0:* 
>  LISTEN  18491/amavisd (mast
> tcp0  0 127.0.0.1:10025 0.0.0.0:* 
>  LISTEN  18449/master
> tcp0  0 127.0.0.1:10026 0.0.0.0:* 
>  LISTEN  18491/amavisd (mast
> tcp0  0 0.0.0.0:33060.0.0.0:* 
>  LISTEN  11369/mysqld
> tcp0  0 0.0.0.0:878 0.0.0.0:* 
>  LISTEN  1120/rpc.statd
> tcp0  0 127.0.0.1:10031 0.0.0.0:* 
>  LISTEN  22299/policyd

Policyd is already running. kill -9 22299 if it refuses to
go down (unlikely).

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Good SPF (only ?) policy deamon ?

[Cami Sardinha]

Xavier Beaudouin wrote:
> Since there is not yet (is this planned?) SPF support to policyd, does
> someone here has a good idea for spf policy server for postfix ?

This functionality will not be built into Policyd, there is a different
policy server which does SPF checking.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Throttling not working well

[Cami Sardinha]

Fernando Schubert wrote:
> Hi everybody!
> 
> I`m facing a problem with SENDERTHROTTLE
> I use it to enforce quota in a small mail cluster (8 servers) 
> environment. I have large traffic and I`m encountering some problems 
> with throttling.
> My defaults are 1000 mails or 40Mb of data in 24 hours. But many times I 
> got quota limit error message with less than 20% from the limit used. I 
> don`t know if my high volume of mail received can affect this or is 
> other fact that is doing this.
> My database is working well, policyd uses more or less 500 simultaneous 
> connections on it but my load is developed to support more than 5000 conns.
> 
> This situation is very complicated for me, because many times I got 
> errors that the box limit is full but when i search in database the 
> _quota_cur and _count_cur are very far from the limit I forced. I`m 
> using SASL_THROTTLING instead of per HOST.

We have Policyd doing sender throttling in some really extremely
loaded ISP's (strictly doing throttling) without any such weird
things happening. Please provide proof (run Policyd with DEBUG = 3)

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Memory hog?

[Cami Sardinha]

Leonardo Rodrigues Magalhães wrote:
> 
> 
> Cami Sardinha escreveu:
>> Once your
>> system starts processing 10 000 concurrent connections to your
>> MX then you need to start raising limits.
> 
>My mail systems probably doesnt handle 1 connections in a whole 
> day . 10k concurrent connections are a huge insane idea to me :)

That was the reason i started Policyd, the other Policy servers did
not scale under insane loads.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Memory hog?

[Cami Sardinha]

Dave wrote:
> 
> ;)
> Cool - thanks for the pointer.  In fact MAXFDS is set to 4096 in the 
> Makefile. I've altered it to 1024 and policyd now only uses 12456kB.  
> 
> 1024 should be more than enough slots for my use but we'll see...  
> 
> Any particular reason it was increased to 4096 in the Makefile?  Did someone 
> find 1023 wasn't enough or was it to be 'on the safe side'?

Policyd is used to protect some isps with > 750 000 mailboxes
which of course is a *lot* of work that Policyd has to do. So
yea, it was increased for performance reasons. 1024 should be
able to *easily* cater for small -> medium setups. Once your
system starts processing 10 000 concurrent connections to your
MX then you need to start raising limits.

Regards,
Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Memory hog?

[Cami Sardinha]

Dave wrote:
> Policyd seems to be taking a huge chunk of memory - is there any way I can 
> control it?
> 
> Currently it runs in 34032 kB.  This wasn't a problem when I had a whole 
> server but I'm currently running on a memory-restricted VPS and I can't see 
> why it wants so much memory.  The server only recives c. 1200 emails a day, 
> 85% of which are local and accepted without Policyd and 10% are rejected on 
> Helo checks before Policyd even sees them.
> 
> Kill & restarting policyd still gives it 34MB.
> 
> I think uLimit applies to every program on the server so that wouldn't be any 
> good - is there another way to restrict policyd's memory usage?

Short answer: No. Policyd pre-allocates all required memory and it does 
not increase.
Long answer: Yes, you can modify the amount of policy threads that will
be accepted  by modifying policyd.h


/* Miscellaneous constants */
#define LISTENQ 1023/* 2nd argument to listen() */
#define MAXLINE 1023/* max text line length */
#define BUFFSIZE8191/* buffer size for reads and writes */
#define BUFSIZE 4095
#ifndef MAXFDS
   #define MAXFDS1023/* max file descriptors   */
#endif


Change at your own peril, if it breaks, you get to keep both pieces. ;)

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd/popb4smtp problem ?

[Cami Sardinha]

Voytek Eymont wrote:
> I'm running Postfix with popb4smtp and policyd, it all works well.
> 
> however, I've now tried sending email from my Palm hand/held over GPRS;
> 
> I do a POP retrieve pass, wait, then try to send, however, Snapper mail
> fails to send and says like below:
> 
> am I getting greylisted and thus not updating popb4smtp...?
> 
> 
> mail client error:
> 
> rcpt to failed 450
> recipent address rejected
> policy rejection
> pls try later

  smtpd_recipient_restrictions =
   permit_mynetworks,
   check_client_access hash:/etc/postfix/pop-before-smtp,
   reject_unauth_destination,
   check_recipient_access hash:/etc/postfix/recipient_no_checks,
   reject_non_fqdn_sender,
   reject_non_fqdn_recipient,
   reject_invalid_hostname,
   reject_non_fqdn_hostname,
   reject_unknown_sender_domain,
   reject_unlisted_recipient,
   check_sender_access hash:/etc/postfix/freemail_access,
   check_recipient_access hash:/etc/postfix/recipient_checks,
   check_recipient_access pcre:/etc/postfix/recipient_checks.pcre,
   check_helo_access hash:/etc/postfix/helo_checks,
   check_sender_access hash:/etc/postfix/sender_checks,
   check_client_access hash:/etc/postfix/client_checks,
   check_client_access pcre:/etc/postfix/client_checks.pcre,
###
   check_policy_service inet:127.0.0.1:10031,
###
   reject_rbl_client zen.spamhaus.org,
  ..snip..
   permit

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] The table throttle_from_instance

[Cami Sardinha]

Hi Aslan,

> I've see more information about the table throttle_from_instance, and I
> found this old thread in the list.
> 
> http://www.mail-archive.com/policyd-users%40lists.sourceforge.net/msg00221.html
> 
> Cami, you said that we can truncate the table throttle_from_instance
> once a month, why is the real necessarily to do it? 
> Can I truncate the table once a week or once a day ? 

You should/can truncate this table once a week but i really should
advise against doing so sooner that can. A message can be transmitted
to various recipients over at least 3 days, truncating the table sooner
than that would lead to your quota count being slightly off (at least
until the quota expire time is tripped).

Regards,
Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Redundant MySQL connection

[Cami Sardinha]

Stanislav Sinyagin wrote:
> --- Cami Sardinha <[EMAIL PROTECTED]> wrote:
>> And what happens when someone requests a 3rd, 4th and 5th MySQL
>> backup option?
> 
> nobody would need that :) 
> The approach that I suggested would work perfectly for 1+1 redundancy.
> If someone wants a bigger redundancy solution, it would anyway require some 
> customization of the code (if not a complete redesign).
> 
> I understand that Mysql dual-master setup is not a rocket science - I'll 
> most probably go that way myself. But the way I suggested would minimize the 
> administrative burden and synchronization efforts. In theory, that could 
> even lead to a self-healing redundant solution... but then it really needs 
> a redesign and refactoring, so that the storage level is separated from 
> the operation logic.

It is not Policyd's job to perform such work. Dual writes increases
latency for each Postfix -> Policyd transaction, introduces another
single point of failure and makes Policyd less robust (especially
when troubleshooting weird problems).

The approach that should be taken is:

1) Set up master -> slave replication
2) Change policyd so that all READS happen from the slave(s)
and WRITES only go to the master.

Such a solution not only makes your environment cleaner and
problems easier to diagnose, but it makes Policyd more robust.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Redundant MySQL connection

[Cami Sardinha]

Stanislav Sinyagin wrote:
> It would be great to implement a new option in policyd. 
> Technically it should not be difficult.
> 
> The new option would list a backup MySQL server/dbname/user/password, 
> and if it's specified, every INSERT statement is duplicated on that server.
> 
> This would allow an elegant redundant solution: two policy gateways 
> would update each other's SQL databases, without the need of complex database 
> synchronization.

And what happens when someone requests a 3rd, 4th and 5th MySQL
backup option?

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] question on smtpd_recipient_restrictions order

[Cami Sardinha]

Leonardo Rodrigues Magalhães wrote:
>   
>I would like to greylist connections that are trying to send messages 
> to unknown addresses. Today those connections receives 'unknown user' 
> and policyd never heard of them. I would like them to receive the Policy 
> Rejection because of greylist=new and, IF they returns, policyd would 
> give greylist=update and only then postfix would give 'unknown user'.

This is not going to slow down the amount of spam you get and
really won't stop dictionary attacks either. Set your hard limit
in Postfix to 5 or something similiar.

Cami

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] chroot breaking name lookups

[Cami Sardinha]

Jan-Frode Myklebust wrote:
> We run policyd v1.81 as user/group policyd, chroot to an empty
> directory /var/empty/policyd/. But then policyd isn't able to
> resolve my MYSQLHOST hostname. If I use ip-address it works fine.
> 
> It complains:
> 
>   policyd: connecting to mysql database: 
>   policyd: mysql_real_connect(): Unknown MySQL server host 
> 'mysqldserver.mydomain.net' (3) 
> 
> It seems it requires access to nscd's socket /var/run/nscd/socket, 
> so if I bind mount this to /var/empty/policyd/var/run/nscd/socket, 
> everything works fine.
> 
>   mount -o bind /var/run/nscd /var/empty/policyd/var/run/nscd
> 
> Any ideas for how to avoid this ? 
> 
> I've tried putting a hosts file, resolv.conf and nsswitch into the chroot/etc,
> setting up a more complete chroot with dev/null, dev/random, to no avail..

The resolution could take place before the chroot() occurs.
Nigel?

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Throttled email will not leave queue

[Cami Sardinha]

Matt Beckman wrote:
> Thanks, John.
> 
> How can I determine if it is down? I opened a telnet session on port 10031 to 
> localhost and it connected. I didn't issue any commands, though, if you have 
> some testing suggestions. Also, the logs show that postfix-policyd connected 
> (including showing the current threshold), then went through the process of 
> sending the first message. The first message was sent and immediately 
> received. The next messages all showed "NOQUEUE: reject: ... Policy 
> Rejection- Quota Exceeded". After 10 minutes, it then showed a series of 
> connections:
> 
> Jul 18 14:50:41 mailserver4 postfix/qmgr[25772]: 7DED49C0FE: from= AT ]domain[ DOT ]com>, size=369, nrcpt=1 (queue active)
> Jul 18 14:50:41 mailserver4 postfix/qmgr[25772]: 5FC0F9C225: from= AT ]domain[ DOT ]com>, size=369, nrcpt=1 (queue active)
> Jul 18 14:50:41 mailserver4 postfix/qmgr[25772]: C83E89C31F: from= AT ]domain[ DOT ]com>, size=370, nrcpt=1 (queue active)
> Jul 18 14:50:41 mailserver4 postfix-policyd: connection from: 127.0.0.1 port: 
> 56370 slots: 0 of 4096 used
> Jul 18 14:50:41 mailserver4 postfix-policyd: connection from: 127.0.0.1 port: 
> 56371 slots: 1 of 4096 used
> Jul 18 14:50:41 mailserver4 postfix-policyd: connection from: 127.0.0.1 port: 
> 56372 slots: 2 of 4096 used
> Jul 18 14:50:54 mailserver4 postfix-policyd: connection from: 127.0.0.1 port: 
> 56373 slots: 3 of 4096 used
> Jul 18 14:55:41 mailserver4 postfix/smtp[1638]: 7DED49C0FE: to=<[EMAIL 
> PROTECTED]>, relay=127.0.0.1[127.0.0.1]:10031, delay=86153, 
> delays=85853/0.06/300/0, dsn=4.4.2, status=deferred (conversation with 
> 127.0.0.1[127.0.0.1] timed out while receiving the initial server greeting)
> Jul 18 14:55:41 mailserver4 postfix/smtp[1639]: 5FC0F9C225: to=<[EMAIL 
> PROTECTED]>, relay=127.0.0.1[127.0.0.1]:10031, delay=86135, 
> delays=85835/0.09/300/0, dsn=4.4.2, status=deferred (conversation with 
> 127.0.0.1[127.0.0.1] timed out while receiving the initial server greeting)
> Jul 18 14:55:41 mailserver4 postfix/smtp[1640]: C83E89C31F: to=<[EMAIL 
> PROTECTED]>, relay=127.0.0.1[127.0.0.1]:10031, delay=86117, 
> delays=85817/0.12/300/0, dsn=4.4.2, status=deferred (conversation with 
> 127.0.0.1[127.0.0.1] timed out while receiving the initial server greeting)
> 
> The messages were never sent after this point.

And the messages never will be.

You did not setup Policyd and Postfix correctly.
(HINT: relay=[..]:10031 <- policyd is not an mta, dont relay mail to it)

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Increased Spam

[Cami Sardinha]

Joe Lanager wrote:
> Cami,
> 
> So once I've identified and added the MTA/networks I want to the
> whitelist I can just run a query to empty out the triplet table?

Yup, "TRUNCATE TABLE triplet;" is the better option.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Increased Spam

[Cami Sardinha]

Joe Lanager wrote:
> Hello,
> 
> We've been using PolicyD for going on a year now in addition to our
> already established antispam solution and are very much impressed with
> it so far.  Initially after deployment there was a relatively small
> amount of spam making it to the milter, but as expected we had some
> users complaining of delayed delivery from some external mail systems.
> Over time these delays have all but disappeared as the triplet table
> populated.  But, we've also seen a steady increase in the amount of spam
> making it to our milter.  So my question is, should I be cleaning out
> the triplet table table or is there some time out or other setting I
> should be adjusting so that keep greylisting working at peak efficiency?

Stats should be generated every 3 months on the top sending MTA/networks
and they should be whitelisted. The rest of the data should be turfed.

I'm looking to implement some new anti-spam ideas/projects when i get
some time that might interest Policyd users. I won't drop any hints yet.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] smtpd_recipient_restriction

[Cami Sardinha]

Aslan Carlos wrote:
> Hi Folks, 
>  I has many problems to use the policyd, maybe some bugs for use in
> my environment, in specify using the throttle rcpt and throttle from
> putting in smtpd_recipient_restriction I looked the policyd doing two
> INSERTs. the first insert in the 'throttle' table inserting the 'mail
> from' and the second insert in throttle_rcpt table inserting the 'rcpt
> to', this is a bug or this is corret? 
> This makes not good sound, because when I put at smptd_recipient
> restrict at my postfix, I just wanna throttle at the 'rcpt to' and not
> 'mail from' too.

Unfortunately i can not understand what you're saying. Please provide
logs with DEBUG=3 and your main.cf / restrictions.

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] feature request (unless of course it's already in there and I just can't read)....

[Cami Sardinha]

Tim B. wrote:
> Something around the lines of:
> 
> Perfect world would be:
> If sender or sender domain wants to send to recipient or recipient 
> domain check to see if it is explicitly allowed or not allowed.
> 
> If there is no explicit rule continue on normally.

I managed to get some time over the weekend to work on it
and work out the logic. It was pretty simple to do.  I'll
be finishing off the documentation today, do some testing
and will post it when done/happy.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] feature request (unless of course it's already in there and I just can't read)....

[Cami Sardinha]

Tim B. wrote:
> 
> Thanks for getting back about this, I wish I knew C better, I'd get 
> working on it myself if I did

I might have some time next week if one of my other projects
finishes early. What exactly are the requirements?

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Sender throttling logic

[Cami Sardinha]

John Beaver wrote:
> Ronan Mullally wrote:
> 
>>  - What effect does whitelisting have on throttles?  Does a message that
>>passes a whitelist get added to a sender's quota?

No, whitelist is a way to bypass Policyd completely.

We are aware that some people would like the functionality
to choose whether to whitelist first or throttle first
and this will be provided in a later release of Policyd
once we have gotten all the other things out of the way.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] feature request (unless of course it's already in there and I just can't read)....

[Cami Sardinha]

Tim B. wrote:
> If this is already in policyd, then I'm over looking it some how so feel 
> free to thwap me upside the head
> 
> I'm currently evaluating policyd and other policy servers for a fairly 
> busy system.  So far I'm working with policyd-v1.80.tar.gz (non 
> development branch). We have a few different versions of postfix 
> installed ranging from 2.1.x to 2.3.x.
> 
> We have a few domains with different users and occasionally there arises 
> the request from one domain to block a particular sender/domain, but 
> another domain wants mail from the same sender/domain.  So what I would 
> like would be some way to say "This sender or sender domain is allowed 
> to send to this recipient or recipient domain, but not anyone else."  
> and also be able to say This sender or sender domain is allowed to send 
> to everyone except a specific recipient or recipient domain.

This would require a whole new module, perhaps called rcpt_acl which
shouldn't be too difficult to implement. Currently there are a lot of
other things in the pipeline so i recommend in the mean time you look
into Postfix's restriction classes.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Blacklist...

[Cami Sardinha]

Sam Przyswa wrote:
> HI,
> 
> I installed Postfix-PolicyD, it's really a very good, perhaps the best, 
> spam filter we tested. I have two question:
> 
> 1) There is some spam again (very few) is it a ways to send the spams 
> headers on the list or something else to add the blacklist-helo and 
> spread this list over the net ?

Policyd never sees this information so it can't use it. What you're
probably after is Pyzor I think its called, or perhaps even DCC.

> 2) The mails rejected are lost, is there a way to put in quarantine dir 
> in Maildir format ?

That would defeat the whole point of greylisting.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Developers mailing list & SVN repository

[Cami Sardinha]

Nigel Kukard wrote:
> 
> Policyd now has a developers mailing list, policyd-devel. The purpose of
> this list is to provide an environment for the developers of Policyd,
> entities maintaining their own patchsets or anyone with something to
> contribute to come and discuss. This list is aimed at the development
> side of Policyd. Anyone interested can join here
> http://lists.sourceforge.net/mailman/listinfo/policyd-devel .
> 
> The policyd-devel mailing list doesn't replace the policyd-users list,
> its merely a list which focuses solely on the development of Policyd
> while the policyd-users list focuses on general questions and support.

Those who have custom patches and ideas that revolve around
future development should subscribe before Monday since that
is when I'll be submitting patches for review.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

[policyd-users] New Policyd maintainer

[Cami Sardinha]

Hi All,

With immediate effect, I'm happy to announce that
Nigel Kukard ([EMAIL PROTECTED]) will be taking over
Policyd. He knows the code fairly well and is
responsible for the non-blocking read()/write()
code that was included some time ago.

I'll be working together with him at the start but
he will be taking the lead. There are a number of
things to implement (scoping, etc..)

Thanks to everyone who has contributed this far.

Regards,
Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] feature request: p0f integration

[Cami Sardinha]

Leonardo Rodrigues Magalhães wrote:
> 
> Cami Sardinha escreveu:
>> I was thinking about this feature request. I personally greylist
>> all incoming mail but i can see merit in your idea since (apparently)
>> up to 95% of spam originates from windows based machines.
>>
>> I don't have time to write this type of functionality (it would be
>> quite a bit of work to do) so if anyone is willing to take a shot
>> at it, let me know.
> 
> seems p0f integration with policyd would be VERY interesting. 
> Unfortunely i dont have skills for taking over the project or even this 
> feature.
> 
> But i came with a question  i dont know exactly how p0f works, 
> but probably it needs to see the 'real' client traffic for identifying 
> it or at least try it. Policyd, as we know, never talk with the real 
> clients. It's called from MTA (postfix in policyd case) and receives 
> only some minor informations about the connection: IP, hostname, sender, 
> recipient, etc etc.
> 
> Question is: Would it be a policy_service the right place for p0f 
> integration ?? I mean, do policyd receives complete information p0f 
> needs for OS fingerprinting ???

The correct place to do this is inside of Policyd.

P0f needs to be running on your MX machines and it
builds a "list" of ip addresses + os. When a request
from Postfix -> Policyd arrives, Policyd will query
the p0f cache (via pipe/socket) and then apply the
necessary restrictions. It's not brain surgery to do
but i don't have the time to do it.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] feature request: p0f integration

[Cami Sardinha]

Chris Covington wrote:
> hello list, Cami,
> 
> I've been using policyd since July 2005 and it's been the best thing
> since sliced bread for us.  I would like to make a feature request (or
> perhaps this can be configured without changing policyd).  I would
> like to, for domains which require very quick responses, greylist only
> some hosts: like those that match a dynamic regex or are 'unknown' to
> postfix.  I was thinking it would be great if I could also greylist
> windows hosts.  p0f seems like the solution for that, is there any way
> I can call policyd so that a windows result from p0f would greylist
> the host?

I was thinking about this feature request. I personally greylist
all incoming mail but i can see merit in your idea since (apparently)
up to 95% of spam originates from windows based machines.

I don't have time to write this type of functionality (it would be
quite a bit of work to do) so if anyone is willing to take a shot
at it, let me know.

Policyd is also in dire need of a rewrite, if someone is willing to
take over the project, please contact me.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] autowhitelisting design question

[Cami Sardinha]

Geert Hendrickx wrote:
>> Some companies have heavily loaded databases so in order to minimize
>> downtime they run the cleanup script weekly.
> 
> In case you're only using whitelisting+greylisting (as we are, ATM), doing
> the auto-whitelisting asynchronously takes aways 1/3rd of the queries for
> each policyd request.

Amount of queries really aren't an issue, the queries are light-weight
and fully indexed.

> Granted, the cleanup run will be heavier, but those can be scheduled in
> off-peak hours.

Heavier? ;) You're talking about an exceptionally heavier query
since there would be comparisons done for *every* triplet whos
threshold matches the auto-whitelisted count. The triplet is
already being checked, so use that information instead of having
to re-query the same data later.

>> I guess if there was a demand for it, it could be user-configurable.
>> Patches will be accepted if there is a demand for it.
> 
> Alternatively, the deletes of auto-whitelisted triplets could be postponed
> to the cleanup script?

That really does not save again, instead it makes the cleanup script
take far longer to process.

What problem are you trying to solve? This method is running under
very large isp's (+700 000 mailboxes) and the MySQL server is doing
just fine.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] autowhitelisting design question

[Cami Sardinha]

Geert Hendrickx wrote:
> 
> I have a question about the auto-whitelisting algorithm; why is it
> implemented in greylist.c and not in cleanup.c?  The way it is done now
> (synchronously) means an extra query ('SELECT COUNT(*) FROM triplet
> WHERE _host='%s' AND _count > 0') for each policyd request, whereas this
> overhead could easily be avoided by performing the auto-whitelisting and
> the associated triplet cleaning asynchronously in the cleanup process.

It used to be done via the cleanup script, but people complained.

Doing it in real-time has the advantage of .. well .. being real
time without so many delays within the duration of cleanup -> cleanup.
Some companies have heavily loaded databases so in order to minimize
downtime they run the cleanup script weekly.

I guess if there was a demand for it, it could be user-configurable.
Patches will be accepted if there is a demand for it.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] postfix-policyd with MySQL accross a WAN

[Cami Sardinha]

Dean Manners wrote:
> Cami,
>   My apologies. Debian sarge (2.4.18 kernel). Policyd was backported
> from testing to sarge.
> 
> # dpkg --list |grep policyd
> ii  postfix-policyd 1.80-2.1
> anti-spam plugin for Postfix
> 
> Installed with Debians apt-get, from our custom package repository.  The
> same backported package was also used on the two working machines.

Can you build it yourself from source as a test?
Please run DEBUG=3 and mail it as an attachment so that
nothing gets wrapped.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] postfix-policyd with MySQL accross a WAN

[Cami Sardinha]

Dean Manners wrote:
> -
> Jun 15 12:02:55 secondary postfix-policyd: connection from: 127.0.0.1 port:
> 49516 slots: 8 of 4096 used Jun 15 12:02:55 secondary postfix-policyd:
> DEBUG: fd: 8 select(): fd 8 is ready for read Jun 15 12:02:55 secondary
> postfix-policyd: DEBUG: fd: 8 connection got an EOF, data_read = 0 Jun 15
> 12:02:55 secondary postfix-policyd: DEBUG: fd: 8 shutting down fd 8 Jun 15
> 12:02:55 secondary postfix-policyd: DEBUG: fd: 9 select(): fd 9 is ready for
> write Jun 15 12:02:55 secondary postfix-policyd: DEBUG: fd: 9 returning
> after 14 bytes of data written Jun 15 12:02:55 secondary postfix-policyd:
> DEBUG: fd: 10 select(): fd 10 is ready for write Jun 15 12:02:55 secondary
> postfix-policyd: DEBUG: fd: 10 returning after 14 bytes of data written Jun
> 15 12:02:55 secondary postfix-policyd: DEBUG: fd: 11 select(): fd 11 is
> ready for read Jun 15 12:02:55 secondary postfix-policyd: DEBUG: fd: 11
> connection got an EOF, data_read = 0 Jun 15 12:02:55 secondary
> postfix-policyd: DEBUG: fd: 11 shutting down fd 11 Jun 15 12:02:55 secondary
> postfix-policyd: shutdown(11): Transport endpoint is not connected
> -
> 
> Any ideas on what to try?

What OS are you running? How was Policyd installed?

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] policyd reject null address

[Cami Sardinha]

jibie wrote:
> Jun  5 00:03:29 mx postfix/smtpd[64668]: NOQUEUE: reject: RCPT from 
> mail.censored.de[??.??.??.??]: 450 4.7.1 <[EMAIL PROTECTED]>: 
> Recipient address rejected: Policy Rejection- Exception triggered by 
> policyd - Simplicato; from=<> to=<[EMAIL PROTECTED]> proto=ESMTP 
> helo=

Please stop making false accusations before providing PROOF.
Run Policyd in DEBUG mode (DEBUG=3) and then submit the Policyd logs.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Sender throttling

[Cami Sardinha]

jibie wrote:
> Cami Sardinha wrote:
>> jibie wrote:
>>   
>>>> If one really wanted to do this in Policyd, you will need to run 2 
>>>> instances of Policyd. (one doing sender throttling by SASL/from
>>>> addresses and another instance throttling by HOST addresses).
>> The code is fairly simple to follow and making those changes shouldn't
>> be that difficult. Such functionality will not be natively supported
>> inside of Policyd itself and any patches you make/create will 1) need
>> to be applied to future revisions and 2) will stop all support when/if
>> things break.
>
> Is there any reason why you didn't include this feature in the first place?

The functionality is already provided via multiple instances of Policyd.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Getting 554 error on whitelisted entries

[Cami Sardinha]

Vladimir wrote:
> I'm having some difficulty getting policyd working under Postfix 2.3.3
> under Centos. I would like to use it in the SMS gateway scenario where I
> limit the amount of messages that can be sent out to an external address
> in a period of time.
> 
> I compiled, installed and started policyd daemon.  I'm using the default
> config file and loaded the sample WHITELIST.sql.
> 
> I added following line in main.cf (I couldn't get the one in
> documentation going at all). Rest would immediately get 554 error
> 
> smtpd_recipient_restrictions = check_policy_service
> inet:127.0.0.1:10031, reject

Get Postfix working without Policyd first. Your config is missing
options and is extremely bare.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Sender throttling

[Cami Sardinha]

jibie wrote:
>> If one really wanted to do this in Policyd, you will need to run 2 
>> instances of Policyd. (one doing sender throttling by SASL/from
>> addresses and another instance throttling by HOST addresses).
> 
> Haha =) This is exactly what I thought of initially, but I dismissed it 
> as a bad hack. Does this issue only exist, because policyd currently 
> only uses a single table for sender throttle info and I guess the code 
> expects the data to be one or the other but not both? 
> 
> I haven't had a chance to look at the code yet, but how amenable do you 
> think it would be to using an extra table and allowing both types of 
> checks? 
> 
> I don't want to muck with everything if there's deep magic here that I 
> don't really understand yet. But if this only requires a little bit of 
> coding to get working, I can try working on it this weekend.

The code is fairly simple to follow and making those changes shouldn't
be that difficult. Such functionality will not be natively supported
inside of Policyd itself and any patches you make/create will 1) need
to be applied to future revisions and 2) will stop all support when/if
things break.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] Sender throttling

[Cami Sardinha]

John Beaver wrote:
> jibie wrote:
>> Is there any way to activate both sender throttling by from address and 
>> ip address? 
>>
>> The reason I ask, is that I wanted to use policyd on an incoming mail 
>> gateway machine, which from time to time, gets a sudden burst of email 
>> from spammers, which i hoped to block with sender throttle by ip and far 
>> less often from users misconfiguring a mailer script and suddenly 
>> sending a large batch of messages from several different ip's with the 
>> same from address. It would be nice to be able to configure policyd to 
>> throttle based on both types of limits
>>
>> (what I need)
>> ==
>> sender provided from
>> sender ip
>>
>> Currently as I understand it, one must enable SASL throttling to 
>> throttle based on the from. Policyd will attempt to throttle by sasl 
>> user, which won't be present on an incoming email to my domain and then 
>> fall back to sender from based throttling.  This throttle option cannot 
>> also be turned on with host/ip based throttling. 
>>
>> Are there any work arounds for this?
> 
> For IP based throttling, I use the postfix anvil feature to keep my 
> system from being overrun.  Not sure how to do it with Policyd itself.

If one really wanted to do this in Policyd, you will need to run 2 
instances of Policyd. (one doing sender throttling by SASL/from
addresses and another instance throttling by HOST addresses).

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] autogreylisting recipients from authorized senders

[Cami Sardinha]

Eric A. Litman wrote:
> 
>> To do exactly what you asked for is difficult because the
>> destination host, A or MX details are not passed as a
>> parameter from postfix to the policy daemon.
> 
> OK, I thought that might be the case.
> 
> The two possible solutions to this that come to mind are to either  
> add the recipient's address to 'whitelist_sender', or parse the RHS  
> of the recipient's address for destination MX/A/[dotted.quad] records  
> and add the results to 'whitelist' as we've been describing.

Sorry. I will not attempt to jump through hoops when it is
*extremely* inconsistent and unreliable. There is nothing
at all that guarantee's that any MX/A record or anything of
the sort is going to be the MTA that delivers the message
to your MX. We have floating MX's on our side, this method
would fail horribly.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] autogreylisting recipients from authorized senders

[Cami Sardinha]

Eric A. Litman wrote:
> I recently switched to policyd from tumgreyspf hoping I'd find a  
> solution to one of the more annoying challenges of greylisting in  
> general. Policyd doesn't yet support what I'm hoping to find, but  
> maybe someone knows of a clever solution to a problem we all face.
> 
> Scenario: you send a message to someone at a domain that has never  
> contacted you in the past, and no grey/whitelist entry exists for  
> them on your mail server. They receive your message quickly, reply  
> immediately, but despite being from someone with whom you know you  
> want to communicate, their message is relegated to going through the  
> normal greylisting process. Generally this doesn't add significant  
> delay, but sometimes it does, and sometimes even a reasonably short  
> delay is more than it should be.
> 
> I'd love to see a solution where recipients of outbound messages  
> could have some combination of the server(s) MX'd for their domain -  
> or in policyd fashion, a configurable number of quads of that/those  
> server's IP addresses - automatically greylisted/whitelisted? Maybe  
> you go simple and only use the address of the server that actually  
> receives the message as the basis for auto(g/w)listing, or maybe you  
> expand all available MX or A records for the RHS of the address.

Have you read policyd.conf?

-
#
# automatic whitelisting  default: off
#
#   this allows whitelisting of remote networks who have sent
#   more than AUTO_WHITELIST_NUMBER of authenticated triplets.
#
# 1=on  0=off
AUTO_WHITE_LISTING=0

#
# whitelist netblock/24:  default: 0
#
#   when hosts get autowhitelisted, should the host be whitelisted
#   or should the entire netblock (class C).
#
# 1=class 0=host
AUTO_WHITELIST_NETBLOCK=0
-

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] changing reject code to 451

[Cami Sardinha]

Andy Lee wrote:
> I just tried defer_code=451 but my logs are still showing that
> defer_if_permit is using 450. Can anyone else confirm? I am running
> Postfix 2.1.5.

This needs to go to the Postfix mailing list.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] changing reject code to 451

[Cami Sardinha]

Benny Pedersen wrote:
> On Fri, May 18, 2007 22:51, Thomas Johnson wrote:
>> Just a thought, but couldn't you also do this in main.cf:
>> defer_code = 451

This is the better route to take.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] throttle_from_instance

[Cami Sardinha]

Filip Cristian wrote:
> Cami Sardinha wrote:
>> Filip Cristian wrote:
>>   
>>> Can anybody please tell me what is the scope of throttle_from_instance
>>> table ? I can't find in to the Readme file or in the config file
>>> anything related to this table and on my server this table has grow very
>>> big (1.2 G).
>> 
>> You can truncate this table once a month.
> 
> Ok, but can you please tell me what is the scope for this table? What
> config option enable the usage of this ?

This is used internally by the throttling module to keep
track of unique mails that flow via Postfix -> Policyd.

If you enable any of the throttling modules this functionality
will kick in automatically.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users

Re: [policyd-users] throttle_from_instance

[Cami Sardinha]

Filip Cristian wrote:
> Can anybody please tell me what is the scope of throttle_from_instance
> table ? I can't find in to the Readme file or in the config file
> anything related to this table and on my server this table has grow very
> big (1.2 G).

You can truncate this table once a month.

Cami

-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
___
policyd-users mailing list
policyd-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/policyd-users