Re: Policy Server Development
On Fri, 16 Apr 2021, post...@ptld.com wrote: On 04-16-2021 1:04 pm, Wietse Venema wrote: As Viktor noted, each smtpd(8) process makes its own connection to a policy service. Then, an smtpd(8) process will reuse its own policy service connection, not a connection that belongs to a different smtpd(8) process. Okay, if im understanding, the expected behavior is it is supposed to be one new spawn for each client connection/event? So the answer im looking for is my script should self terminate when it detects the client (postfix) disconnect? Is that the expected behavior, there are no other clues given by postfix to the policy server when it's no longer needed? We presume that your listener, conceptually speaking, is spawning a thread / process for every new (source(address,port),dest(address,port)) tuple it sees. Put that out of your mind. Logically we're in the context of a /connection/, represented by that tuple. Within that connection multiple commands could be sent, or not: don't hang up prematurely. However: they might say "goodbye" or not. If the other end of the connection decides to hang up, there should be some notification to your process and your read should terminate with some special condition (read coming back empty, "end of file" flag, ...) or exception; if that happens, assume they're not coming back, no need to say "goodbye" yourself. It's never a good idea to assume that the network is reliable. ;-) (I'm sure you've double and triple-checked that you're not leaving something unread or unwritten, and flushing all output buffers if that's what it takes.) -- Fred Morris
Re: Policy Server Development
post...@ptld.com: > On 04-16-2021 1:04 pm, Wietse Venema wrote: > > As Viktor noted, each smtpd(8) process makes its own connection to > > a policy service. Then, an smtpd(8) process will reuse its own > > policy service connection, not a connection that belongs to a > > different smtpd(8) process. > > Okay, if im understanding, the expected behavior is it is supposed to be > one new spawn for each client connection/event? So the answer im looking No that would be inefficient. We are not comunicating. Wietse
Re: Policy Server Development
On 04-16-2021 1:28 pm, Viktor Dukhovni wrote: Of course. Since it is spawned for a single connection, once that connection is closed, it couldn't possibly get any further requests. This is not the same as handling one request, you still need a loop to handle one or requests until EOF. I will go that route, trying to detect smtpd(8) disconnect. Thank you all for the clarification.
Re: Policy Server Development
On Fri, Apr 16, 2021 at 01:22:25PM -0400, post...@ptld.com wrote: > On 04-16-2021 1:04 pm, Wietse Venema wrote: > > As Viktor noted, each smtpd(8) process makes its own connection to > > a policy service. Then, an smtpd(8) process will reuse its own > > policy service connection, not a connection that belongs to a > > different smtpd(8) process. > > Okay, if im understanding, the expected behavior is it is supposed to be > one new spawn for each client connection/event? No. One per smtpd(8) process. Each smtpd(8) process may handle multiple client connections (up to $max_use). > So the answer im looking for is my script should self terminate when > it detects the client (postfix) disconnect? Of course. Since it is spawned for a single connection, once that connection is closed, it couldn't possibly get any further requests. This is not the same as handling one request, you still need a loop to handle one or requests until EOF. -- Viktor.
Re: Policy Server Development
On 04-16-2021 1:04 pm, Wietse Venema wrote: As Viktor noted, each smtpd(8) process makes its own connection to a policy service. Then, an smtpd(8) process will reuse its own policy service connection, not a connection that belongs to a different smtpd(8) process. Okay, if im understanding, the expected behavior is it is supposed to be one new spawn for each client connection/event? So the answer im looking for is my script should self terminate when it detects the client (postfix) disconnect? Is that the expected behavior, there are no other clues given by postfix to the policy server when it's no longer needed?
Re: Policy Server Development
On 04-16-2021 1:04 pm, Wietse Venema wrote: What evidence do you have that you have zombie processes, and not simply processes that are idle waiting for work? Wietse Watching ps, for each incoming email a new spfpolicy and userpolicy gets spawned. After a few minutes each spfpolicy terminates. None of the userpolicy ever terminate, the number of instances never decreases only grows. Even with 15+ userpolicy instances loaded, every new email always starts another instance. I watched it for an hour and they never terminated, only when i restarted postfix did they finally terminate.
Re: Policy Server Development
On 04-16-2021 12:43 pm, Viktor Dukhovni wrote: On Fri, Apr 16, 2021 at 11:50:12AM -0400, post...@ptld.com wrote: master.cf: userpolicy unix - n n - 0 spawn user=mail argv=/usr/libexec/postfix/per-user-policy This means one process per connection. So when there are multiple smtpd(8) processes, each one will spawn a separate policy service. Okay. But is that desirable? Is it wrong? Should i change it? Change which to what? Is it the maxproc=0? I did that per advice on SMTPD_POLICY_README. The policy service should exit once smtpd(8) disconnects. No matter what? Meaning postfix is not intended to reuse a policy server it spawned for the next incoming email? Or are you just saying the way it is currently configured that would be best to do? And if so then how should i set things up to do things the intended way? Im still not clear on what is the intended behavior in a perfect world.
Re: Policy Server Development
post...@ptld.com: > There are no errors or warnings. The script runs and works. I can see > the values (request=smtpd_access_policy, etc) saved by the script to > log. I can also run the script from console with no errors or warnings. > Everything works, only problem is script never ends and becomes zombie. > > What should i expect to happen? As Viktor noted, each smtpd(8) process makes its own connection to a policy service. Then, an smtpd(8) process will reuse its own policy service connection, not a connection that belongs to a different smtpd(8) process. Also, an smtpd(8) process will not immediately close the policy service connection. This is to avoid the overhead of creating processes and making connections for each SMTP client request. What you call zombies could be just Postfix daemons waiting for more work. What evidence do you have that you have zombie processes, and not simply processes that are idle waiting for work? Wietse
Re: Policy Server Development
There are no errors or warnings. The script runs and works. I can see the values (request=smtpd_access_policy, etc) saved by the script to log. I can also run the script from console with no errors or warnings. Everything works, only problem is script never ends and becomes zombie. What should i expect to happen? Is it intended each instance of the script should handle just one email then self terminate? Is there something in the master.cf telling postfix to start a new instance per email that i can change? Can you elaborate what you mean my script isn't doing what i think? I forgot to mention im using Postfix v3.3.1 On 04-16-2021 12:37 pm, Wietse Venema wrote: Your script is not doing what you think it does. Are there any warnings logged on the Postfix side? Wietse
Re: Policy Server Development
On Fri, Apr 16, 2021 at 11:50:12AM -0400, post...@ptld.com wrote: > master.cf: >userpolicy unix - n n - 0 spawn user=mail > argv=/usr/libexec/postfix/per-user-policy This means one process per connection. So when there are multiple smtpd(8) processes, each one will spawn a separate policy service. The policy service should exit once smtpd(8) disconnects. > My postfix setup it is not reusing connections. Every email spawns a new > instance. And since the script waits for the next request it doesn't > self terminate causing the system to fill up with zombie instances of > the script. I would be very surprised if actual zombie processes were involved. The spawn(8) services waits for its child process, and master(8) waits for its child spawn(8) processes, so if there are zombie processes, they are perhaps because your script forks a subprocess and does not collect its status in a timely manner. I really would not recommend PHP by the way. Try Python, Perl or Ruby. -- Viktor.
Re: Policy Server Development
post...@ptld.com: > As of now the policy script writes values supplied by postfix to a log > file and returns the expected action=dunno and empty line. I designed > the PHP to run in a loop with no time outs for it to be available for > multiple request. My understanding of SMTPD_POLICY_README is postfix > will reuse the policy service multiple times. > > "Unless there was an error, the server must not close the connection, so > that the same connection can be used multiple times." > Did i take this too literal? > > My postfix setup it is not reusing connections. Every email spawns a new > instance. And since the script waits for the next request it doesn't > self terminate causing the system to fill up with zombie instances of > the script. Your script is not doing what you think it does. Are there any warnings logged on the Postfix side? Wietse
Policy Server Development
I am writing a policy server in PHP. I am confused by some of postfix behavior. I designed the PHP service in this manner; /usr/libexec/postfix/per-user-policy: #!/usr/bin/php master.cf: userpolicy unix - n n - 0 spawn user=mail argv=/usr/libexec/postfix/per-user-policy main.cf: userpolicy_time_limit = 3600 smtpd_recipient_restrictions = check_policy_service unix:private/userpolicy As of now the policy script writes values supplied by postfix to a log file and returns the expected action=dunno and empty line. I designed the PHP to run in a loop with no time outs for it to be available for multiple request. My understanding of SMTPD_POLICY_README is postfix will reuse the policy service multiple times. "Unless there was an error, the server must not close the connection, so that the same connection can be used multiple times." Did i take this too literal? My postfix setup it is not reusing connections. Every email spawns a new instance. And since the script waits for the next request it doesn't self terminate causing the system to fill up with zombie instances of the script. I thought the behavior should be postfix spawns a policy service, uses that same connection for multiple emails until it hits the limit, kills the policy script and spawns a new one as a type of memory management. I tried having my script listen for any SIG??? from postfix to shut it down gracefully, but i never detected any. What is the correct way? Is there something wrong in my config causing a new spawn per email or is that how it is supposed to work? Is postfix supposed to terminate the script? Or what is the method im supposed to use so my script knows when to self terminate? Or should the script be designed as a one time use and always terminate after sending an action= ?
