Listen directive, port specification
Hi, So the Listen directive now takes an optional protocol argument. In the documentation it states that if the protocol isn't specified then if the port number is 443, it is set to "https" otherwise "http". This seems like a good idea. What about if the port number is 25, personally I want this to automatically set the protocol to "smtp". Sort of like our own /etc/services. I can of course see this turning into default protocol specifiers for multiple port numbers so I anticipate some type of config file similar to MIMEMagic in the future. But for now, where is the code that set's the protocol to "https" if the port number is 443? I don't see it in the Listen directive handler. Also is there a preferred way for a protocol module to accept a connection? Right now I have a directive "SmtpProtocol" that dictates whether mod_smtpd accepts a connection on a server_rec. There is also ap_get_server_protocol(server_rec *s) that returns the protocol based on the Listen directive. The second way is similar to the way handlers are selected. Rian
Re: prefork mpm in linux: ap_process_connection isn't called on connection
On Mar 31, 2006, at 6:42 AM, Jeff Trawick wrote: On 3/28/06, Rian A Hunter <[EMAIL PROTECTED]> wrote: On the httpd trunk in prefork.c ap_process_connection isn't called until there is data on the new connection (instead of just when a client connects). Is there a kernel accept filter enabled? No, it is a default clean install. Are you suggesting that for me to get the behavior I want that there should be one enabled? Rian
mod_smtpd documentation location
Hi Everyone, I have written up some preliminary documentation (not done yet) that describes how to write a module for mod_smtpd. It is based on the format of the manualpage dtd used in the Apache Documentation Project. I'm also planning on writing a modulesynopsis document for users of mod_smtpd. Where should these files go? Should I add a "doc" subdirectory to mod_smtpd or what? I want it to be easily build-able and available online, etc. Rian
Re: [mod_smtpd] patch need for the SIZE extension
On Feb 8, 2006, at 11:22 PM, Brian J. France wrote: On Feb 8, 2006, at 9:28 PM, Rian Hunter wrote: On Feb 8, 2006, at 10:56 AM, Brian J. France wrote: +1, I like the idea of the storing the setting in the per connection instance. Leaving the default in the core and copying it to an per connection struct would allow modules to tweak the setting per connection. One other addition to the smtpd_session_rec I would like to see would be something like r->notes or r->subprocess_env. Some way for modules to set flags that would change the behavior of another module. As an example I might want to add a new modules that changes the max data size based on either connection ip or mail from address. This module would want to set a flag telling the size module to not show the numerical size limit in the ehlo response as it might change based on the the mail from address. Something like r->notes does exist, check scr->session_config. session_config is a ap_conf_vector_t, which I think is more like r- >per_dir_config or r->request_config. r->notes and r->subprocess_env is a apr_table_t. +1, I see what you mean. Rian
Re: [mod_smtpd] patch need for the SIZE extension
On Feb 8, 2006, at 10:56 AM, Brian J. France wrote: +1, I like the idea of the storing the setting in the per connection instance. Leaving the default in the core and copying it to an per connection struct would allow modules to tweak the setting per connection. One other addition to the smtpd_session_rec I would like to see would be something like r->notes or r->subprocess_env. Some way for modules to set flags that would change the behavior of another module. As an example I might want to add a new modules that changes the max data size based on either connection ip or mail from address. This module would want to set a flag telling the size module to not show the numerical size limit in the ehlo response as it might change based on the the mail from address. Something like r->notes does exist, check scr->session_config. +1, if I understand you currently. Changing scr->should_disconnect to scr->allows_commands which is a bit-field of what commands that are allowed. Then smtp_protocol.c handles denying command instead of having a module hook every command and doing it, right? Yeah exactly right! Rian
Re: [mod_smtpd] patch need for the SIZE extension
On Feb 7, 2006, at 8:15 PM, Brian J. France wrote: Hi Rian, Before I started converting my other modules to the new code I figured I would start with writing a new module to handle the SIZE extension. I needed to apply the following patch (link below) to the mod_smtpd code to get access at the max data size. +0, Well this is an interesting point. The max data size should be settable and gettable from extending modules. In a way max_data is a lot like the list of extensions: should it be set on every connection or upon server initialization (ie should it's scope be per connection or per mod_smtpd instance). I chose that the list of extensions might want to be per-connection since modules may not want to offer all clients all extensions, just the same way modules may want to enforce different max_data sizes for different clients. Where do you think these variables belong? One thing I do know is that max_data doesn't belong in mod_smtpd's configuration structure and rather either in some other "per mod_smtpd instance" structure accessible to extending modules or just in scr (the per connection structure). Right? I think that would be cleaner and more modular. We should talk about this some more before I apply this patch. I hooked the mail from hook, check for a valid SIZE in mail_parameters and check to make sure it is not over the limit. If it is over the limit I can use smtpd_respond_oneline to send the 552 "message exceeds fixed maximum message size" line back to the client, but what should the function return to make it force a QUIT or REST command as anything but SMTPD_DONE sends more stuff to the client. Should I just return SMTPD_DONE and set scr->should_disconnect? Could we tweak it to support two different settings, one would only allow QUIT only and the other would allow QUIT and REST (to start over). +1, Since quit and rset are both handled by mod_smtpd there should probably be another variable called scr->only_quit_or_rset, also because I think there are other times when the client should only issue a quit or rset. More discussion follows though: My basic design strategy has been that mod_smtpd should do as little as possible or what will be practical to a large amount of extending modules. If mod_smtpd sets a variable in a structure it should use it. If an extending module sets a variable in a structure it should use it. Having an "only_quit_or_rset" variable in the scr structure with mod_smtpd consciously checking for it, but never changing the value itself sort of violates that strategy. That doesn't mean I'm against it because like I already mentioned I think a lot of modules will need this sort of thing. Have you thought about hooking into all the commands, and then sending a "503 Only QUIT and RSET" from your SIZE module until a rset is received? I only added the scr->should_disconnect logic since it's what should happen when a simple module wants to SMTPD_DENY some connections. I didn't want every module that wanted to implement connection policy to have to deal with being hooked into all the commands, and just be able to return SMTPD_DENY. The scr->should_disconnect situation I just explained (where I didn't want every modules to have to deal with hooking all the commands for a redundant task) could be applied to any potential module that wants to disallow any of the built in commands with instead a bit-field that specifies which mod_smtpd core commands are currently allowed. I think I like this idea the best, what do you think? Rian Hunter
Re: svn commit: r374754 - in /httpd/site/trunk: docs/ docs/mod_smtpd/ docs/modules/ xdocs/ xdocs/mod_smtpd/ xdocs/modules/
On Feb 3, 2006, at 5:36 PM, Paul Querna wrote: Garrett Rooney wrote: On 2/3/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: +mod_smtpd started it's life as a 2005 Google Summer of Code project +taken on by Rian Hunter and Jem Berkes with +mentors Nick Kew and Paul Quernastrong>. It continues +its life being developed and maintained by Rian Hunter with help from the httpd +developers. Ok, now we're seeing this both in the mod_mbox and mod_smtpd web pages. This is supposed to be a community project, ASF projects don't have lead developers, and they don't generally go out of their way to assign credit like this, that's normally kept to the CHANGES file and the svn commit logs. Putting this sort of thing on the web pages seems quite inappropriate to me. +1. It is perhaps notable that it was _started_ as a SoC project, but the rest of it can most likely be nuked. We do have some similar language under http://httpd.apache.org/ABOUT_APACHE.html too. -Paul +1 I used mod_mbox as the model for the mod_smtpd page. Excuse my ignorance. Rian
Re: Bouncing messages in mod_smtpd
On Feb 1, 2006, at 7:51 PM, Garrett Rooney wrote: Well, we already have a small SMTP implementation in the SMTP queue module, that could be made somewhat more generic and used for this sort of thing. Yeah I was thinking about that! I'm just so fearful of code-bloat and possibly maintaining an SMTP client but I may just end up doing that. Rian Hunter
Bouncing messages in mod_smtpd
Hey! January has brought in some great improvements to mod_smtpd. If you compare the code now to the code six months ago you'll see that it is very clean and very workable now, not that it wasn't before it's just that now some distinctions have been made that makes thinking about extending mod_smtpd a lot easier. I expect plugin writers to bring up some points where mod_smtpd will have to be extended, and those extensions will happen when the time is right. Anyway, I'm facing one issue. When an SMTP envelope has more than one recipient and the queue plugins are only able to queue some of them mod_smtpd has the responsibility to bounce the unqueued messages back to the sender. Of course in a proper set up this shouldn't happen because mod_smtpd won't give a "250 Ok" response to the "RCPT" command if it doesn't support the recipient, but there might be cases where a plugin is unable to queue for other unexpected reasons. mod_smtpd needs a bouncing mechanism! I need some help with this because I am not sure how to approach this. Should I implement an entire SMTP client in mod_smtpd to bounce messages? Should I relegate this responsibility to a sendmail command or what platform specific mailer there is? Are there other options? I want to dedicate the rest of this week to writing some drafts of documentation and a website for mod_smtpd, so if someone would like to volunteer to mess with bouncing that would be a great help! The code to look at is in http://svn.apache.org/repos/asf/httpd/mod_smtpd/trunk/src/ smtp_protocol.c, the function is smtpd_bounce_unqueued (smtpd_session_rec *). Thanks!! Rian Hunter
Re: Best Available Version?
On Jan 10, 2006, at 11:57 PM, Paul Querna wrote: Rian Hunter wrote: "This version of Apache is principally a security and bug fix release, and represents the best available version of Apache HTTP Server." That string is currently under the "Apache 2.0.55 Released" section of http://httpd.apache.org/. With the release of Apache 2.2.0, is this still correct? No, its not. Feel free to commit a fix. -Paul I don't think I have commit access to the site repository. Here's a patch anyway. Rian Hunter bestversion.patch Description: Binary data
Header Handling in mod_smtpd
Hi, I just wanted to poll dev on how mod_smtpd should handle mail headers after receiving the data command. Currently mod_smtpd parsers headers if there are any, then when that is done sends the body to the message buffering loop. A message usually looks like this: ---begin_rfc822_message--- From: Rian Hunter <[EMAIL PROTECTED]> To: Rian Hunter <[EMAIL PROTECTED]> Subject: Testing. Hi this is the body of the message. ---end--- mod_smtpd saves: ---begin body--- Hi this is the body of the message. ---end--- when normal mail clients usually show the body like this: ---begin body--- Hi This is the body of the message. ---end--- Does it seem like a good idea to save the extra line-break as the beginning of the body or remove it before letting the plugins deal with the body. Keep in mind that the reason it is like this now is to keep the loop simple so it can also handle header-less messages (the parser eats the bad data). Thanks! Rian Hunter
Best Available Version?
"This version of Apache is principally a security and bug fix release, and represents the best available version of Apache HTTP Server." That string is currently under the "Apache 2.0.55 Released" section of http://httpd.apache.org/. With the release of Apache 2.2.0, is this still correct? Rian Hunter
Re: mod_smtpd changes
On Sat, 2005-12-31 at 13:47, Brian J. France wrote: > On Dec 31, 2005, at 1:23 PM, Rian Hunter wrote: > > Any comments, ideas and criticisms are highly welcomed! Thanks! > > Any changes on how recipients and queue/deliver is handled? > > I started implementing my proposal from a while back and got stuck on > how to handle errors. If a transaction has multiple valid recipients > and each one is handle by different queue modules, but only one of > queue modules returned a failed, is the whole thing a failure or > should the other messages be delivered and what should be returned > for a status for the DATA command? > > I am thinking that if all recipients are valid and can be handle by a > queue module then DATA will always return OK/200 what ever success > is. If a queue module then finds an error happens while trying to > really deliver the message, a bounce need to be generated. This > means we should have a common way to generate a bounce message > instead of having every module come up with its own. I've been thinking about this a lot and what tradeoff's to make in regard to handling the envelope. I think the main envelope of mod_smtpd will only support one recipient (= each queue module will handle one recipient at a time). What I was thinking is to make smtpd_run_queue a RUN_FIRST hook, so it will stop whenever a module returns SMTPD_ACCEPT (more about this below). A default queue will be installed that is inserted absolutely last, where if it is called it will bounce the message (because all the rest SMTPD_DECLINED). But this doesn't work in a setting where one address can be handled by multiple modules, so instead smtpd_run_queue would be a RUN_ALL hook and modules will set a flag in the envelope structure if the message was delivered correctly, then mod_smtpd checks if it was or wasnt' then sends a bounce message for the addresses that weren't. How should a queue module know if it should a certain recipient address? There are two options: * Have each queue module maintain its own list or criteria for which it will queue a message. * Let mod_smtpd be notified what queue modules should handle which recipient addresses. Then it sets in each recipients envelope structure a user defined string for a "queue handler." Each queue module checks the queue handler string to see if it handles that envelope. This is not the best either because it only allows for one queuer per recipient. The first option follows the needs of mod_smtpd but at the same time I really like the idea of a standardized of way of specifying which modules get what addresses. This simplifies configuration, eliminates code bloat and uses less cpu cycles. I'm still thinking about the best way to implement something like that. Rian Hunter
mod_smtpd changes
Hi, I'm planning on revamping mod_smtpd and resolving some problems with it that kind of put it at a dead-lock because of many unelegant solutions to a hard problem. First I plan on removing all the unnecessary hook code, and adjust mod_smtpd to use regular (apr?) hooks. That will really simplify things and increase build portability. Also I want to fix the folder hierarchy and put all the src in one folder and the Makefiles in others, so it isn't so UNIX-centric. There are some parts where it's hard for mod_smtpd to conform to the way httpd does input filters, I'm going to iron that out. Also the envelope abstraction needs some working on so it can extended by other plugins and other miscellaneous things. Basically I want to get mod_smtpd to a stated where it is comfortable for plugin writers to develop for and sysadmins to use. Practicality is goal #1, whereas before it was extensibility. Any comments, ideas and criticisms are highly welcomed! Thanks! Rian Hunter
Re: autotools woe
On Thu, 2005-12-29 at 10:05, Philip M. Gollucci wrote: > Can you do : > > cd /var/db/pkg > ls -ld gnu-* Nothing in there starts with gnu- > and > > env | grep -i path > On Thu, 2005-12-29 at 10:05, Philip M. Gollucci wrote: > Can you do : > > cd /var/db/pkg > ls -ld gnu-* nothing in there s > and > > env | grep -i path > On Thu, 2005-12-29 at 10:05, Philip M. Gollucci wrote: > Can you do : > > cd /var/db/pkg > ls -ld gnu-* nothing in there s > and > > env | grep -i path > PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/home/rian/bin Thanks! rian
Re: mod_smtpd_access_dbd code review, mod_smtpd_auth plan
On Sep 14, 2005, at 9:33 AM, Brian J. France wrote: http://www.brianfrance.com/software/apache/mod_smtpd_access_dbd.tar.gz I ported this from mod_dbi_pool to mod_dbd, but I haven't had time to go back in and clean up the code. I liked the layout when it was using mod_dbi_pool, but I just hacked it up to convert it to mod_dbd. Since I haven't had the time to clean it up so I figured I would get some comments on it to make sure I am going in the right direction. I have a another trip next week and I plan on using the time to clean it up and also start working on the mod_smtpd_auth* modules. My plan right now is to build a skeleton mod_smtpd_auth module that will handle plain, login, cram-md5 (and possibly digest-md5). mod_smtpd_auth will define two hooks for other auth modules, one that takes username and password for authorizing and one for which methods it enables. mod_smtpd_auth_pwd will enable plain and login because it only has the encrypted password mod_smtpd_auth_dbd will enable all auth methods Thoughts/comments? Brian I will review your modules (including mod_smtpd_load, which i think should now be named mod_smtpd_connect_load) very soon for incorporation into the modules tree. Rian Hunter
Re: svn commit: r232338 [1/2] - in /httpd/mod_smtpd/trunk: ./ Makefile.in README autogen.sh configure configure.ac mod_smtpd.h smtp.h smtp_core.c smtp_protocol.c
On Sep 18, 2005, at 3:30 PM, Paul Querna wrote: Rian Hunter wrote: On Sep 17, 2005, at 9:50 PM, Paul Querna wrote: Sorry for bring this up so late [ snip ] Why are these APR Optional Functions? Why can't they be AP_DECLARE_HOOK's? -Paul I decided that they should be optional because after talking to some people I was under the impression that only modules included in the core httpd distribution should use AP_DECLARE_HOOK and all others should be optional hooks because they might not always be loaded. If you are running a SMTP server, mod_smtpd will always be loaded, and if its not, it should be treated as a fatal error The optional functions will make it silently ignore any missing symbols, which isn't a good thing. I have no problem changing it, but if I do I'll have to change all the currently checked in mod_smtpd plugins. Are you 100% this is the right thing to do? -rian
Re: mod_smtpd and old versions of apache2
On Sep 17, 2005, at 1:27 PM, Garrett Rooney wrote: So I'm playing around with mod_smtpd_queue_smtp, a module that has mod_smtpd "queue" messages by forwarding them on to another smtp server, and I ran into a little snag. It appears that in older versions of apache2 (2.0.53 in this case) conn->input_filters->ctx is NULL when smtpd_create_conn_rec gets called. This results in a crash when mod_smtpd tries to poke around in there looking for the client socket, so it can set a big-ass timeout on it. Is it even necessary to do this? Making the timeout conditional on the context being non-null seems to fix the problem, and things seem to work ok, but I have to wonder why it's needed in the first place... -garrett Thanks for bringing this up. The timeout setting code is an archaism from when mod_smtpd used the ap_rgetline functions for I/O. ap_rgetline relied on a timeout set by one of the filters in http_filters.c to set a timeout, but since those filters weren't used i had to do it manually in mod_smtpd but now it has it's own I/O functions so it doesn't need to set this. -rian
Re: svn commit: r289845 - in /httpd/mod_smtpd/trunk/modules/queue: ./ mod_smtpd_queue_postfix/ mod_smtpd_queue_postfix/mod_smtpd_queue_postfix.c
On Sep 17, 2005, at 8:35 PM, Garrett Rooney wrote: On 9/17/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: +static int cleanup_socket; This global worries me. How will it behave in a multithreaded MPM? -garrett Right, I'll fix that up. -rian
Re: svn commit: r232338 [1/2] - in /httpd/mod_smtpd/trunk: ./ Makefile.in README autogen.sh configure configure.ac mod_smtpd.h smtp.h smtp_core.c smtp_protocol.c
On Sep 17, 2005, at 9:50 PM, Paul Querna wrote: Sorry for bring this up so late [ snip ] Why are these APR Optional Functions? Why can't they be AP_DECLARE_HOOK's? -Paul I decided that they should be optional because after talking to some people I was under the impression that only modules included in the core httpd distribution should use AP_DECLARE_HOOK and all others should be optional hooks because they might not always be loaded. If this is not the case, this should change soon. -rian
to the users of mod_smtpd
Hi, To Plugin Writers: Mod_smtpd is pretty heavily input filters based now. There were some sneaky bugs in the code that I got rid of (the ability to type a single '.' and not terminate the message) and some major cleanups in the data handler code. The biggest changes are that now there is a filter that translates a single '.' into an EOS bucket, and an '..' into an '.'. Also header parsing now goes into a filter too. This is all pretty sweet stuff. To Users: Stay tuned for a postfix plugin and an auto-built modules directory. Housecleaning: Mod_smtpd has been written as high-level as possible so as not to reinvent the wheel and deal with io security issues. Regardless it is in dire need of a code/security audit. Does anyone know of any protocol-level security tests or topics to keep in mind that can be used to "robustify" mod_smtpd. Showstoppers: The autobuild system still currently does not check for libapreq2 (a necessity of one of the filters). Since I hate Autoconf and Makefiles I've been punting this, even though it's high priority. Patches anyone? Mod_smtpd is also in dire need of documentation and probably its own website, both are coming soon. Thanks! Rian Hunter
Re: mod_smtpd module review
On Aug 30, 2005, at 11:19 AM, Brian J. France wrote:
This past week I have finished up a few modules and ready for review.
http://www.brianfrance.com/software/apache/mod_smtpd_load.tar.gz
mod_smtpd_load:
This module allows rejecting connection (temporarily) based on
server load
It is not very cross platform (any os with getloadavg), but I am
sure we
can work on that.
I have finished mod_smtpd_access_dbi, but after talking with Paul
on IRC
I need to convert it to use mod_dbd instead of mod_dbi_pool.
Working on that
now and will post another message when done. I could build a tar
ball of this
module if anybody is interested as the the flow will not change,
just how the
db connection is handled.
mod_smtpd_access_dbi:
This module is similar to sendmails access file. It allows
checking of the ip/hostname/from/to items via a database to see if
things should be rejected. It uses mod_dbi_pool and libdbi.
Thanks to Paul for mod_dbi_pool and code examples from
mod_vhost_dbi.
This is great!
A few things I have found while writing these modules. How about
changing smtpd_return_data from this:
typedef struct smtpd_return_data {
apr_pool_t *p;
/* list of messages */
apr_array_header_t *msgs;
} smtpd_return_data;
to something like this:
typedef struct smtpd_return_data {
apr_pool_t *p;
int code;
/* list of messages */
apr_array_header_t *user_msgs;
apr_array_header_t *log_msgs;
} smtpd_return_data;
While doing the mod_smtpd_load module I found when I want to deny a
connection I can set what message the user will get, but I also
want to log a different message instead of the default "Connection
Denied" (current I log my own and the default gets logged). Of
course this might be another thread of how and what do we plan on
logging.
IMO I think that logging twice is fine. mod_smtpd logs "Connection
Denied" because a server admin may want to know what connections are
getting denied for debugging and etc. and I feel its absolutely
necessary, same with the MAIL command. If your module wants to log
extra information, I think that's OK.
Maybe for clarity all mod_smtpd's logs can be prefixed with
"mod_smtpd:" and other modules can follow suit.
I would also like to set the error code, because looking over
rfc0821 I think it should return 452 or may be that needs to be a
default for smtpd_run_connect soft errors (552 for hard errors).
Should we allow the module to set the error code?
Error codes can be sent with SMTPD_DONE. Otherwise error codes won't
change in the next fifty years for things like SMTPD_DENY and
friends. I'll look into changing those specific codes. Most of the
error codes I got from Postfix and Qpsmtpd but of course they aren't
perfect.
In mod_smtpd_access_dbi I found it strange that I get a string that
looks like this: " <[EMAIL PROTECTED]>" instead of
"[EMAIL PROTECTED]". For the mail/rcpt hooks should we send a struct
that has the full line sent, the data from the full line and the
parsed email address? I have some code that duplicates the string
and then remove spaces and < from the beginning and > from the end,
but that seems like it should be done before my function is
called. Another problem I can see is when we get into things like
the size options:
MAIL FROM:<[EMAIL PROTECTED]> SIZE=50
Do we want every module to have to parse the email address
(removing < >) and the SIZE?
This is a very good point. Honestly I'm not really sure since I'm not
experienced with writing a plugin based architecture, but I'm almost
positive that most people will say that mod_smtpd should parse the
email address and options along with it. When I first wrote mod_smtpd
I figured I'd pass the un-parsed string after from: verbatim to let
the modules decide what to do with it. Maybe to allow some crazy
things. Either way, having mod_smtpd parse sounds like the right idea
so how about this hook for mail:
smtp_run_mail(smtp_conn_rec *, smtp_return_data *, char
*parsed_address, apr_array_header_t *options);
The options won't be stored as name->value pairs: more like an array
of strings like "SIZE=5". Does that sound good?
Rian Hunter
Re: mod_smtpd overhaul
On Tue, 2005-08-23 at 11:04, Brian J. France wrote: > On Aug 23, 2005, at 12:18 AM, Jem Berkes wrote: > > I noticed a couple posts about examples, there is now one as I have > > committed all the RBL stuff I wrote. See: > > > > https://svn.apache.org/repos/asf/httpd/mod_smtpd/trunk/mod_smtpd_rbl/ > > > > This hooks into mod_smtpd in two places and returns various data (e.g. > > if > > the client IP is blacklisted then mod_smtpd is told to deny mail). I > > hope > > it serves as a good example, it seems to work quite nicely to give > > mod_smtpd all the DNSBL/RHSBL features in a modular fashion. > > Now that we have the means to start writing modules for smtpd should we > create a modules directory? Something like: > > /repos/asf/httpd/mod_smtpd/trunk/modules/access > mod_smtpd_rbl : "RBL" (DNSBL/RHSBL) support to mod_smtpd > mod_smtp_access_db : Add sendmail access.db type support > mod_smtp_access_dbi : Add sendmail access.db type support via SQL > > /repos/asf/httpd/mod_smtpd/trunk/modules/auth > mod_smtpd_auth_plain > mod_smtpd_auth_login > mod_smtpd_auth_cram_md5 > mod_smtpd_auth_digest_md5 > > or should this be a skeleton auth modules that needs plugins: > > mod_smtpd_auth : supports plain, login, cram_md5, digest_md5 > mod_smtpd_auth_pwd : add /etc/passwd auth support for plain and login > mod_smtpd_auth_db : add db auth support for all auth > mod_smtpd_auth_dbi : add SQL auth support for all auth > > /repos/asf/httpd/mod_smtpd/trunk/modules/misc > mod_smtpd_load : Allow temp error if load to high on the server +1, Jem since you have checked in the first plugin for mod_smtpd would you mind creating a directory structure similar to this if it seems fine to you? -rian
mod_smtpd hook system
I've changed the hook system for mod_smtpd a little to allow more flexibility and module cooperation. Basically it works like this now: When a hook is called smtpd_run_*, each registered hook is called until one returns something other than SMTPD_DECLINED or SMTPD_OK. If all the hooks return SMTPD_DECLINED, then the call to smtpd_run_* returns SMTPD_DECLINED. If all hooks return SMTPD_DECLINED but one or more return SMTPD_OK, then the call to smtpd_run_* returns SMTPD_OK. This is different from the conventional "run all" hook we have where the call to smtpd_run_* returns SMTPD_OK even if it was declined by all the hooks. This is also different from a "run first" hook, where calling hooks stop as soon as one returns something other than declined. The purpose of this change was to allow the call to smtpd_run_queue to call all the hooks even if one already accepted it with SMTPD_OK (so other modules can process other recipients), and to know that at least one accepted the message. If they all returned SMTPD_DECLINED then tell the client that the message was not queued. Of course the other SMTPD_DENY* return codes still apply. This required a new run implementor macro that I've dubbed APR_IMPLEMENT_OPTIONAL_HOOK_RUN_ALL_MOD and defined in smtpd.h, but would something like this make it into the APR release? It seems silly and hackish within smtpd.h. Also should a APR_IMPLEMENT_OPTIONAL_HOOK_RUN_FIRST also be defined in APR? Thanks. -rian
Re: mod_smtpd overhaul
On Tue, 2005-08-23 at 11:04, Brian J. France wrote:
> Is there a way to add more options to the EHLO response (an example)?
Sure. Extensions are registered on a per-connection basis. Ideally you
would register an SMTP extension at the connect hook like so:
---
smtpd_retcode default_connect(smtpd_conn_rec *scr,
smtpd_return_data *out) {
smtpd_register_extension(scr, "MY EXTENSION");
return SMTPD_OK;
}
register_hooks() {
APR_OPTIONAL_HOOK(smtpd, connect,
default_connect, NULL, NULL,
APR_HOOK_MIDDLE);
}
-
You do it at connect so when the user gives ehlo, the server can respond
with the list of all the extensions supported by the various modules.
There will be more documentation/example code for mod_smtpd very soon.
-rian
Re: svn commit: r235759 - in /httpd/mod_smtpd/trunk: mod_smtpd.h smtp.h smtp_core.c smtp_protocol.c
On Mon, 2005-08-22 at 23:37, Garrett Rooney wrote:
> > Modified: httpd/mod_smtpd/trunk/mod_smtpd.h
> > URL:
> > http://svn.apache.org/viewcvs/httpd/mod_smtpd/trunk/mod_smtpd.h?rev=235759&r1=235758&r2=235759&view=diff
> > ==
> > --- httpd/mod_smtpd/trunk/mod_smtpd.h (original)
> > +++ httpd/mod_smtpd/trunk/mod_smtpd.h Mon Aug 22 10:22:27 2005
> > @@ -63,7 +63,7 @@
> > SMTPD_STATE_GOT_HELO,
> > SMTPD_STATE_GOT_MAIL,
> > SMTPD_STATE_GOT_RCPT
> > - } smtpd_request_state;
> > + } smtpd_trans_state;
> >
> >typedef enum {
> > SMTPD_PROTOCOL_SMTP,
> > @@ -72,78 +72,90 @@
> >
> >typedef struct smtpd_return_data {
> > apr_pool_t *p;
> > -char *msg;
> > +/* list of messages
> > + null terminated */
> > +char **msgs;
>
> Why is this a char **? It seems kind of error prone to require the
> module programmer to allocate and manage that array correctly, this
> feels like another case where an APR array would potentially simplify
> things.
Good point. I'll do this. Originally there was a reason why I didn't
want it to be a APR array, but that reason doesn't apply anymore.
> >} smtpd_return_data;
> >
> > - typedef struct smtpd_request_rec {
> > + typedef struct smtpd_trans_rec {
> > apr_pool_t *p;
> > -conn_rec *c;
> > -server_rec *s;
> > -
> > -/* spooled data file pointer */
> > -apr_file_t *tfp;
> >
> > /* where are we in the current transaction */
> > -smtpd_request_state state_vector;
> > +smtpd_trans_state state_vector;
>
> Out of curiosity, why is this called a "state_vector", there's only one
> element, so it doesn't feel much like a vector to me (at least not in
> the std::vector sense of the word from c++).
Well this is another thing from old code. Originally the state was a bit
vector, but now it's just incremental values so the name should
certainly change.
> > +apr_status_t
> > +smtpd_getline(smtpd_conn_rec *scr, char *data, apr_size_t dlen)
> > +{
> > + apr_status_t rc;
> > + apr_bucket *e;
> > + const char *str;
> > + char *pos, *last_char = data;
> > + apr_size_t len, bytes_handled = 0;
> > +
> > + while (1) {
> > +rc = ap_get_brigade(scr->c->input_filters, scr->bb_in, AP_MODE_GETLINE,
> > + APR_BLOCK_READ, 0);
> > +if (rc != APR_SUCCESS) return rc;
>
> Putting the body of the if statement on the same line as the if is kind
> of sketchy IMO. In particular it means that when you're single stepping
> through the code in a debugger it's impossible to easily tell if the
> statement is true or not based on the fact that you stepped onto that line.
>
> Also, APR_SUCCESS is defined to be zero, so you can write that in less
> characters by saying:
>
> if (rc)
>return rc;
>
> Although that's just a style thing.
I'll make a seperate change for this.
> > +while(!APR_BRIGADE_EMPTY(scr->bb_in)) {
> > + e = APR_BRIGADE_FIRST(scr->bb_in);
> > +
> > + rc = apr_bucket_read(e, &str, &len, APR_BLOCK_READ);
> > + if (rc != APR_SUCCESS) return rc;
> > + apr_bucket_delete(e);
> > +
> > + if (len == 0) continue;
> > +
> > + /* Would this overrun our buffer? If so, we'll die. */
> > + if (dlen < bytes_handled + len) {
> > + if (data) {
> > + /* ensure this string is NUL terminated */
> > + if (bytes_handled > 0) {
> > + data[bytes_handled-1] = '\0';
> > + }
> > + else {
> > + data[0] = '\0';
> > + }
> > + }
> > + return APR_ENOSPC;
> > + }
>
> The indentation in this section seems kind of screwed up. Speaking of
> indentation, none of the mod_smtpd code follows the httpd style
> guidelines, if that's going to eventually change it might be good to get
> it out of the way sooner rather than later, as those kind of changes
> tend to obscure the revision history of the code. See
> http://httpd.apache.org/dev/styleguide.html for details about the style
> guidelines.
I'll make a completely independent commit for style problems.
>
> > - apr_socket_t *csd=((core_net_rec *)r->input_filters->ctx)->client_socket;
> > + // r->request_config = ap_create_request_config(r->pool);
>
> C++ style comment. This will break some C compilers. Plus, it's an
> example of leaving old code commented out in a function, which is a bad
> habit to get into. If the code has a reason to be there, at least leave
> a comment as to why, if not, just kill it.
>
> > - r = smtpd_create_request(c);
> > - ap_update_child_status(r->connection->sbh, SERVER_BUSY_KEEPALIVE, r);
> > + scr = smtpd_create_conn_rec(c);
> > + // ap_update_child_status(scr->c->sbh, SERVER_BUSY_KEEPALIVE, r);
>
> Another case of that...
I left that code in there just in case others knew how to change emulate
the same behavior. It was more like a note, but I should have definitely
commented on why it was there.
>
> -garrett
Tha
Re: mod_smtpd overhaul
On Mon, 2005-08-22 at 23:17, Brian J. France wrote: > Trying to build build a smtpd module, but having a problem. I added a > line like this: > >smtpd_hook_connect( smtpd_access_dbi_connect, NULL, NULL, > APR_HOOK_MIDDLE ); > > in the register_hooks function, but when I load the module I get > undefined function _smtpd_hook_connect. > > Is this the right way or is there an example module I could compare > with? Actually no, you should use: APR_OPTIONAL_HOOK(smtpd, connect, smtpd_access_dbi_connect, NULL, NULL, APR_HOOK_MIDDLE); Since it's an optional hook. Thanks. -rian
mod_smtpd overhaul
Hi, I just have committed mod_smtpd with changes that free it from request_rec. There are now two major structures in mod_smtpd: smtpd_conn_rec, and smtpd_trans_rec. These are distinguished by how often these get cleared. Also mod_smtpd no longer uses the ap_getline and ap_rprintf functions for I/O. There are three new functions for I/O called smtpd_getline, smtpd_respond_oneline and smtpd_respond_multiline. This should allow the use of filters in mod_smtpd although I may have implemented this wrong. Comments are requested. Joe: For the data command I still call ap_get_brigade with AP_MODE_GETLINE, just because SMTP is a line oriented protocol was there any explicit reason you wanted it to be called with AP_MODE_READBYTES? You can check it out from https://svn.apache.org/repos/asf/httpd/mod_smtpd/trunk/. -rian
mod_smtpd overhaul
Hi, I just have committed mod_smtpd with changes that free it from request_rec. There are now two major structures in mod_smtpd: smtpd_conn_rec, and smtpd_trans_rec. These are distinguished by how often these get cleared. Also mod_smtpd no longer uses the ap_getline and ap_rprintf functions for I/O. There are three new functions for I/O called smtpd_getline, smtpd_respond_oneline and smtpd_respond_multiline. This should allow the use of filters in mod_smtpd although I may have implemented this wrong. Comments are requested. Joe: For the data command I still call ap_get_brigade with AP_MODE_GETLINE, just because SMTP is a line oriented protocol was there any explicit reason you wanted it to be called with AP_MODE_READBYTES? You can check it out from https://svn.apache.org/repos/asf/httpd/mod_smtpd/trunk/. -rian
Re: Supporting RBL in mod_smtpd
On Aug 19, 2005, at 9:58 AM, Brian J. France wrote: On Aug 18, 2005, at 6:22 PM, Rian Hunter wrote: Don't do this just yet, mod_smtpd is changing completely! completely = structures/io. I should commit my changes very soon so you can start working on this. Any ETA on this? I will be on a 4 hour flight on Monday and was going to download svn head to play with and try to start building some modules. Hopefully later today I should have this completely done and checked in. -rian
Re: Supporting RBL in mod_smtpd
On Aug 18, 2005, at 7:11 PM, Jem Berkes wrote: Here is my current plan for introducing the RBL support in mod_smtpd, using the existing mod_dnsbl_lookup which I posted earlier. This way of accomplishing the RBL support should not require any code modification to mod_smtpd itself. Nick and Rian, let me know if I should be going about this a different way? I thought the most modular fashion would be to create a mod_smtpd_rbl that registers the following mod_smtpd hooks: smtpd_run_connect (might deny service to connecting IP, per request_rec) smtpd_run_mail (might deny service to this envelope domain, per loc) +1 These would query whitelists and blacklists, whatever is available. I don't mind whipping up this bridging mod_smtpd_rbl module, but if it seems excessive to introduce a new module for this purpose then the other way of doing this would be to add the RBL supporting code into mod_smtpd itself. Don't do this just yet, mod_smtpd is changing completely! completely = structures/io. I should commit my changes very soon so you can start working on this. Either way it's done, RBLs are still not required and mod_dnsbl_lookup does not have to be present for mod_smtpd to function normally. However, adding a new bridging module has the advantage of leaving mod_smtpd code alone and taking advantage of the hooks interface. +1 -rian
Re: New mod_smtpd release
On Aug 16, 2005, at 6:47 AM, Nick Kew wrote: Rian Hunter wrote: [chop] I think I should have looked harder before replying ... looks like you've done more than I realised. I guess what I was asking about slots easily in to data_post ? Which question exactly? -rian
Re: New mod_smtpd release
On Aug 16, 2005, at 6:37 AM, Nick Kew wrote:
Spooling and dealing with multiple recipients are things I'd like to
think about now. ISTR posting about this before: I think each
recipient
needs a separate request processing cycle, because each recipient may
have completely different processing rules. Do you think we can deal
with that by creating a new suprequest for each recipient? If so,
we'll need to allow filter hooks both before and after subrequesting,
and pass each one the spooled input data.
I've thought about this and it has sort of been an avoided topic but
I think the current plan is that in smtpd_run_queue (the hook where
modules should queue or deliver the message) each registered hook
will be called with the recipient list and access to the message
until one of them returns something other than SMTPD_DECLINED or
SMTPD_OK (unlike smtpd_run_rcpt, which will stop when a module
returns SMTPD_OK).
For instance let's say you have a mail message with two recipients:
[EMAIL PROTECTED] and [EMAIL PROTECTED] and you want each to be dealt
with by different modules because one is a local address and one
should be relayed. The modules are mod_smtpd_queue_local and
mod_smtpd_queue_relay. Both of the modules have some means of
configuration which lets them know what email addresses they accept.
When smtpd_run_queue is called on them, they only accept the email
addresses they are configured for and ignore the rest. Of course if
not one module returns SMTPD_OK then we respond with some sort of 4**
error saying that we don't have a queue-er. When a server admin is
configuring the accepted email addresses for each module he should be
sure the their respective lists are mutually exclusive else he'll get
the same addresses being handled by two modules which is probably bad
news.
This design seems to satisfy the modularity that I was expecting from
mod_smtpd. Disabling local delivery or relay can be as simple as not
loading a module in a server config. Maybe mod_smtpd can even specify
a convention for email address acceptance lists like:
SmtpAcceptList my_list1 mysql://foo
SmtpAcceptList my_list2 regex:[EMAIL PROTECTED]
LocalQueueAccept my_list1
RelayQueueAccept my_list2
and mod_smtpd_queue_{local,relay} both know that they accept mail
from my_list1 and my_list2 repectively.
The problem with this design is that you have repetitious string
comparisons for each hook on smtp_run_queue. This is not a big
problem except for when the recipient list is long (~40 recipients)
and you have maybe three modules handling mail queuing. My short term
solution is that each module remove mail addresses from the recipient
list that they handled, so the list grows shorter as each queue hook
is called (this also solves the problem of email addresses handled
twice). Another short term fix for this is having mod_smtpd hash the
recipient list (md5?) as another list passed so comparisons and
lookups are quicker. I think this problem is a trade-off though for
our modularity.
Either way, lacking header parsing in mod_smtpd is being
impractically pedant since probably 99% of SMTP transfers involve
messages in the RFC 2822/MIME formats. Although I think that
maybe there will be a plugin that wants data from the DATA
command verbatim. I still feel this needs some thought.
Agreed. Maybe a hook for DATA could deal with pathological cases,
with
normal header/body handling the default? But I haven't thought it
through: I wasn't even aware of the notion of smtp-without-RFC(2)822.
I figure we'll have an input filter registered and if the data
doesn't look like RFC-(2)822, then it passes it on verbatim.
-rian
Re: [PATCH] use arrays in smtpd_request_rec
On Aug 15, 2005, at 10:05 PM, Garrett Rooney wrote:
Joe Schaefer wrote:
Garrett Rooney <[EMAIL PROTECTED]> writes:
Index: smtp_protocol.c
===
--- smtp_protocol.c(revision 232680)
+++ smtp_protocol.c(working copy)
[...]
+for (i = 0; i < sr->extensions->nelts; ++i) {
+ ap_rprintf(r, "%d-%s\r\n", 250, ((char **)sr->extensions-
>nelts)[i]);
^
That should be "elts", shouldn't it?
Yes indeed, it should. One of the problems with data structures
that require casting in order to access what you've stored in
them... The version that Rian committed is slightly different, but
still has the same problem. I'd post a patch, but it's probably
faster to fix it by hand than it is to detach a patch from an email
and apply it.
Over in Subversion-land we have a couple of macros defined for
dealing with apr arrays, they make it rather difficult to make this
kind of mistake, and I'd love to see them in the main APR release,
but the last time it was brought up I believe someone objected to
their inclusion...
-garrett
Fixed. I didn't even catch this because I don't test mod_smtpd with a
plugin that registers extensions. Things like this beckon some kind
of test suite module. Someone can get to this after mod_smtpd is redone.
-rian
Re: [PATCH] use arrays in smtpd_request_rec (was Re: smtpd_request_rec questions)
On Aug 14, 2005, at 11:08 PM, Garrett Rooney wrote: Rian Hunter wrote: This patch looks good but I have some questions. You seem to use the returned pointers from apr_array_push without checking if they are NULL. Even in apr_array_push, apr_palloc is used without checking for NULL even though apr_palloc can definitely return NULL. Because of that, I'm not sure whether or not you don't check for NULL on purpose. Could you explain? Thanks. Well, it depends on what your general attitude towards checking for errors in memory allocation. In many projects it's generally considered to be the kind of error you can't effectively recover from anyway, so cluttering up the code with if (foo == NULL) checks is kind of pointless, you're likely to have been killed by a kernel OOM checker before that can do anything useful, or you could be on an OS that doesn't even return NULL (memory overcommit), so the checks are pointless anyway. The only way to be safe is to make sure that algorithmicly your program can't allocate unbounded amounts of memory, then tune your box and app so that this kind of problem doesn't happen in practice. APR generally doesn't bother checking for this kind of error for just this reason, same with Subversion and if I'm not mistaken Apache HTTPD itself. -garrett Thanks for this information! After looking at code in httpd it seems this is the case. I'll change the mod_smtpd code to reflect this convention. -rian
Re: New mod_smtpd release
On Aug 15, 2005, at 10:22 AM, Joe Schaefer wrote: "Jem Berkes" <[EMAIL PROTECTED]> writes: Well there's also another problem. RFC 2821 (SMTP) doesn't define a particular message format for SMTP (in wide use there the RFC 822 and MIME message formats). I don't think that mod_smtpd should assume a RFC 822 or MIME message format since its strictly a SMTP module, that's why I agree with this Now I'm confused; 2821 S-2.3.1 defines "SMTP content" as headers + body. What am I overlooking? 2821 s-2.3.1 says: If the content conforms to other contemporary standards, the headers form a collection of field/value pairs structured as in the message format specification [32]; the body, if structured, is defined according to MIME [12]. Personally I interpret this to mean that the content may not conform to "other contemporary standards" although I do doubt the existence of non-RFC 2822 header formats. I still think header parsing should be in another module. Of course this module is free to register itself as an mod_smtpd filter and do what it needs to do, but it shouldn't be part of the main mod_smtpd. If you put in into a separate module, that means there will be no hooks which can expect the headers to be in r->headers_in. So every hook that needs them will need to parse it themselves, which seems absolutely redundant. Furthermore it is a requirement (MUST) for a 2821 compliant server to implement loop detection. That means at least one hook will *always* care about the Received: headers, for every smtp transaction. Why you wouldn't want to provide an API for hook authors to use for inspecting headers, seems like a very spartan choice, which IMO is counter to the spirit of httpd. Well not exactly. A module that parses headers can register itself as an input_filter for mod_smtpd. It can parse the headers and the headers can be accessed by a get_smtpd_headers(request_rec*) function or similar exported by the parsing module, and modules that rely on this module will know about that API. This module can even be included with the default mod_smptd package. Do you agree that this is possible? The module that implements loop detection could rely on that module and also be included with the defautl mod_smtpd package. Either way, lacking header parsing in mod_smtpd is being impractically pedant since probably 99% of SMTP transfers involve messages in the RFC 2822/MIME formats. Although I think that maybe there will be a plugin that wants data from the DATA command verbatim. I still feel this needs some thought. -rian
Re: [PATCH] use arrays in smtpd_request_rec (was Re: smtpd_request_rec questions)
This patch looks good but I have some questions. You seem to use the
returned pointers from apr_array_push without checking if they are
NULL. Even in apr_array_push, apr_palloc is used without checking for
NULL even though apr_palloc can definitely return NULL.
Because of that, I'm not sure whether or not you don't check for NULL
on purpose. Could you explain? Thanks.
-rian
On Aug 14, 2005, at 8:52 PM, Garrett Rooney wrote:
Garrett Rooney wrote:
Rian Hunter wrote:
Ah I didn't even realize the key allocation, I'll fix that. Thanks!
The reason I don't use an apr_array_t or similar is that I
thought that the number of elements in that type has to be fixed
and can't be automatically extended and allocated on the fly, If
I'm wrong I can change these structures to use apr_array_t (or
you can send in a patch if you like).
APR arrays are variable length, you use apr_array_push and
apr_array_pop to add and remove entries. I'll throw together the
patch sometime today.
And here's the patch. This is totally untested, since I haven't
yet gotten around to making mod_smtpd do anything other than
compile yet, but it's pretty simple and should give you an idea of
what I meant.
-garrett
* mod_smtpd.h
(smtpd_request_rec::e_index,
smtpd_request_rec::r_index): removed.
(smtpd_request_rec::extensions,
smtpd_request_rec::rcpt_to): change from a hash to an array.
* smtp_core.c
(smtpd_register_extension): push the extension onto the array.
(smtpd_clear_request_rec): allocate the new arrays.
* smtp_protocol.c
(helo): remove ext and ext_next variables, iterate over the array
instead of doing backflips to iterate over the hash.
(rcpt): remove new_elt variable, just push the rcpt address onto
the array.
Index: mod_smtpd.h
===
--- mod_smtpd.h(revision 232680)
+++ mod_smtpd.h(working copy)
@@ -90,16 +90,13 @@
// by default smtp
smtpd_protocol_type extended;
-// current max index of the extension hash
-int e_index;
-apr_hash_t *extensions;
+apr_array_header_t *extensions;
// string of who this mail is from
char *mail_from;
-// current max index of the rcpt_to hash
-int r_index;
-apr_hash_t *rcpt_to;
+apr_array_header_t *rcpt_to;
+
// hostname we were helo'd with
char *helo;
} smtpd_request_rec;
Index: smtp_protocol.c
===
--- smtp_protocol.c(revision 232680)
+++ smtp_protocol.c(working copy)
@@ -121,7 +121,6 @@
HANDLER_DECLARE(ehlo) {
int i = 0, retval = 0;
- char *ext = NULL, *ext_next;
smtpd_request_rec *sr = smtpd_get_request_rec(r);
if (buffer[0] == '\0') {
@@ -161,21 +160,13 @@
}
// print out extension
- ext = apr_hash_get(sr->extensions, &i, sizeof(i));
retval = 250;
- if (ext) {
+ if (sr->extensions->nelts) {
ap_rprintf(r, "%d-%s\r\n", 250, sr->helo);
-
-while (1) {
- i++;
- if ((ext_next = apr_hash_get(sr->extensions, &i, sizeof(i {
-ap_rprintf(r, "%d-%s\r\n", 250, ext);
- } else {
-ap_rprintf(r, "%d %s\r\n", 250, ext);
-break;
- }
- ext = ext_next;
+
+for (i = 0; i < sr->extensions->nelts; ++i) {
+ ap_rprintf(r, "%d-%s\r\n", 250, ((char **)sr->extensions-
>nelts)[i]);
}
} else {
ap_rprintf(r, "%d %s\r\n", 250, sr->helo);
@@ -344,7 +335,6 @@
char *loc;
char *allocated_string;
int retval = 0;
- int *new_elt;
// need mail first
if ((sr->state_vector != SMTPD_STATE_GOT_MAIL) &&
@@ -413,19 +403,8 @@
// add a recipient
if ((allocated_string = apr_pstrdup(sr->p, loc))) {
-new_elt = apr_palloc(sr->p, sizeof(int));
+(*((char **)apr_array_push(sr->rcpt_to))) = allocated_string;
-if (!new_elt) {
- ap_rprintf(r, "%d %s\r\n", 421, "Error: Internal");
- // out of memory close connection
- retval = 0;
- goto end;
-}
-
-*new_elt = sr->r_index;
-apr_hash_set(sr->rcpt_to, new_elt,
- sizeof(*new_elt), allocated_string);
-sr->r_index++;
sr->state_vector = SMTPD_STATE_GOT_RCPT;
ap_rprintf(r, "%d %s\r\n", 250, "Ok");
Index: smtp_core.c
===
--- smtp_core.c(revision 232680)
+++ smtp_core.c(working copy)
@@ -127,11 +127,7 @@
SMTPD_DECLARE_NONSTD(void)
smtpd_register_extension(smtpd_request_rec *sr, const char *line)
{
- int *cur = apr_palloc(sr->p, sizeof(int));
- *cur = sr->e_index;
-
- apr_hash_set(sr->extensions, cur, sizeof(*cur), line);
- (sr->e_index)++;
+ (*((char **)apr_array_push(sr->extensions))) = apr_pstrdup(sr-
>p, line);
}
//
Re: New mod_smtpd release
On Aug 14, 2005, at 8:12 PM, Joe Schaefer wrote: Rian Hunter <[EMAIL PROTECTED]> writes: On Aug 14, 2005, at 1:22 PM, Joe Schaefer wrote: +RELEASE SHOWSTOPPERS: + + + smtp_process_connection_internal should take a "smtp_proto_rec" + argument (which is what the current "smtp_request_rec" struct + should be renamed to). I can easily rename smtpd_request_rec but I don't think I should pass it to smtpd_process_connection internal only because the hooks take a request_rec*. The hooks can still take a request_rec, but IMO the protocol's state management shouldn't be done from a request_rec. I'd still like to see one request correspond to one MAIL FROM/RCPT TO/DATA sequence, so that whenever the state gets reset, the whole request_rec pool gets cleaned up. This make sense. smtpd_request_rec does what you say. After looking at smtp_core.c it seems that request_rec isn't really something needed for mod_smtpd (and really never was). After figuring out the input filters situation, i'll probably do away with request_rec (since it isn't needed for connection-level filters) and just stick to smtpd_proto_rec. Any objections? They need request_rec to use filters (even though i don't currently enable convenient use of filters yet). The request_rec slot can be NULL for connection-level filters. But I'd create a request_rec sometime before I added an smtp protocol filter, which would just do the "."-decoding, similar to how http_in deals with TE. Yeah I agree. Ideally header parsing should be done in an mod_smtpd plugin not in mod_smtpd. I respectfully disagree, because I'd like different hooks to have increasing state information available to them through the request_rec. In particular I'd like to see the smtp filtering API match httpd, by first parsing the headers, but passing the rest of the data through r->input_filters, with smtp_in translating the final "." line into an EOS bucket. Well there's also another problem. RFC 2821 (SMTP) doesn't define a particular message format for SMTP (in wide use there the RFC 822 and MIME message formats). I don't think that mod_smtpd should assume a RFC 822 or MIME message format since its strictly a SMTP module, that's why I still think header parsing should be in another module. Of course this module is free to register itself as an mod_smtpd filter and do what it needs to do, but it shouldn't be part of the main mod_smtpd. The modules that will specifically rely on this header parsing module will know how to obtain the header information using the conventions specified by that parsing module. -rian
Re: smtpd_request_rec questions
On Aug 14, 2005, at 4:21 PM, Garrett Rooney wrote: I was just looking at the smtpd_request_rec in mod_smtpd, and I had a few questions. It seems that extensions and rcpt info is being stored in an apr_hash_t, but it's only being keyed by integer. If you're only going to use ints as keys, it seems like an apr_array_header_t would be more appropriate. Also, while the extensions has is specifically allocating its keys, the rcpt_to hash is reusing the r_index variable each time, so if more than one entry is added things aren't going to work, since that location in memory is having its value changed after its been used as a key. Anyway, I'd be happy to produce patches to either switch to an array or fix the key issue in the rcpt_to section, depending on which way we want to go. Personally, I'd prefer the array version though. -garrett Ah I didn't even realize the key allocation, I'll fix that. Thanks! The reason I don't use an apr_array_t or similar is that I thought that the number of elements in that type has to be fixed and can't be automatically extended and allocated on the fly, If I'm wrong I can change these structures to use apr_array_t (or you can send in a patch if you like). -rian
Re: mod_smtpd compile problem
On Aug 14, 2005, at 2:05 PM, Garrett Rooney wrote: So I'm having a little trouble getting mod_smtpd to compile, once I fixed up the configure script to find apxs and apache correctly, I end up with the following error: $ make /usr/bin/apxs2 -Wc,"-Wall" -o mod_smtpd.la -c smtp_core.c smtp_protocol.c /usr/bin/libtool --silent --mode=compile gcc -prefer-pic -pipe -I/ usr/include/xmltok -I/usr/include/openssl -Wall -O2 - DAP_HAVE_DESIGNATED_INITIALIZER -DLINUX=2 -D_REENTRANT - D_XOPEN_SOURCE=500 -D_BSD_SOURCE -D_SVID_SOURCE -D_GNU_SOURCE -pipe -I/usr/include/xmltok -I/usr/include/openssl -Wall -O2 -pthread -I/ usr/include/apache2 -I/usr/include/apr-0 -I/usr/include/apr-0 -I/ usr/include -Wall -c -o smtp_core.lo smtp_core.c && touch smtp_core.slo smtp_core.c: In function `register_hooks': smtp_core.c:348: error: `default_queue' undeclared (first use in this function) smtp_core.c:348: error: (Each undeclared identifier is reported only once smtp_core.c:348: error: for each function it appears in.) smtp_core.c:349: error: `default_rcpt' undeclared (first use in this function) apxs:Error: Command failed with rc=65536 . make: *** [mod_smtpd.la] Error 1 Sorry I removed one part of code and not the other. -rian
Re: New mod_smtpd release
On Aug 14, 2005, at 1:22 PM, Joe Schaefer wrote: snip... +CURRENT RELEASE NOTES: + + Virtual hosts a'la mod_ftpd don't work. It does work like this: Listen 80 Listen 25 NameVirtualHost *:80 NameVirtualHost *:25 ServerName localhost DocumentRoot htdocs ServerName localhost SmtpProtocol On no? +RELEASE SHOWSTOPPERS: + + + smtp_process_connection_internal should take a "smtp_proto_rec" argument + (which is what the current "smtp_request_rec" struct should be renamed to). I can easily rename smtpd_request_rec but I don't think I should pass it to smtpd_process_connection internal only because the hooks take a request_rec*. They need request_rec to use filters (even though i don't currently enable convenient use of filters yet). Unless I add a member to smtpd_proto_rec that is a pointer to the related request_rec (solely so filters can work) and use smtpd_proto_rec as the main structure. + +CURRENT VOTES: + + +TODO ISSUES: + + The request i/o is driven around ap_rgetline, when it really + should be using input filters. I have to look a little more into this. +WISH LIST: + + Link against libapreq2 so we can use its header and multipart parsers. + apreq's header parser would help in implementing rfc2821 loop- detection, + and in providing the header collection as r->headers_in for "data" + hooks to examine. Ideally header parsing should be done in an mod_smtpd plugin not in mod_smtpd. -rian
New mod_smtpd release
Hi, I've checked in mod_smtpd 0.9 and its API should be completely frozen by now. This version of mod_smtpd is heavily based on Qpsmtpd, so the same extensibility you expect from Qpsmtpd can be achieved with this version of mod_smtpd. I haven't written any documentation yet but here is a quick run-down of how to use it: In your httpd.conf, make sure you have SmtpProtocol On, if you are setting up a virtualhost make sure the virtualHost container has the ServerName directive (duh). This version of mod_smtpd is callback based, very similar to Qpsmtpd. Here is a list of all the hooks you can register: smtpd_run_unrecognized_command smtpd_run_connect smtpd_run_reset_transaction smtpd_run_helo smtpd_run_ehlo smtpd_run_mail smtpd_run_rcpt smtpd_run_vrfy smtpd_run_quit smtpd_run_data smtpd_run_data_post smtpd_run_data_queue You can register a hook to one of these by calling: APR_OPTIONAL_HOOK(smtpd, /* hook name */ vrfy, /* function address */ default_vrfy, NULL, NULL, APR_HOOK_FIRST); In your register hooks function. Each hook you register should return smtpd_retcode, to see what retcodes make sense for each callback you should look at smtp_protocol.c until I write better documentation. To see what argument each different type of hook takes look at smtp_core.c. The code is very small and simple and shouldn't be too hard to figure out if you're familiar with apache modules. Currently it works with httpd 2.0 and up. You can checkout this code out from: http://svn.apache.org/repos/asf/httpd/mod_smtpd/trunk/ Have Fun! -rian
httpd 2.1 and ap_hook_create_request()
While working on mod_smtpd and verifying it works with current httpd I had a problem getting the ap_rgetline function to properly work without calling ap_run_create_request(). I figured the root of this problem was that core.c registers a net time filter that sets a default timeout on the socket in its create_request hook. It isn't an issue to call ap_run_create_request similar to how its done in http_core.c the only problem is that the create_request hook registered in http_core.c adds filters irrelevant to mod_smtpd (namely http_header_filter). My temporary solution to this problem is to call apr_socket_timeout_set() on the socket in r->input_filters->ctx- >client_socket manually without calling ap_run_create_request(). This seems a little dirty to me, but I'm not completely sure. Is a call to ap_run_create_request() relevant for a module like mod_smtpd, or is my temporary solution to ignore it correct enough? If calling ap_run_create_request should be necessary for a module like mod_smtpd, then other modules shouldn't register filters on the request_rec without checking if its the appropriate protocol/request. -rian
Re: Initial mod_smtpd code.
On Jul 19, 2005, at 7:39 PM, Jorge Schrauwen wrote: Wouldn't something like ServerName www.apache.org:81 ServerAlias httpd.apache.org internal_dev ServerAdmin [EMAIL PROTECTED] ErrorLog logs/error.log TransferLog logs/access.log SMTP On # like with mod_dav (dav on) so reusing existing command, then this main an user of www.apache.org lets call him [EMAIL PROTECTED] that the following addresse point to the same. [EMAIL PROTECTED] [EMAIL PROTECTED] this seems to be more logical to me. Personally I think this is too much reuse of configuration in a context that may not make sense to a user or admin. Either way, conf like that is irrelevant to core mod_smtpd. -rian
Re: Initial mod_smtpd code.
On Jul 19, 2005, at 6:51 AM, Nick Kew wrote: the problem i found when i did my poc is when there is in the command, different destination email. It's difficult here to keep the virtualHost scheme. It would be nice to keep a conf file like ServerName mail.bla.com SmtpUserMap mail.bla.com-user.map or SmtpRelay host Yes. I think there is a logical difficulty here. smtpd_create_request is run before process_smtp_connection_internal, and the design doesn't allow for multiple recipients to be processed differently. My own feeling was that multiple recipients require a request each. Maybe we could get that in your design by creating a subrequest for each RCPT TO? This is possible but I'm not sure what the advantage is. Would you mind setting up a hypothetical situation where this is necessary? I figure that ultimately the handler will be responsible for dealing with each rcpt to differently. Early on I wanted configuration possibility similar to: # Listen 21 # mod_smtpd conf SmtpProtocol On SetHandler unix-module # mod_smtpd_unix AcceptDomains thelaststop.net www.thelaststop.net Relay On # mod_smtpd_easyfilter # matches against email in MAIL TO: smtp command RegexMailTo "/thelaststop.net$/" SetHandler maildir-module # mod_smtpd_maildir conf MaildirBase "/usr/local/virtual" MailboxLimit 50M # Simple spam filter using mod_smtpd_easyfilter # Default handler does nothing with mail message RegexHeader "Subject" "/cialis/" SetHandler none ## In the case of this httpd.conf embedded filtering mechanism, I figured the handlers could be changed based on the certain RegExs right before ap_run_handler() was called. Maybe ap_run_fixups() could be called, and my hypothetical mod_smtpd_easyfilter would have a fixups hook where it accomplished something similar to this situation. Although after thinking about it I realize now that mod_smtpd_easyfilter couldn't set different handlers for different rcpt tos. Is this what you meant? I think this requires some more thought considering different smtp connections and server requirements. The main drawback to sub- requesting each rcpt to is that we have two different handlers trying to read data from the socket. Is this problem solved by spooling the data, and letting the two separate requests read from the spool bucket? -rian
Re: Initial mod_smtpd code.
Nifty! I had some compilation problems involving regex, so in the attached patch I use ap_regex.h and change some defines. Hope this doesn't break anything. that was a good idea, ap_regex.h was implictly getting included for me. The other bug I partially fixed was, strstr in smtp_protocol.c only does exact matches so uppercase commands like MAIL FROM would fail. I added support for the upper case, but this needs to be improved still because mixed case doesn't work. Is there an APR function like stristr? agh i forgot about an upper case FROM: or TO:. process_smtp_connection_internal() already lowercases the actual command, but not the rest of the params. Either MAIL from: or RCPT to: would work, but not FROM or TO. Anyway, stristr can be easily hacked by calling strstr with two lowercased strings. The overall structure and the approach you took is very nice, easy to understand. I would recommend adding a hook immediately upon the client connection, because an external module (maybe for DNSBLs, or some rate limiting control) might not even want us to return a greeting at all -- i.e. close with "554 Service unavailable" right away. Hmm. That sounds like a good idea, maybe there already is a hook defined that could deal with this, I'll look into it. But I like what you have, would be happy to keep working around this design. Thanks! Do you now have svn access? After I apply your patches I'll see about checking this in so more of us can deal with this code. It sucks that most are busy with ApacheCon right now. -rian
Initial mod_smtpd code.
Hi, This is my first attempt at writing an experimental version of mod_smtpd. I don't yet have svn access yet so this code can be downloaded from http://rian.merseine.nu/mod_smtpd-0.1.tar.gz. This implementation shows my vision for mod_smtpd and it isn't perfect, so please test/look at the code and think about where its method of extensibility works and doesn't work. This implementation is different from Jem's view of hooks for each smtp command (i do something similar but not so drastic as hooks for each smtp command, i have a apr_hash_t with function pointers for each different smtp command). Jem/Paul/Nick: I'm especially interested in what you think about the design I've laid out in this implementation. -rian
Re: Questions from a newbie
Jem Berkes wrote: For example, for SMTP support we are contemplating the protocol unit (mod_smtpd) passing on mail to a module that specifically delivers mail. How do those two entities communicate with each other? A mod_smtp_deliver would get a potentially large chunk of data (const char* ?) from mod_smtpd and deliver the mail via procmail, etc. This is a loose binding by the way since all received mails do not necessarily have to be delivered. I was thinking of calling the handlers, and letting the handlers read from the input_filters list in the request_rec, check out the ap_rgetline function. Does this sound like a bad idea to you? any security issues? The other solution (letting mod_smtpd read the whole thing into a buffer and then passing the buffer) seems way too memory hungry for me. -rian
Re: mod_smtpd design.
Ben Laurie wrote:
Another approach still would require a fairly large change to the core
and many modules, but it strikes me as a better option...
struct http_data {
.
.
.
};
struct smtp_data {
.
.
.
};
struct request_rec {
.
. /* common stuff */
.
struct http_data *http;
struct smtp_data *smtp;
}
If there was going to be a large change to core, request_rec and friends
how about:
struct request_rec {
/* common stuff */
char *protocol_name; // different from r->protocol,
// but maybe doesn't have to be
struct ap_conf_vector_t *request_config;
};
The request_config vector would store all the protocol specific data.
Protocol_name would be so modules new what they were dealing with. To
retrieve protocol specific data define this in each protocol module:
smtpd_request_rec *get_smtpd_request(request_rec *r) {
return ap_get_module_config(r->request_config, &MODULE_NAME);
}
Then in handler modules:
type misc_smtp_handler(request_rec *r) {
smtpd_request_rec *smtp_data;
if (strncmp("http", r->protocol_name, 4)) {
// decline to handle, this module doesn't handle
// http requests.
}
//then get smtpd specific data
smtp_data = get_smtpd_request(r);
// do some handlin'
}
The advantage to this approach is a less bulky (but more all
encompassing) request_rec with support for an arbitrary amount of
protocols and protocol specific data.
-rian
more mod_smtpd stuff [was Summer of Code]
Jem Berkes wrote: To address one of the points brought up on IRC, if there is actually a non-experimental target for this software any time soon it would make more sense to support the 2.0 server as I think few production servers would be running 2.1? I'm just speaking from what I saw, that is, among my colleagues I do not know anyone who has tried Apache 2.1 yet. Many are still stuck at 1.3, but those people suck anyway :) I think that deciding between 2.1 and 2.0 isn't a big deal since we can have source compatibility between them. You do make a good point when you mention all the poeple who still use 1.3. I think we should support 2.0, although (i'm not trying to sound negative) I'm pretty sure that most of the people that will be using mod_smtpd will be people writing specialized smtp setups (and running the latest httpd anyway) and not people who run production mail servers, even though mod_smtpd will be powerful enough to be a production mail server i just think sys admins are more stubborn about changing their mail server setup than the web server setup. -rian
mod_smtpd design.
Hi, Currently there are two approaches we are looking at for mod_smtpd. We can use the existing request_rec structure, and store smtp specific data in a structure stucture in the r->request conf vector. With this we can reuse some of the existing core hooks that make sense (handler, create_request) while also adding some new ones for smtp (envelope, called after all necessary commands have been sent). The downside is that a lot of the request_rec members are extraneous for smtp. The other approach is to use a custom smtp_request_rec structure for an smtp session. This has the advantage/disadvantage of defining new hooks only necessary for smtp, but the disadvantage is that currently in httpd 2.x filters that alter content data require a valid request_rec. It would be possible to pass a bogus request_rec with the filters set to ap_add_*_filter*() (or ap_run_insert_filter() so we let reuse core's handle the Set(Out|In)putFilter directive), except that seems a little hackish to me. Currently a session specific smtp data structure will have the fields: (string) (he|eh)lo argument (string) mail from argument (string) rctp to argument a module specific data structure wlil have the fields (table) supported smtp extensions (could be extended by other modules with ap_run_post_config()) (bool) smtp enabled? (string) server identifier (mod_smtpd by default) Since filters are something of great importance to mod_smtpd, i'm leaning toward keeping request_rec, and storing session specific data in r->request_conf, although I am very open to doing away with request_rec completely and definining new hooks (which also means adding new handler directives, eg. SetSmtpHandler). Anyone have any thoughts and additional comments/considerations? Here is a list of hooks i was thinking that mod_smtpd should call: ap_hook_auth_checker /* maybe this one after helo, mail from, rcpt to but before data */ ap_hook_create_request /* after the request_rec has been created */ ap_hook_fixups /* right before calling handler */ ap_hook_handler /* called to handle request or session */ ap_hook_insert_filter/* right before calling handler */ either ap_hook_auth_checker or a new hook we'll define for smtp (tentatively called "envelope"). that's still up in the air. If we go with the discarding request_rec design, basically we'll still call these logical hooks. Are there any complications I've overlooked? Any criticism or comments are very welcome!! -rian
Re: mod_smtpd project planning
Paul A Houle wrote: Jem Berkes wrote: Hi all, I'm another student working on mod_smtpd Been running httpd 2.x since it appeared, but am new to development. What does mod_smtpd do? Is it a sendmail replacer or does it let people request content via smtp or what? It's a SMTP protocol frontend for httpd. It will have the power to be a sendmail replacer or to supply content via SMTP because it will delegate most of the actual handling to other modules. All the details haven't been worked out yet, but it will make use of the Apache 2.x filters and handlers. For Instance: [core] -> [mod_smtpd] -> [mod_insert_special_use_mail_handler] a setup like that could be used, but let's say you want to filter out junk mail. Use an input filter! [core] -> [mod_smtpd] -> [mod_junk_mail_filter] -> [mod_other_thing] No rewrite! The handler can be anything and the possibilities are endless (just like in httpd). As a proof of concept we will probably write or reuse a basic delivery module. Another upside to this is that lets say you want to implement a new proprietary feature for SMTP (like a content supplier). Your modules just has to register which extensions it supports (for a response to the ehlo command), and you don't have to worry about the details of writing a new SMTP server when most of the tedious work has already been taken care of. -rian
