A BUGNOTE has been added to this bug. ====================================================================== http://www.dbmail.org/mantis/bug_view_advanced_page.php?bug_id=0000057 ====================================================================== Reported By: maXXmaster Assigned To: aaron ====================================================================== Project: DBMail Bug ID: 57 Category: PIPE delivery (dbmail-smtp) Reproducibility: N/A Severity: feature Priority: normal Status: acknowledged ====================================================================== Date Submitted: 23-Jul-04 23:37 CEST Last Modified: 18-May-05 16:46 CEST ====================================================================== Summary: direct mailforwarding with [EMAIL PROTECTED] Description: it doesn't seem that there is a way to store "special" mails like spam to a different folder than INBOX when pipeing the message to dbmail-smtp. i'm not sure if it is an imap-standard, but many imapservers allow you to send a message to [EMAIL PROTECTED]
is there a way to realise that? ====================================================================== ---------------------------------------------------------------------- danweber - 24-Jul-04 03:38 CEST ---------------------------------------------------------------------- Yes we need this feature added so we can start using tmda with exim too. ---------------------------------------------------------------------- aaron - 24-Jul-04 17:05 CEST ---------------------------------------------------------------------- There's a command line option to specify an alternate mailbox along with the destination address: dbmail -m mailbox {-t [headerfield] | -d [emailaddress]} If there's a way to make this work with Exim, it would be much easier to document such a method rather than adding new support for the username+mailbox syntax. ---------------------------------------------------------------------- maXXmaster - 25-Jul-04 20:40 CEST ---------------------------------------------------------------------- i actually wanted to use it with amavis-new which is able to append an foldername after the mailadress (incase of spam or viruses).. i'll probably have to find a way to make it work with the -m mailboxname option,.. let's see ;o) edited on: 25-Jul-04 20:40 ---------------------------------------------------------------------- aaron - 25-Jul-04 20:44 CEST ---------------------------------------------------------------------- Ok, keep us posted. Interfacing with other parts of the mail system is, quite naturally, a high priority :-) Is there a published standard advocating for this hack, btw? Some de facto rule book that we can follow if it turns out to be necessary to support this syntax? ---------------------------------------------------------------------- maXXmaster - 25-Jul-04 22:56 CEST ---------------------------------------------------------------------- well, not really,.. at least i couldn't find one within the last hour. my (probably) working solution is to use the following chain: postfix (:25) -> amavis-new (:10024) -> postfix#2 (:10025) -> procmail (-> script which does the conversion from [EMAIL PROTECTED] to the dbmail-smtp syntax you mentioned) -> dbmail maybe i can write more about the necessary configurations tomorrow. ---------------------------------------------------------------------- maXXmaster - 26-Jul-04 11:52 CEST ---------------------------------------------------------------------- i tried some things and came up with quite a good solution. i use a small (php)script insted of the dbmail-smtp program in postfix (inside master.cf) to extract the +folder from the emailaddress and pipe it to dbmail-smtp. the only problem is, that dbmail-smtp cannot add messages to a special mailbox, if i want to forward them to an emailaddress. it just works with direct usernames (user -u markus vs -d [EMAIL PROTECTED]) so either i ask the dbmail-database which useralias i should use for special mailaddresses (that is exactly what dbmail-smtp should actually do), or some of you guys add a few lines to make dbmail-smtp work with mailboxes AND emailaddresses or to make it work with [EMAIL PROTECTED] any suggestions? =) edited on: 26-Jul-04 11:52 ---------------------------------------------------------------------- maXXmaster - 26-Jul-04 13:00 CEST ---------------------------------------------------------------------- > Is there a published standard advocating for this hack, btw? > Some de facto rule book that we can follow if it turns out > to be necessary to support this syntax? yes, there is. i found an rfc which "specifies" the [EMAIL PROTECTED] syntax. it can be found here: http://www.ietf.org/rfc/rfc3598.txt i hope that is enough information to the syntax ;o) ---------------------------------------------------------------------- aaron - 26-Jul-04 16:15 CEST ---------------------------------------------------------------------- Thanks for the RFC link, I'll look into it. Indicentally, there's no difference between -u username and -d address. The address lookup checks usernames first... I'll start a thread on dbmail-dev about whether or not we should keep both options as synonyms, or make them different again. ---------------------------------------------------------------------- maXXmaster - 26-Jul-04 16:49 CEST ---------------------------------------------------------------------- are these two options really the same? hmm, maybe in version 2.* but in 1.2.9 (which i am using) they are different. at least i get a message telling me to use -u username and not -d address if i want to store directly to a mailbox. ---------------------------------------------------------------------- aaron - 26-Jul-04 18:15 CEST ---------------------------------------------------------------------- That's correct, they were merged at a fairly deep level in the new delivery chain. My thinking was twofold (and in fact, this should probably go into an FAQ or RATIONALE file someplace)... - It seems odd to have to add an alias to your own username in order to support domain-local delivery without expansion, as is common when you use "mail" on the command line. - It seems really odd to have to add an alias to your own username when your username is a full email address, as is common for folks running multiple separate email domains from a single server. ---------------------------------------------------------------------- aaron - 26-Jul-04 18:22 CEST ---------------------------------------------------------------------- So basically, here's my take on this so far: - [EMAIL PROTECTED] is a common and accepted format that we should support. It is possible that some people will not want to use such an expansion, while many other would, and, therefore: - we need another command line option to handle this syntax. Because -u and -d have been merged at a very deep level in the delivery chain, it would make sense then to use -d to support this syntax and -u to not support this syntax. The changes can be made at a very high level, merely by adding a check for anything bounded by "+" and "@", removing that portion of the address and assigning it to be the mailbox destination. - as a significant UI mechanism, this would need to be added before 2.0 ---------------------------------------------------------------------- maXXmaster - 26-Jul-04 21:47 CEST ---------------------------------------------------------------------- sounds good to me ;o) a small sidenote: i couldn'n find a good howto about settingup dbmail+amavis-new(+spamassassin+clamav)+postfix so if there are more people interested in that, i could try writing one. ---------------------------------------------------------------------- aaron - 27-Jul-04 10:24 CEST ---------------------------------------------------------------------- Check out the latest CVS and let me know if it is working as you would expect. ---------------------------------------------------------------------- aaron - 27-Jul-04 10:28 CEST ---------------------------------------------------------------------- Oh, and I might mention that this is the 2.0 CVS. If you're running 1.2.x you will have to use a script as a helper and you may be out of luck on the mail folder thing. ---------------------------------------------------------------------- maXXmaster - 01-Aug-04 09:42 CEST ---------------------------------------------------------------------- so the mailfolder stuff will just work in 2.x ? =( as dbmail is working on a small production-server i want(ed) to use the stable 1.2.x version. so i'll probably have to use my helper-script ... ---------------------------------------------------------------------- ilja - 04-Aug-04 16:26 CEST ---------------------------------------------------------------------- So, does it all work like it should? If so, we can close this bug :) ---------------------------------------------------------------------- aaron - 05-Aug-04 07:56 CEST ---------------------------------------------------------------------- Just ran it through its paces, and it works as expected. Neat new feature! Now if Mr. maXXmaster would like to write README.amavisnew... ---------------------------------------------------------------------- maXXmaster - 05-Aug-04 22:34 CEST ---------------------------------------------------------------------- it is possible to create unlimited folders by sending messages to [EMAIL PROTECTED] that is not really good, in case of mailbombs. wouldn't it be better to store mails in the inbox if the folder doesn't exist? ---------------------------------------------------------------------- aaron - 06-Aug-04 01:29 CEST ---------------------------------------------------------------------- Good point, and one that isn't handled with the little-m and big-M options... medium-Mm for providing a mailbox in case the one specified does not exist. It may actually be a really big problem that someone could specify a mailbox from outside of your mail system. This syntax is something that DBMail needs to understand from upstream (amavis, exim) but which needs to be filtered from the outside world (perhaps by exim or postfix before initial delivery). Creating mailboxes on the fly was a feature that I added to simply adding users and to enable users to be authenticated from LDAP without requiring additional steps to enable each user in DBMail. However, now we have a situation where the mailbox can be specified by an external sender. Probably a very bad thing. ---------------------------------------------------------------------- ilja - 06-Aug-04 10:28 CEST ---------------------------------------------------------------------- Can't we change this easily? We can still let the db_find_create_mailbox() function create the mailbox, but only when its name is "INBOX". The change to db_find_create_mailbox() is easily made. Just add a if (strcmp(name, "INBOX") == 0) to the function and set mailbox_idnr to 0 is it fails. There are three places in the sources where db_find_create_mailbox() is used. One is in db.c, in db_insert_message(). This use does not have to be changed, because the value of mailbox_idnr is checked later on. There are two places in sort/sort.c where the function is used. These will have to be changed. Aaron, sort/sort.c is your code, do you want to take a shot at this? Ilja ---------------------------------------------------------------------- aaron - 06-Aug-04 18:03 CEST ---------------------------------------------------------------------- I'm not a big fan of an INBOX-only solution; I'd much rather be able to ensure that +mailbox addresses cannot arrive from across the network. The question I have now is if there's a difference between: To: [EMAIL PROTECTED] -> Postfix -> Amavis -> DBMail ([EMAIL PROTECTED], into box) To: [EMAIL PROTECTED] -> Postfix -> Amavis (+box) -> DBMail ([EMAIL PROTECTED], into box) DBMail most likely would have no way to cleanly separate the two situations, so either we have to limit what DBMail can do (INBOX only, ugh) or make sure that somebody is disallowing incoming messages of this format (either Postfix or Amavis, if there's a way to have them strip out the +box detail). ---------------------------------------------------------------------- maXXmaster - 07-Aug-04 07:14 CEST ---------------------------------------------------------------------- i tried finding config-examples (at least for postix) to switch-off "plus detail" ([EMAIL PROTECTED]) but couldn't find anything. maybe it'S just too early, but i think it would be a good setting for the config-file! .) to enable all +folder .) just to allow storing to existing folders (otherwise INBOX) .) or to disable it at all. would that work for you? ---------------------------------------------------------------------- aaron - 08-Aug-04 06:59 CEST ---------------------------------------------------------------------- This is pretty much an intractable problem right now. There only easy solution is to find a way to trash the +folder part immediately from the outside SMTP server. The other three solutions are: - comment out the code and put this off until later. - provide a mechanism to specify a default mailbox to use in case the provided one does not exist (the default mailbox would be created if not found, to prevent the corner case of a nonexistant INBOX, for example). - disable auto mailbox creation altogether, except a hardcoded INBOX. A few months ago, I started to write code for the second situation, apparently with a very good insight that we'd want auto mailbox creation be option in some cases -- that code requires a number of deep changes. I'm confident that they could be debugged within the coming 2.0rc8, but I'd rather not have that pressure. Disabling auto mailbox creation altogether isn't a very good option because we do need it for LDAP users or perhaps if the user somehow deletes their own INBOX folder entry. Hardcoding an INBOX exception strikes me as quite ugly. Even if we did disable this address syntax, someone who wants to work with TMDA or Amavis-new would have to write a script that used the dbmail-smtp -m "mailbox" option, which is vulnerable to exactly the same attack. So it would be the mail admin hanging themself instead of DBMail doing it for them. So that leaves us with one good option and one crumby one: - figure out how to filter this at the MTA. - go for more new code. Bummer. ---------------------------------------------------------------------- maXXmaster - 08-Aug-04 13:17 CEST ---------------------------------------------------------------------- it really seems to be a tricky problem. as a wrote before. i couldn't find a way to tell postfix to strip-off the +folder part (still i believe there is a way). nevertheless i think it is a good start to be able to work with amavis the way you implemented it now. (as long as 1.2.9 is the officially stable version it won't be a problem to leave it the way it is right now) if there might be a better solution in rc9 i welcome it. ---------------------------------------------------------------------- aaron - 12-Aug-04 11:58 CEST ---------------------------------------------------------------------- Changing the status to acknowledged, which is what we're using for things that may be considered for a future release. In the mean time, Ilja is going to pull this out of the pre-2.0 tree because making it work correctly will be far too painful at this time. ---------------------------------------------------------------------- danweber - 25-Aug-04 17:43 CEST ---------------------------------------------------------------------- If you are using exim, you can create a modified userforward router and a filter rules set to implement this. I have done so in the following. userforward: debug_print = "R: userforward for [EMAIL PROTECTED]" driver = redirect domains = +local_domains local_part_suffix = +* local_part_suffix_optional user = mail file = /var/local/sieve/${local_part}.${domain} no_verify no_expn check_ancestor allow_filter directory_transport = address_directory file_transport = address_file pipe_transport = address_pipe reply_transport = address_reply skip_syntax_errors syntax_errors_to = [EMAIL PROTECTED] syntax_errors_text = \ This is an automatically generated message. An error has\n\ been found in your .forward file. Details of the error are\n\ reported below. While this error persists, you will receive\n\ a copy of this message for every message that is addressed\n\ to you. If your .forward file is a filter file, or if it is\n\ a non-filter file containing no valid forwarding addresses,\n\ a copy of each incoming message will be put in your normal\n\ mailbox. If a non-filter file contains at least one valid\n\ forwarding address, forwarding to the valid addresses will\n\ happen, and those will be the only deliveries that occur. and for the filter set if $local_part_suffix: begins "+" then pipe "/usr/sbin/dbmail-smtp -u <place user here> -m ${substr_1_1000:$local_part_suffix}" endif ---------------------------------------------------------------------- niblettda - 06-Dec-04 15:13 CET ---------------------------------------------------------------------- I've created a patch for DBMail-2.0.1 that does this. I'm not quite sure why this was such confusion. My understanding of how this feature should work is: MTA accepts [EMAIL PROTECTED] (MTA should not care) MTA passes to LDA (DBMail). LDA is responsible to verify that Folder exists. If folder exists, deliver to that folder. If folder does not exist, check allowed create list. If folder not on allowed create list, deliver message to INBOX. The reason I wrote this patch is that I too wanted to use Amavisd-new with SpamAssassin. I did find that I also had to hack/patch Amavis to add the folder extension at the tag2 level not just the kill level. I'll be offering that patch to Amavisd-new as well. If anyone is interested in my patches or help in setting up Postfix + Amavisd-new + SpamAssassin + PostgreSQL + DBMail I'll offer what I can. I should mention that at this time my patch is only for dbmail-lmtpd. I do intend to add it to dbmail-smtpd as well. I can be contacted by niblettda at gru.com edited on: 06-Dec-04 15:11 edited on: 06-Dec-04 15:13 ---------------------------------------------------------------------- aaron - 07-Dec-04 01:14 CET ---------------------------------------------------------------------- Please do post your patches! Thanks for taking the time to figure out how to hack this in, I'm interested in your approach (although the next paragraph probably slams it) and if you have a mechanism that is safe against "mailbox spam". We tried doing it as a mid-level hack, and it was a mess. Your hack sounds like it's at a top-level (dbmail-lmtpd or dbmail-smtp), which might work well but is probably bad from an architecture point of view. I have a fix in mind for getting the right information down into db_find_create_mailbox, which is the function that really needs to know (and indeed takes a flag exactly for this purpose) -- just that there's one layer above it, insert_message, that hides the mailbox complexity but for this feature needs to share it. ---------------------------------------------------------------------- niblettda - 07-Dec-04 19:32 CET ---------------------------------------------------------------------- Aaron, I uploaded the patch. I wasn't sure on the policy on things like that. Here is the basic run down on what I did. 1) Added "remove_folder()" function to misc.c. It's basically the reverse of "find_bounded()". This was needed so that in lmtp.c I could strip out the folder before the name was checked. 2) In lmtp.c I added the remove_folder after the find_bounded for the RCPT <name>. 3) I added a find_bounded with '+', '@' to give me the folder name. This could probably be more efficient, but I didn't want to deal with allocating memory and freeing it correctly. I then add the folder to the dsn.mailbox of the current address. This seemed like the best place to add that IMHO. 4) The 1st of the big hacks is in sort.c where I added an else statement to the if (mailbox == NULL). What I did was do a call to db_findmailbox to see if the mailbox exists. From that point I compare the requested mailbox to a list (in this case SPAM) which are allowed auto-create boxes. 5) Second massive hack is that I made a change in the db.c function so use the POSIX matching to deal with case sensitivity issues. Things that I need to work on and I am working on, are: 1) I need to figure out how to specify in the config file, a list of auto-create mailboxes. I'm not sure how I'll be able to do that. First stab will be a simple space delimited list, and a strcmp like I do now. This could be better, but until I understand the ReadConfig features better it will have to do. 2) I'd like a better way to deal with the case sensitivity issues with the database. For example, I name my spam box "Spam", but amavis delivers it (by choice) as "SPAM". I don't want a new box created for me, nor do I think it should be. Once I can sort that out it should remove the need to alter the db.c file. 3) I need to fix the portion in sort.c since I just found a bug. :) If I don't find the mailbox I need to make sure it's set to INBOX. Oops. 4) Create a README for the patch. As for your thought about putting the changes in find_create_mailbox, I disagree with your approach. I very much so think this should be a top level change. I can certainly see a seperate function that maybe does all this, and my v2 of the patch will probably do just that. To me, there is no point to reopen everything at the mid-level when all the information is present at the top level. It was very easy to access the mailbox, address and dsn struct, so why not use that to your advantage. Plus at the top level is the only place you should ever need to care about username+folder. That all needs to be decided before you deliver the message. After that point it's all up to IMAP and the user. The problem with why I had to edit sort.c is that I was having a really hard time getting ahold of the useridnr at that point in lmtp.c. Really all I needed was the useridnr so that I could use the db_findmailbox function. If there is a better way to see if the mailbox exists for the username already then this entire thing should be able to be consolidated down to one function. If I understood how that struct part worked and received I'd be much better off. Obviously I don't completely understand the code as well as you all, so there may be other reasons for your choices. I'd be happy to work with you on this any way you like, since this is a business requirement for me. I haven't done C in years so feel free to tighten/correct any code that I have. It took me nearly a day to remember what a pointer really was. :) Thanks! edited on: 07-Dec-04 19:32 ---------------------------------------------------------------------- niblettda - 21-Dec-04 15:29 CET ---------------------------------------------------------------------- I have created a new patch that works better than my first. Please use the dbmail-folders.patch. In this patch I no longer edited the sort/sort.c file and was able to contain all my major changes in misc.c. There I have created a function that determines if the user's folder exists, or can be created. Also in this version I've added an option in dbmail.conf where you can specify what folders can be created by DBMail (Spam, Trash, Xyz, etc). If the requested folder doesn't exist or is in the create list, the Email will be delivered to the INBOX. I hope later this week to have SMTPD working the same way. ---------------------------------------------------------------------- Wolfram - 03-Feb-05 19:01 CET ---------------------------------------------------------------------- What comes to my mind here is using dbmail_acl for controlling mail submission to [EMAIL PROTECTED] This is afaics the purpose of the *post* right in RFC 2086 (IMAP4 ACL extension). So if the "anyone" user has post_flag set to 1 in dbmail_acl for a mailbox (other than INBOX), mail should be taken - otherwise put the mail into INBOX. just my 2c ---------------------------------------------------------------------- niblettda - 18-May-05 16:46 CEST ---------------------------------------------------------------------- I have updated my patch for DBMail v2.0.4. Nothing major changed, there was a simple rename of the my_free() function to dm_free() function. I have added the new patch to the attached files. In reference to Wolfram's comment about dbmail_acl I will admit that I don't fully understand that is going on with that table. If I understand the comment correctly, you can set the post_flag to 1 a folder will always be created. While this does sound like a more reasonable place to reference for folder creation, I still agree with aaron. With this set to 1 you could folder bomb an account and just create tons of folders. For this reason I think you still need to specify exactly WHAT folders can be created, that don't already exist. Bug History Date Modified Username Field Change ====================================================================== 23-Jul-04 23:37maXXmaster New Bug 24-Jul-04 03:38danweber Bugnote Added: 0000110 24-Jul-04 03:38danweber Priority normal => high 24-Jul-04 03:38danweber Status new => confirmed 24-Jul-04 17:05aaron Bugnote Added: 0000111 25-Jul-04 20:40maXXmaster Bugnote Added: 0000112 25-Jul-04 20:40maXXmaster Bugnote Edited: 0000112 25-Jul-04 20:44aaron Bugnote Added: 0000113 25-Jul-04 22:56maXXmaster Bugnote Added: 0000114 26-Jul-04 11:51maXXmaster Bugnote Added: 0000115 26-Jul-04 11:52maXXmaster Bugnote Edited: 0000115 26-Jul-04 13:00maXXmaster Bugnote Added: 0000116 26-Jul-04 16:15aaron Bugnote Added: 0000117 26-Jul-04 16:49maXXmaster Bugnote Added: 0000118 26-Jul-04 18:15aaron Bugnote Added: 0000119 26-Jul-04 18:22aaron Bugnote Added: 0000120 26-Jul-04 21:47maXXmaster Bugnote Added: 0000121 27-Jul-04 10:24aaron Bugnote Added: 0000122 27-Jul-04 10:28aaron Bugnote Added: 0000123 01-Aug-04 09:42maXXmaster Bugnote Added: 0000127 04-Aug-04 16:26ilja Bugnote Added: 0000138 05-Aug-04 07:56aaron Bugnote Added: 0000144 05-Aug-04 07:56aaron Assigned To => aaron 05-Aug-04 07:56aaron Resolution open => fixed 05-Aug-04 07:56aaron Status confirmed => resolved 05-Aug-04 16:00ilja Status resolved => closed 05-Aug-04 22:34maXXmaster Bugnote Added: 0000150 05-Aug-04 22:34maXXmaster Resolution fixed => reopened 05-Aug-04 22:34maXXmaster Status closed => feedback 06-Aug-04 01:29aaron Bugnote Added: 0000151 06-Aug-04 10:28ilja Bugnote Added: 0000152 06-Aug-04 10:29ilja Bug Monitored: ilja 06-Aug-04 18:03aaron Bugnote Added: 0000153 07-Aug-04 07:14maXXmaster Bugnote Added: 0000154 08-Aug-04 06:59aaron Bugnote Added: 0000155 08-Aug-04 13:17maXXmaster Bugnote Added: 0000156 12-Aug-04 11:58aaron Bugnote Added: 0000157 12-Aug-04 11:58aaron Status feedback => acknowledged 19-Aug-04 11:39ilja Priority high => normal 25-Aug-04 17:43danweber Bugnote Added: 0000206 06-Dec-04 15:10niblettda Bugnote Added: 0000424 06-Dec-04 15:10niblettda Bug Monitored: niblettda 06-Dec-04 15:11niblettda Bugnote Edited: 0000424 06-Dec-04 15:13niblettda Bugnote Edited: 0000424 07-Dec-04 01:14aaron Bugnote Added: 0000427 07-Dec-04 19:05niblettda File Added: dbmail-spam.patch 07-Dec-04 19:30niblettda Bugnote Added: 0000432 07-Dec-04 19:32niblettda Bugnote Edited: 0000432 21-Dec-04 15:16niblettda File Added: dbmail-folders.patch 21-Dec-04 15:29niblettda Bugnote Added: 0000457 03-Feb-05 19:01Wolfram Bugnote Added: 0000576 18-May-05 16:46niblettda Bugnote Added: 0000704 ======================================================================