A BUGNOTE has been added to this bug.
======================================================================
http://www.dbmail.org/mantis/bug_view_advanced_page.php?bug_id=0000057
======================================================================
Reported By: maXXmaster
Assigned To: aaron
======================================================================
Project: DBMail
Bug ID: 57
Category: PIPE delivery (dbmail-smtp)
Reproducibility: N/A
Severity: feature
Priority: normal
Status: acknowledged
======================================================================
Date Submitted: 23-Jul-04 23:37 CEST
Last Modified: 03-Feb-05 19:01 CET
======================================================================
Summary: direct mailforwarding with [EMAIL PROTECTED]
Description:
it doesn't seem that there is a way to store "special" mails like spam to a
different folder than INBOX when pipeing the message to dbmail-smtp. i'm
not sure if it is an imap-standard, but many imapservers allow you to send
a message to [EMAIL PROTECTED]
is there a way to realise that?
======================================================================
----------------------------------------------------------------------
danweber - 24-Jul-04 03:38 CEST
----------------------------------------------------------------------
Yes we need this feature added so we can start using tmda with exim too.
----------------------------------------------------------------------
aaron - 24-Jul-04 17:05 CEST
----------------------------------------------------------------------
There's a command line option to specify an alternate mailbox along with
the destination address:
dbmail -m mailbox {-t [headerfield] | -d [emailaddress]}
If there's a way to make this work with Exim, it would be much easier to
document such a method rather than adding new support for the
username+mailbox syntax.
----------------------------------------------------------------------
maXXmaster - 25-Jul-04 20:40 CEST
----------------------------------------------------------------------
i actually wanted to use it with amavis-new which is able to append an
foldername after the mailadress (incase of spam or viruses).. i'll
probably have to find a way to make it work with the -m mailboxname
option,..
let's see ;o)
edited on: 25-Jul-04 20:40
----------------------------------------------------------------------
aaron - 25-Jul-04 20:44 CEST
----------------------------------------------------------------------
Ok, keep us posted. Interfacing with other parts of the mail system is,
quite naturally, a high priority :-)
Is there a published standard advocating for this hack, btw? Some de facto
rule book that we can follow if it turns out to be necessary to support
this syntax?
----------------------------------------------------------------------
maXXmaster - 25-Jul-04 22:56 CEST
----------------------------------------------------------------------
well, not really,.. at least i couldn't find one within the last hour. my
(probably) working solution is to use the following chain:
postfix (:25) -> amavis-new (:10024) -> postfix#2 (:10025) -> procmail (->
script which does the conversion from [EMAIL PROTECTED] to the dbmail-smtp
syntax you mentioned) -> dbmail
maybe i can write more about the necessary configurations tomorrow.
----------------------------------------------------------------------
maXXmaster - 26-Jul-04 11:52 CEST
----------------------------------------------------------------------
i tried some things and came up with quite a good solution.
i use a small (php)script insted of the dbmail-smtp program in postfix
(inside master.cf) to extract the +folder from the emailaddress and pipe
it to dbmail-smtp. the only problem is, that dbmail-smtp cannot add
messages to a special mailbox, if i want to forward them to an
emailaddress. it just works with direct usernames (user -u markus vs -d
[EMAIL PROTECTED])
so either i ask the dbmail-database which useralias i should use for
special mailaddresses (that is exactly what dbmail-smtp should actually
do), or some of you guys add a few lines to make dbmail-smtp work with
mailboxes AND emailaddresses or to make it work with [EMAIL PROTECTED]
any suggestions? =)
edited on: 26-Jul-04 11:52
----------------------------------------------------------------------
maXXmaster - 26-Jul-04 13:00 CEST
----------------------------------------------------------------------
> Is there a published standard advocating for this hack, btw?
> Some de facto rule book that we can follow if it turns out
> to be necessary to support this syntax?
yes, there is. i found an rfc which "specifies" the [EMAIL PROTECTED]
syntax.
it can be found here:
http://www.ietf.org/rfc/rfc3598.txt
i hope that is enough information to the syntax ;o)
----------------------------------------------------------------------
aaron - 26-Jul-04 16:15 CEST
----------------------------------------------------------------------
Thanks for the RFC link, I'll look into it. Indicentally, there's no
difference between -u username and -d address. The address lookup checks
usernames first... I'll start a thread on dbmail-dev about whether or not
we should keep both options as synonyms, or make them different again.
----------------------------------------------------------------------
maXXmaster - 26-Jul-04 16:49 CEST
----------------------------------------------------------------------
are these two options really the same? hmm, maybe in version 2.* but in
1.2.9 (which i am using) they are different. at least i get a message
telling me to use -u username and not -d address if i want to store
directly to a mailbox.
----------------------------------------------------------------------
aaron - 26-Jul-04 18:15 CEST
----------------------------------------------------------------------
That's correct, they were merged at a fairly deep level in the new delivery
chain. My thinking was twofold (and in fact, this should probably go into
an FAQ or RATIONALE file someplace)...
- It seems odd to have to add an alias to your own username in order to
support domain-local delivery without expansion, as is common when you use
"mail" on the command line.
- It seems really odd to have to add an alias to your own username when
your username is a full email address, as is common for folks running
multiple separate email domains from a single server.
----------------------------------------------------------------------
aaron - 26-Jul-04 18:22 CEST
----------------------------------------------------------------------
So basically, here's my take on this so far:
- [EMAIL PROTECTED] is a common and accepted format that we should
support.
It is possible that some people will not want to use such an expansion,
while many other would, and, therefore:
- we need another command line option to handle this syntax.
Because -u and -d have been merged at a very deep level in the delivery
chain, it would make sense then to use -d to support this syntax and -u to
not support this syntax. The changes can be made at a very high level,
merely by adding a check for anything bounded by "+" and "@", removing
that portion of the address and assigning it to be the mailbox
destination.
- as a significant UI mechanism, this would need to be added before 2.0
----------------------------------------------------------------------
maXXmaster - 26-Jul-04 21:47 CEST
----------------------------------------------------------------------
sounds good to me ;o)
a small sidenote:
i couldn'n find a good howto about settingup
dbmail+amavis-new(+spamassassin+clamav)+postfix so if there are more
people interested in that, i could try writing one.
----------------------------------------------------------------------
aaron - 27-Jul-04 10:24 CEST
----------------------------------------------------------------------
Check out the latest CVS and let me know if it is working as you would
expect.
----------------------------------------------------------------------
aaron - 27-Jul-04 10:28 CEST
----------------------------------------------------------------------
Oh, and I might mention that this is the 2.0 CVS. If you're running 1.2.x
you will have to use a script as a helper and you may be out of luck on
the mail folder thing.
----------------------------------------------------------------------
maXXmaster - 01-Aug-04 09:42 CEST
----------------------------------------------------------------------
so the mailfolder stuff will just work in 2.x ? =( as dbmail is working on
a small production-server i want(ed) to use the stable 1.2.x version. so
i'll probably have to use my helper-script ...
----------------------------------------------------------------------
ilja - 04-Aug-04 16:26 CEST
----------------------------------------------------------------------
So, does it all work like it should? If so, we can close this bug :)
----------------------------------------------------------------------
aaron - 05-Aug-04 07:56 CEST
----------------------------------------------------------------------
Just ran it through its paces, and it works as expected. Neat new feature!
Now if Mr. maXXmaster would like to write README.amavisnew...
----------------------------------------------------------------------
maXXmaster - 05-Aug-04 22:34 CEST
----------------------------------------------------------------------
it is possible to create unlimited folders by sending messages to
[EMAIL PROTECTED] that is not really good, in case of mailbombs.
wouldn't it be better to store mails in the inbox if the folder doesn't
exist?
----------------------------------------------------------------------
aaron - 06-Aug-04 01:29 CEST
----------------------------------------------------------------------
Good point, and one that isn't handled with the little-m and big-M
options... medium-Mm for providing a mailbox in case the one specified
does not exist.
It may actually be a really big problem that someone could specify a
mailbox from outside of your mail system. This syntax is something that
DBMail needs to understand from upstream (amavis, exim) but which needs to
be filtered from the outside world (perhaps by exim or postfix before
initial delivery).
Creating mailboxes on the fly was a feature that I added to simply adding
users and to enable users to be authenticated from LDAP without requiring
additional steps to enable each user in DBMail. However, now we have a
situation where the mailbox can be specified by an external sender.
Probably a very bad thing.
----------------------------------------------------------------------
ilja - 06-Aug-04 10:28 CEST
----------------------------------------------------------------------
Can't we change this easily?
We can still let the db_find_create_mailbox() function create the mailbox,
but only when its name is "INBOX".
The change to db_find_create_mailbox() is easily made. Just add a
if (strcmp(name, "INBOX") == 0)
to the function and set mailbox_idnr to 0 is it fails.
There are three places in the sources where db_find_create_mailbox() is
used.
One is in db.c, in db_insert_message(). This use does not have to be
changed, because the value of mailbox_idnr is checked later on.
There are two places in sort/sort.c where the function is used. These will
have to be changed.
Aaron, sort/sort.c is your code, do you want to take a shot at this?
Ilja
----------------------------------------------------------------------
aaron - 06-Aug-04 18:03 CEST
----------------------------------------------------------------------
I'm not a big fan of an INBOX-only solution; I'd much rather be able to
ensure that +mailbox addresses cannot arrive from across the network. The
question I have now is if there's a difference between:
To: [EMAIL PROTECTED] -> Postfix -> Amavis -> DBMail ([EMAIL PROTECTED], into
box)
To: [EMAIL PROTECTED] -> Postfix -> Amavis (+box) -> DBMail ([EMAIL PROTECTED],
into
box)
DBMail most likely would have no way to cleanly separate the two
situations, so either we have to limit what DBMail can do (INBOX only,
ugh) or make sure that somebody is disallowing incoming messages of this
format (either Postfix or Amavis, if there's a way to have them strip out
the +box detail).
----------------------------------------------------------------------
maXXmaster - 07-Aug-04 07:14 CEST
----------------------------------------------------------------------
i tried finding config-examples (at least for postix) to switch-off "plus
detail" ([EMAIL PROTECTED]) but couldn't find anything. maybe it'S just too
early, but i think it would be a good setting for the config-file!
.) to enable all +folder
.) just to allow storing to existing folders (otherwise INBOX)
.) or to disable it at all.
would that work for you?
----------------------------------------------------------------------
aaron - 08-Aug-04 06:59 CEST
----------------------------------------------------------------------
This is pretty much an intractable problem right now. There only easy
solution is to find a way to trash the +folder part immediately from the
outside SMTP server.
The other three solutions are:
- comment out the code and put this off until later.
- provide a mechanism to specify a default mailbox to use in case the
provided one does not exist (the default mailbox would be created if not
found, to prevent the corner case of a nonexistant INBOX, for example).
- disable auto mailbox creation altogether, except a hardcoded INBOX.
A few months ago, I started to write code for the second situation,
apparently with a very good insight that we'd want auto mailbox creation
be option in some cases -- that code requires a number of deep changes.
I'm confident that they could be debugged within the coming 2.0rc8, but
I'd rather not have that pressure.
Disabling auto mailbox creation altogether isn't a very good option
because we do need it for LDAP users or perhaps if the user somehow
deletes their own INBOX folder entry. Hardcoding an INBOX exception
strikes me as quite ugly.
Even if we did disable this address syntax, someone who wants to work with
TMDA or Amavis-new would have to write a script that used the dbmail-smtp
-m "mailbox" option, which is vulnerable to exactly the same attack. So it
would be the mail admin hanging themself instead of DBMail doing it for
them.
So that leaves us with one good option and one crumby one:
- figure out how to filter this at the MTA.
- go for more new code.
Bummer.
----------------------------------------------------------------------
maXXmaster - 08-Aug-04 13:17 CEST
----------------------------------------------------------------------
it really seems to be a tricky problem. as a wrote before. i couldn't find
a way to tell postfix to strip-off the +folder part (still i believe there
is a way). nevertheless i think it is a good start to be able to work with
amavis the way you implemented it now. (as long as 1.2.9 is the officially
stable version it won't be a problem to leave it the way it is right now)
if there might be a better solution in rc9 i welcome it.
----------------------------------------------------------------------
aaron - 12-Aug-04 11:58 CEST
----------------------------------------------------------------------
Changing the status to acknowledged, which is what we're using for things
that may be considered for a future release. In the mean time, Ilja is
going to pull this out of the pre-2.0 tree because making it work
correctly will be far too painful at this time.
----------------------------------------------------------------------
danweber - 25-Aug-04 17:43 CEST
----------------------------------------------------------------------
If you are using exim, you can create a modified userforward router and a
filter rules set to implement this. I have done so in the following.
userforward:
debug_print = "R: userforward for [EMAIL PROTECTED]"
driver = redirect
domains = +local_domains
local_part_suffix = +*
local_part_suffix_optional
user = mail
file = /var/local/sieve/${local_part}.${domain}
no_verify
no_expn
check_ancestor
allow_filter
directory_transport = address_directory
file_transport = address_file
pipe_transport = address_pipe
reply_transport = address_reply
skip_syntax_errors
syntax_errors_to = [EMAIL PROTECTED]
syntax_errors_text = \
This is an automatically generated message. An error has\n\
been found in your .forward file. Details of the error are\n\
reported below. While this error persists, you will receive\n\
a copy of this message for every message that is addressed\n\
to you. If your .forward file is a filter file, or if it is\n\
a non-filter file containing no valid forwarding addresses,\n\
a copy of each incoming message will be put in your normal\n\
mailbox. If a non-filter file contains at least one valid\n\
forwarding address, forwarding to the valid addresses will\n\
happen, and those will be the only deliveries that occur.
and for the filter set
if $local_part_suffix: begins "+"
then
pipe "/usr/sbin/dbmail-smtp -u <place user here> -m
${substr_1_1000:$local_part_suffix}"
endif
----------------------------------------------------------------------
niblettda - 06-Dec-04 15:13 CET
----------------------------------------------------------------------
I've created a patch for DBMail-2.0.1 that does this. I'm not quite sure
why this was such confusion. My understanding of how this feature should
work is:
MTA accepts [EMAIL PROTECTED] (MTA should not care)
MTA passes to LDA (DBMail).
LDA is responsible to verify that Folder exists.
If folder exists, deliver to that folder.
If folder does not exist, check allowed create list.
If folder not on allowed create list, deliver message to INBOX.
The reason I wrote this patch is that I too wanted to use Amavisd-new
with SpamAssassin. I did find that I also had to hack/patch Amavis
to add the folder extension at the tag2 level not just the kill level.
I'll be offering that patch to Amavisd-new as well.
If anyone is interested in my patches or help in setting up Postfix +
Amavisd-new + SpamAssassin + PostgreSQL + DBMail I'll offer what I can.
I should mention that at this time my patch is only for dbmail-lmtpd. I
do intend to add it to dbmail-smtpd as well.
I can be contacted by niblettda at gru.com
edited on: 06-Dec-04 15:11
edited on: 06-Dec-04 15:13
----------------------------------------------------------------------
aaron - 07-Dec-04 01:14 CET
----------------------------------------------------------------------
Please do post your patches! Thanks for taking the time to figure out how
to hack this in, I'm interested in your approach (although the next
paragraph probably slams it) and if you have a mechanism that is safe
against "mailbox spam".
We tried doing it as a mid-level hack, and it was a mess. Your hack sounds
like it's at a top-level (dbmail-lmtpd or dbmail-smtp), which might work
well but is probably bad from an architecture point of view. I have a fix
in mind for getting the right information down into
db_find_create_mailbox, which is the function that really needs to know
(and indeed takes a flag exactly for this purpose) -- just that there's
one layer above it, insert_message, that hides the mailbox complexity but
for this feature needs to share it.
----------------------------------------------------------------------
niblettda - 07-Dec-04 19:32 CET
----------------------------------------------------------------------
Aaron,
I uploaded the patch. I wasn't sure on the policy on things like that.
Here is the basic run down on what I did.
1) Added "remove_folder()" function to misc.c. It's basically the
reverse
of "find_bounded()". This was needed so that in lmtp.c I could strip out
the folder before the name was checked.
2) In lmtp.c I added the remove_folder after the find_bounded for the
RCPT <name>.
3) I added a find_bounded with '+', '@' to give me the folder name. This
could probably be more efficient, but I didn't want to deal with
allocating
memory and freeing it correctly. I then add the folder to the
dsn.mailbox of the current address. This seemed like the best place to
add that IMHO.
4) The 1st of the big hacks is in sort.c where I added an else statement
to
the if (mailbox == NULL). What I did was do a call to db_findmailbox to
see if the mailbox exists. From that point I compare the requested
mailbox
to a list (in this case SPAM) which are allowed auto-create boxes.
5) Second massive hack is that I made a change in the db.c function so
use
the POSIX matching to deal with case sensitivity issues.
Things that I need to work on and I am working on, are:
1) I need to figure out how to specify in the config file, a list of
auto-create mailboxes. I'm not sure how I'll be able to do that. First
stab will be a simple space delimited list, and a strcmp like I do now.
This could be better, but until I understand the ReadConfig features
better
it will have to do.
2) I'd like a better way to deal with the case sensitivity issues with
the
database. For example, I name my spam box "Spam", but amavis delivers it
(by choice) as "SPAM". I don't want a new box created for me, nor do I
think
it should be. Once I can sort that out it should remove the need to
alter
the db.c file.
3) I need to fix the portion in sort.c since I just found a bug. :) If I
don't find the mailbox I need to make sure it's set to INBOX. Oops.
4) Create a README for the patch.
As for your thought about putting the changes in find_create_mailbox, I
disagree with your approach. I very much so think this should be a top
level change. I can certainly see a seperate function that maybe does
all
this, and my v2 of the patch will probably do just that. To me, there is
no point to reopen everything at the mid-level when all the information
is
present at the top level. It was very easy to access the mailbox,
address
and dsn struct, so why not use that to your advantage. Plus at the top
level
is the only place you should ever need to care about username+folder.
That
all needs to be decided before you deliver the message. After that point
it's all up to IMAP and the user.
The problem with why I had to edit sort.c is that I was having a really
hard
time getting ahold of the useridnr at that point in lmtp.c. Really all I
needed was the useridnr so that I could use the db_findmailbox function.
If
there is a better way to see if the mailbox exists for the username
already
then this entire thing should be able to be consolidated down to one
function.
If I understood how that struct part worked and received I'd be much
better
off.
Obviously I don't completely understand the code as well as you all, so
there
may be other reasons for your choices. I'd be happy to work with you on
this
any way you like, since this is a business requirement for me.
I haven't done C in years so feel free to tighten/correct any code that I
have.
It took me nearly a day to remember what a pointer really was. :)
Thanks!
edited on: 07-Dec-04 19:32
----------------------------------------------------------------------
niblettda - 21-Dec-04 15:29 CET
----------------------------------------------------------------------
I have created a new patch that works better than my first. Please use
the
dbmail-folders.patch.
In this patch I no longer edited the sort/sort.c file and was able to
contain
all my major changes in misc.c. There I have created a function that
determines
if the user's folder exists, or can be created.
Also in this version I've added an option in dbmail.conf where you can
specify
what folders can be created by DBMail (Spam, Trash, Xyz, etc). If the
requested
folder doesn't exist or is in the create list, the Email will be
delivered
to the INBOX.
I hope later this week to have SMTPD working the same way.
----------------------------------------------------------------------
Wolfram - 03-Feb-05 19:01 CET
----------------------------------------------------------------------
What comes to my mind here is using dbmail_acl for controlling mail
submission
to [EMAIL PROTECTED]
This is afaics the purpose of the *post* right in RFC 2086 (IMAP4 ACL
extension).
So if the "anyone" user has post_flag set to 1 in dbmail_acl for a mailbox
(other than INBOX), mail should be taken - otherwise put the mail into
INBOX.
just my 2c
Bug History
Date Modified Username Field Change
======================================================================
23-Jul-04 23:37maXXmaster New Bug
24-Jul-04 03:38danweber Bugnote Added: 0000110
24-Jul-04 03:38danweber Priority normal => high
24-Jul-04 03:38danweber Status new => confirmed
24-Jul-04 17:05aaron Bugnote Added: 0000111
25-Jul-04 20:40maXXmaster Bugnote Added: 0000112
25-Jul-04 20:40maXXmaster Bugnote Edited: 0000112
25-Jul-04 20:44aaron Bugnote Added: 0000113
25-Jul-04 22:56maXXmaster Bugnote Added: 0000114
26-Jul-04 11:51maXXmaster Bugnote Added: 0000115
26-Jul-04 11:52maXXmaster Bugnote Edited: 0000115
26-Jul-04 13:00maXXmaster Bugnote Added: 0000116
26-Jul-04 16:15aaron Bugnote Added: 0000117
26-Jul-04 16:49maXXmaster Bugnote Added: 0000118
26-Jul-04 18:15aaron Bugnote Added: 0000119
26-Jul-04 18:22aaron Bugnote Added: 0000120
26-Jul-04 21:47maXXmaster Bugnote Added: 0000121
27-Jul-04 10:24aaron Bugnote Added: 0000122
27-Jul-04 10:28aaron Bugnote Added: 0000123
01-Aug-04 09:42maXXmaster Bugnote Added: 0000127
04-Aug-04 16:26ilja Bugnote Added: 0000138
05-Aug-04 07:56aaron Bugnote Added: 0000144
05-Aug-04 07:56aaron Assigned To => aaron
05-Aug-04 07:56aaron Resolution open => fixed
05-Aug-04 07:56aaron Status confirmed => resolved
05-Aug-04 16:00ilja Status resolved => closed
05-Aug-04 22:34maXXmaster Bugnote Added: 0000150
05-Aug-04 22:34maXXmaster Resolution fixed => reopened
05-Aug-04 22:34maXXmaster Status closed => feedback
06-Aug-04 01:29aaron Bugnote Added: 0000151
06-Aug-04 10:28ilja Bugnote Added: 0000152
06-Aug-04 10:29ilja Bug Monitored: ilja
06-Aug-04 18:03aaron Bugnote Added: 0000153
07-Aug-04 07:14maXXmaster Bugnote Added: 0000154
08-Aug-04 06:59aaron Bugnote Added: 0000155
08-Aug-04 13:17maXXmaster Bugnote Added: 0000156
12-Aug-04 11:58aaron Bugnote Added: 0000157
12-Aug-04 11:58aaron Status feedback => acknowledged
19-Aug-04 11:39ilja Priority high => normal
25-Aug-04 17:43danweber Bugnote Added: 0000206
06-Dec-04 15:10niblettda Bugnote Added: 0000424
06-Dec-04 15:10niblettda Bug Monitored: niblettda
06-Dec-04 15:11niblettda Bugnote Edited: 0000424
06-Dec-04 15:13niblettda Bugnote Edited: 0000424
07-Dec-04 01:14aaron Bugnote Added: 0000427
07-Dec-04 19:05niblettda File Added: dbmail-spam.patch
07-Dec-04 19:30niblettda Bugnote Added: 0000432
07-Dec-04 19:32niblettda Bugnote Edited: 0000432
21-Dec-04 15:16niblettda File Added: dbmail-folders.patch
21-Dec-04 15:29niblettda Bugnote Added: 0000457
03-Feb-05 19:01Wolfram Bugnote Added: 0000576
======================================================================
