[Dovecot] CATENATE/literal8 issue
Using 2.2.2, I see this:
C: 6 APPEND INBOX (\seen) 16-May-2013 22:05:14 -0600 CATENATE (URL
/INBOX;UIDVALIDITY=1255685337/;UID=48812/;SECTION=HEADER TEXT ~{40}
S: 6 NO [UNKNOWN-CTE] Binary input allowed only when the first part is binary.
Why is there this limitation? It seems to me that CATENATE is
confusing the content-type encoding of the data/part itself with the
encoding of the IMAP literal.
A literal 8 is nothing more than a series of OCTET's that *may*
contain nulls, but not necessarily. i.e., in the above example the 40
octets of data are US-ASCII text, which is perfectly acceptable to
send as a literal8. (Client rationale: If BINARY exists on the
server, we don't bother to scan IMAP literal's for null data -- we
just send them as literal8's. It's an optimization that I would hate
to get rid of.)
michael
Re: [Dovecot] Sieve/pigeonhole with Exim and Dovecot LDA
On 21/05/13 01:38, Gedalya wrote: On 05/20/2013 07:37 PM, Sebastian Arcus wrote: That's an interesting one. I've been running several sites for a few years now with exim in smart relay - without connection_max_messages = 1 - and had no problems so far. Maybe it's because only few lan clients are involved - or I've been lucky so far :-) The point is that the transport, and then in turn the authenticator are meant to potentially process more than one message in a single connection. What is the meaning of $sender_address or $header_*? The sender of which message? The headers from which message? If you do anything message-specific at this stage, you need to set this so only one message is sent per connection, so that message-specific variables can be meaningful. That makes sense - I was just surprised it hasn't bitten me in the back so far. I'll amend the configs to process one message at a time in the future. I can only assume so far it was using the sender address of the first message to be processed?
Re: [Dovecot] Outlook 2013 - mounting folders with XLIST
Hello, i do some tests with dovecot 2.1.7 and activated a default special-use config. To get it work with outlook 2013, i also added XLIST to imap capability string. basically this is working. is somebody also using special-use folders successful with outlook 2013? yes, testet a few times, works fine, without Junk folder , cause outlook wants this handled by it own, but i ve seen reg patches to change this My serverside setup now is completed. I did a lot of tests last weeks and experienced some strange behaviour of some clients. Outlook 2013 is only working when adding XLIST manually to imap_capability imap_capability = +XLIST This is because outlook 2013 not supports rfc 6154 but the deprecated XLIST standard invented by google. So the problem with junkfolder is not a bug in Outlook 2013, in rfc 6154 spamfolder is tagged by \Junk, in XLIST standard \Spam is used. I did see that when using a gmailaccount in outlook 2013. Adding XLIST capability to dovecot seems to be a problem for other Clients. k9 is able to work with rfc 6154 servers. But if k9 finds XLIST and SPECIAL-USE together in capabilitystring it seems to prefer XLIST requests. Because of dovecot is accepting XLIST requests, but outputs rfc 6154 details, k9 seems to be confused and dont finds special Folders. rfc 6154 is similar but not identical to XLIST. If you dont test with really individual foldernames, you get tricked by clients behaviour. I looked around and the most imap-servers of hosting companies etc. provide XLIST feature, Special-USE unfortunately only a few. So i did now some changes to dovecot sources on my own. I added \Spam as allowed special-use attribute and created a new function for XLIST Requests. So if XLIST is requested, Clients gets lines of output with XLIST and \Junk is replaced with \Spam. So all is done in the code and i dont need to change my userdb-config. Testing this server with different clients was successful. All of them did find their special folders and worked fine, outlook 2013 also finds spamfolder now. So this changes contribute to consolidate a deprecated standard but i have to find a way where all users can benefit from new features. This is not a request to change something in dovecot, this is a call to decision makers to support one rfc Standard. Hajo
Re: [Dovecot] CATENATE/literal8 issue
On 21.5.2013, at 9.40, Michael M Slusarz slus...@curecanti.org wrote:
Using 2.2.2, I see this:
C: 6 APPEND INBOX (\seen) 16-May-2013 22:05:14 -0600 CATENATE (URL
/INBOX;UIDVALIDITY=1255685337/;UID=48812/;SECTION=HEADER TEXT ~{40}
S: 6 NO [UNKNOWN-CTE] Binary input allowed only when the first part is binary.
Why is there this limitation? It seems to me that CATENATE is confusing the
content-type encoding of the data/part itself with the encoding of the IMAP
literal.
A literal 8 is nothing more than a series of OCTET's that *may* contain
nulls, but not necessarily. i.e., in the above example the 40 octets of data
are US-ASCII text, which is perfectly acceptable to send as a literal8.
(Client rationale: If BINARY exists on the server, we don't bother to scan
IMAP literal's for null data -- we just send them as literal8's. It's an
optimization that I would hate to get rid of.)
Well, the problem is that if it does contain NULs, the MIME part needs to be
converted to something that doesn't. And to do that it needs to modify the
previous header, which with current code was already read.. So to fix that it
would need to read the whole message into a temporary file before actually
saving it, which makes performance worse for the normal case..
Or are you saying that the error is fine if the text contains NULs, but simply
should be allowed as long as it doesn't?
Re: [Dovecot] search and UTF-8 normalization forms (NFD)
On Mi, 15 Mai 2013, Timo Sirainen wrote: On 11.5.2013, at 18.13, Florian Zeitz florob at babelmonkeys.de wrote: So... I had a look at this. Turns out that the current implementation of Unicode decomposition (Step 2(b) in i;unicode-casemap) in Dovecot is broken. It only handles decomposition properties that include a tag. I've attached a hg export that fixes this. Thanks, added to v2.1 and v2.2 hg. Thanks, but there seems to be still a problem left. Sender search yields all Krüger mails without fts_lucene. But with fts_lucene enabled - and files in lucene-indexes/ existing - it's not. (If I delete the lucene-index files and search for sender, result is correct - but only until they are recreated.) Lutz
Re: [Dovecot] dsync-2.2.2 incorrectly synchronizes subscription status of deleted mailbox
** On Monday 20 of May 2013 17:33:44 Timo Sirainen wrote: On Mon, 2013-05-20 at 15:29 +0200, Karol Jurak wrote: It seems that dsync-2.2.2 doesn't correctly synchronize subscription status of a deleted mailbox. Thanks, fixed: http://hg.dovecot.org/dovecot-2.2/rev/9878986a028d I performed the test and subscription state of a deleted mailbox is still incorrectly synchronized: the mailbox is being subscribed on the first server (the one it was originally deleted on) instead of being unsubscribed on the other. One thing changed however: a subsequent attempt to unsubscribe this mailbox on the first server is now correctly synchronized. Previously every attempt to unsubscribe a nonexistent mailbox was being reverted by dsync. -- Karol Jurak
Re: [Dovecot] CATENATE/literal8 issue
Quoting Timo Sirainen t...@iki.fi:
On 21.5.2013, at 9.40, Michael M Slusarz slus...@curecanti.org wrote:
Using 2.2.2, I see this:
C: 6 APPEND INBOX (\seen) 16-May-2013 22:05:14 -0600 CATENATE
(URL /INBOX;UIDVALIDITY=1255685337/;UID=48812/;SECTION=HEADER
TEXT ~{40}
S: 6 NO [UNKNOWN-CTE] Binary input allowed only when the first part
is binary.
Why is there this limitation? It seems to me that CATENATE is
confusing the content-type encoding of the data/part itself with
the encoding of the IMAP literal.
A literal 8 is nothing more than a series of OCTET's that *may*
contain nulls, but not necessarily. i.e., in the above example the
40 octets of data are US-ASCII text, which is perfectly acceptable
to send as a literal8. (Client rationale: If BINARY exists on the
server, we don't bother to scan IMAP literal's for null data -- we
just send them as literal8's. It's an optimization that I would
hate to get rid of.)
Well, the problem is that if it does contain NULs, the MIME part
needs to be converted to something that doesn't. And to do that it
needs to modify the previous header, which with current code was
already read..
Is altering the header something that BINARY/CATENATE is allowed to
do? Especially regarding the header. I know there is language about
the server changing the CTE, but this is potentially troubling since
cryptographic signatures may rely on the header text. Changing things
will break the message.
I can see the server altering the body text to match the header. But
I think the reverse is bothersome.
Or are you saying that the error is fine if the text contains NULs,
but simply should be allowed as long as it doesn't?
This. As mentioned before, it seems the code is simply assuming that
the text part contains NULs without ever checking it. My reading of
the literal8 is that there is no requirement that NULs MUST exist in
the string.
In our code, the append data is often from code that the IMAP library
doesn't have access to. So at APPEND time, it is unaware whether the
data contains NUL or not - it just has a blob of data and a length.
If BINARY exists, it is much easier for us to simply send as literal8
and stream the data - no extra overhead is needed on our side. Since
each individual byte need to be handled by the server as it comes in,
it seems much more efficient to do NUL checking there.
michael
Re: [Dovecot] CATENATE/literal8 issue
On 21.5.2013, at 21.24, Michael M Slusarz slus...@curecanti.org wrote: Or are you saying that the error is fine if the text contains NULs, but simply should be allowed as long as it doesn't? This. As mentioned before, it seems the code is simply assuming that the text part contains NULs without ever checking it. My reading of the literal8 is that there is no requirement that NULs MUST exist in the string. In our code, the append data is often from code that the IMAP library doesn't have access to. So at APPEND time, it is unaware whether the data contains NUL or not - it just has a blob of data and a length. If BINARY exists, it is much easier for us to simply send as literal8 and stream the data - no extra overhead is needed on our side. Since each individual byte need to be handled by the server as it comes in, it seems much more efficient to do NUL checking there. It's not just about NUL. It's also about if plain LFs can be converted to CRLFs. Anyway .. the BINARY APPEND converts only the MIME parts that you send with Content-Transfer-Encoding: binary. Are you sending such header to Dovecot? If not, there's actually no difference to a regular APPEND from Dovecot's point of view (I think). If a non-binary MIME part contains NUL, what is Dovecot supposed to do? Change it to some other character? Fail the APPEND? Should there be a difference between how literal vs literal8 is handled in such case?
Re: [Dovecot] CATENATE/literal8 issue
Quoting Timo Sirainen t...@iki.fi:
Anyway .. the BINARY APPEND converts only the MIME parts that you
send with Content-Transfer-Encoding: binary. Are you sending such
header to Dovecot?
I don't think so. I noticed the CATENATE error when I was stripping a
simple text/html part out of a multipart/alternative message. The
master message header has a single MIME header:
Content-Type: multipart/alternative;
boundary=WPFVNCCY4GPWDK6HNJXHWWE7J94BSS
For the record, here's the entire transaction, along with the fallback
APPEND w/out using literal8 that was successful on the identical data:
C: 6 APPEND INBOX (\seen) 16-May-2013 22:05:14 -0600 CATENATE (URL
/INBOX;UIDVALIDITY=1255685337/;UID=48812/;SECTION=HEADER TEXT ~{40}
S: 6 NO [UNKNOWN-CTE] Binary input allowed only when the first part is binary.
C: 8 APPEND INBOX (\seen) 16-May-2013 22:05:14 -0600 CATENATE (URL
/INBOX;UIDVALIDITY=1255685337/;UID=48812/;SECTION=HEADER TEXT {40+}
C: [LITERAL DATA: 40 bytes]
C: URL /INBOX;UIDVALIDITY=1255685337/;UID=48812/;SECTION=1.MIME URL
/INBOX;UIDVALIDITY=1255685337/;UID=48812/;SECTION=1 TEXT {40+}
C: [LITERAL DATA: 40 bytes]
C: TEXT {113+}
C: [LITERAL DATA: 113 bytes]
C: TEXT {42+}
C: [LITERAL DATA: 42 bytes]
C: )
S: 8 OK [APPENDUID 1255685337 48885] Append completed.
If a non-binary MIME part contains NUL, what is Dovecot supposed to
do? Change it to some other character? Fail the APPEND? Should there
be a difference between how literal vs literal8 is handled in such
case?
I would say there is no doubt: fail the APPEND. It should be the
client's responsibility to correctly format the data.
I appreciate that Dovecot does its best to try to Do The Right Thing
(Cyrus is much stricter about input, for example). But at some point
us client authors have to be at least somewhat competent, and it is
not asking to much for us to accept that GIGO.
michael
Re: [Dovecot] Dovecot 2.2.1 LDA and sieve (lack of) errors
On 05/18/2013 07:32 PM, Jim McNamara wrote: On 05/18/2013 05:35 PM, Stephan Bosch wrote: Now this is interesting - I do have Pigeonhole and sieve enabled according to doveconf -n, but the only mention of sieve is as a subdirectory of the user's home, no mention of the sieve module at all. I do see that Pigeonhole 0.4.0 is enabled. Also, there have only been 0 mentions of lda in the current dovecot log, and that log is 6 hours old. I'll try running dovecot-lda manually after checking out the man page. Thanks again for the help! Turns out again it was a qmail issue - the dovecot logs didn't contain any sieve info because for my install's point of view, sieving was the last step of qmail send, so there was a subdirectory of difference. Here's the massive verbose log of one message, you see the plugin loaded and everything is finally behaving. Thanks again for the help and insight Stephan, I appreciate your time. If you're in NYC sometime, I owe you a drink! 2013-05-21 08:20:40.761051500 delivery 129452: success: May_21_08:20:40_lda:_Debug:_Loading_modules_from_directory:_/usr/local/lib/dovecot/May_21_08:20:40_lda:_Debug:_Module_loaded:_/usr/local/lib/dovecot/lib10_quota_plugin.so/May_21_08:20:40_lda:_Debug:_Module_loaded:_/usr/local/lib/dovecot/lib90_sieve_plugin.so/May_21_08:20:40_lda:_Debug:_auth_input:_jim@domain.com_uid=89_gid=89_home=/home/vpopmail/domains/domain.com/jim_quota_rule=*:backend=20S/May_21_08:20:40_lda:_Debug:_Added_userdb_setting:_plugin/quota_rule=*:backend=20S/May_21_08:20:40_lda(j...@domain.com):_Debug:_Effective_uid=89,_gid=89,_home=/home/vpopmail/domains/domain.com/jim/May_21_08:20:40_lda(j...@domain.com):_Debug:_quota:_No_quota_setting_-_plugin_disabled/May_21_08:20:40_lda(j...@domain.com):_Debug:_Namespace_inbox:_type=private,_prefix=,_sep=.,_inbox=yes,_hidden=no,_list=yes,_subscriptions=yes_location=maildir:/home/vpopmail/domains/domain.com/jim/Maildir/May_21_08:20:40_lda(j...@domain.com):_Debug:_maildir++:_root=/home/vpopmail/domains/domain.com/jim/Maildir,_index=,_indexpvt=,_control=,_inbox=/home/vpopmail/domains/domain.com/jim/Maildir,_alt=/May_21_08:20:40_lda(j...@domain.com):_Debug:_quota:_No_quota_setting_-_plugin_disabled/May_21_08:20:40_lda(j...@domain.com):_Debug:_none:_root=,_index=,_indexpvt=,_control=,_inbox=,_alt=/May_21_08:20:40_lda(j...@domain.com):_Debug:_Destination_address:_jim@domain.com_(source:_user@hostname)/May_21_08:20:40_lda(j...@domain.com):_Debug:_sieve:_Pigeonhole_version_0.4.0_initializing/May_21_08:20:40_lda(j...@domain.com):_Debug:_sieve:_include:_sieve_global_dir_is_not_set;_it_is_currently_not_possible_to_include_`:global'_scripts./May_21_08:20:40_lda(j...@domain.com):_Debug:_sieve:_script_file_/home/vpopmail/domains/domain.com/jim/.sieve/dovecot.sieve_not_found/May_21_08:20:40_lda(j...@domain.com):_Debug:_sieve:_user's_script_~/.sieve/dovecot.sieve_doesn't_exist_(using_default_script_location_instead)/May_21_08:20:40_lda(j...@domain.com):_Debug:_sieve:_using_the_following_location_for_user's_Sieve_script:_/usr/local/etc/dovecot/sieve/default.sieve;name=main_script/May_21_08:20:40_lda(j...@domain.com):_Debug:_sieve:_loading_script_/usr/local/etc/dovecot/sieve/default.sieve;name=main_script/May_21_08:20:40_lda(j...@domain.com):_Debug:_sieve:_script_binary_/usr/local/etc/dovecot/sieve/default.svbin_successfully_loaded/May_21_08:20:40_lda(j...@domain.com):_Debug:_sieve:_binary_save:_not_saving_binary_/usr/local/etc/dovecot/sieve/default.svbin,_because_it_is_already_stored/May_21_08:20:40_lda(j...@domain.com):_Debug:_sieve:_executing_script_from_/usr/local/etc/dovecot/sieve/default.svbin/May_21_08:20:40_lda(j...@domain.com):_Info:_sieve:_msgid=e1uelxv-0004nf...@sub1.domain.com:_stored_mail_into_mailbox_'INBOX.nagios.folder2'/did_0+0+1/
[Dovecot] should dovecot store maildir files with CRLF or LF?
Hi. I've made a strange observation. When having Dovecot (at least) with maildir and moving (via IMAP) mail received by some client (Evolution 3.4) into it the following happens: Regardless of whether the mail was originally(!) set with CRLF or LF (i.e. when I use netcat to submit the plain SMTP to the relaying MTA). When the client (Evolution) had received the mail via POP3 before moving it via IMAP into Dovecot... then the maildir file within dovecot is all LF. When the client however received it via IMAP, before in turn moving it on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF, i.e. the body is CRLF, the headers are terminated... Well the actually bug here is probably in Evolution (as so many others... o.O)... but I wondered... what is Dovecot expected to write files? Platform end-of-line markers (i.e. LF in case of UNIX) or always network end-of-line markers (CRLF)? Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] should dovecot store maildir files with CRLF or LF?
I'm not sure if this has any bearing on what you reported here, but I do this: fetchmail-procmail-dovecot-Claws-Mail When I open certain emails, the empty lines between paragraphs are missing. If this is part of what you're reporting, I can pay more attention as to which emails display this anomaly. Thanks, SteveT Steve Litt* http://www.troubleshooters.com/ Troubleshooting Training * Human Performance On Wed, 22 May 2013 00:45:49 +0200 Christoph Anton Mitterer cales...@scientia.net wrote: Hi. I've made a strange observation. When having Dovecot (at least) with maildir and moving (via IMAP) mail received by some client (Evolution 3.4) into it the following happens: Regardless of whether the mail was originally(!) set with CRLF or LF (i.e. when I use netcat to submit the plain SMTP to the relaying MTA). When the client (Evolution) had received the mail via POP3 before moving it via IMAP into Dovecot... then the maildir file within dovecot is all LF. When the client however received it via IMAP, before in turn moving it on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF, i.e. the body is CRLF, the headers are terminated... Well the actually bug here is probably in Evolution (as so many others... o.O)... but I wondered... what is Dovecot expected to write files? Platform end-of-line markers (i.e. LF in case of UNIX) or always network end-of-line markers (CRLF)? Cheers, Chris.
Re: [Dovecot] should dovecot store maildir files with CRLF or LF?
At 12AM +0200 on 22/05/13 you (Christoph Anton Mitterer) wrote: I've made a strange observation. When having Dovecot (at least) with maildir and moving (via IMAP) mail received by some client (Evolution 3.4) into it the following happens: Regardless of whether the mail was originally(!) set with CRLF or LF (i.e. when I use netcat to submit the plain SMTP to the relaying MTA). Mail sent by SMTP has to have CRLF line endings. (Unless you're using BINARYMIME, but I don't think that's at all widely implemented yet.) If your mailserver accepts LF-only line endings it ought to translate them into CRLF before sending the message on. When the client (Evolution) had received the mail via POP3 before moving it via IMAP into Dovecot... then the maildir file within dovecot is all LF. What line endings is the POP server sending? Can you verify this without involving Evolution? Is the POP server Dovecot or something else? The original (djb) definition of Maildir assumed that messages would be written to the maildir with LF line endings, and both MTA and POP server would translate back to CRLF as needed. Dovecot (as a POP server) can deal with messages in either format, and should always return them to clients with CRLF. When the client however received it via IMAP, before in turn moving it on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF, i.e. the body is CRLF, the headers are terminated... IMAP is similar, in that line endings on the wire are always supposed to be CRLF; it's a little more complicated in that clients can also upload messages. If I APPEND a message with mixed line endings to a Maildir Dovecot mailbox, the message is written to the Maildir with LF-only endings and comes back over IMAP with CRLF-only. Can you confirm what is actually being sent over the wire? Ben
Re: [Dovecot] should dovecot store maildir files with CRLF or LF?
On Wed, 2013-05-22 at 02:54 +0100, Ben Morrow wrote: Mail sent by SMTP has to have CRLF line endings. (Unless you're using BINARYMIME, but I don't think that's at all widely implemented yet.) If your mailserver accepts LF-only line endings it ought to translate them into CRLF before sending the message on. Sure, and I assume postfix does the later... but what's mandatory for SMTP isn't a standard/recommendation/best-practise/etc on what IMAP servers should do internally. What line endings is the POP server sending? Can you verify this without involving Evolution? Yeah... guess I should teach myself speaking POP3 as well ;) Is the POP server Dovecot or something else? no idea... that's the one of my ISP... The original (djb) definition of Maildir assumed that messages would be written to the maildir with LF line endings, and both MTA and POP server would translate back to CRLF as needed. Dovecot (as a POP server) can deal with messages in either format, and should always return them to clients with CRLF. When the client however received it via IMAP, before in turn moving it on via IMAP into Dovecot, then the maildir file is mixed CRLF and LF, i.e. the body is CRLF, the headers are terminated... IMAP is similar, in that line endings on the wire are always supposed to be CRLF; it's a little more complicated in that clients can also upload messages. If I APPEND a message with mixed line endings to a Maildir Dovecot mailbox, the message is written to the Maildir with LF-only endings and comes back over IMAP with CRLF-only. IIRC such mixing is forbidding by the most recent RFC defining the format of internet mail messages... neither CR nor LF is allowed to exist (which was allowed to in the old standards and then didn't mean a newline, but rather the character CR respectively LF for itself). This is also the reason why I wonder a bit what Dovecot is doing, cause if it's compliant, then the outside world should always only see CRLF now, right? = But then it makes no sense to store mixed CRLF / LF / CR, if a buggy client presents it with that via IMAP. = It does IMHO however make sense to consider whether it may store mails in LF-only(! no mixing)... when this is the platform's native end-of-line marker. = On the other hand; I'd prefer to have this homogenous... so at least when dovecot get's new mail via IMAP, I would recommend it should use generally CRLF and - one would need to think about the following more carefully - convert and single LF / CR to it. Whether it can/should do such conversation with mail picked up from new/ respectively the LDA/MDA... is another topic... but if possible, I'd do so as well. Timo, if you read along, what do you think? Can you confirm what is actually being sent over the wire? You mean when Dovecot re-exports the stuff via IMAP? Haven't checked that yet.. Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
