Re: Local access to IMAP mailboxes
Aki Tuomi wrote: > > > > Is this that corrupted index case? It even causes the panic in the > > daemon. Can I fix the index or somehow reindex the mailbox? > > > > You can use doveadm force-resync -u vas "*" Thanks! -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
> On 28 September 2018 at 19:07 Victor Sudakov wrote:
>
>
> This time an Android IMAP client complains about the "unexpected end
> of the network stream" while opening a mailbox named "dss", and
> dovecot logs:
>
> Sep 28 23:01:34 admin dovecot: imap-login: Login: user=, method=PLAIN,
> rip=78.140.19.131, lip=95.179.154.55, mpid=85536, TLS,
> session=<6ITSkfB2JfNOjBOD>
> Sep 28 23:01:35 admin dovecot: Error: imap(vas)<85536><6ITSkfB2JfNOjBOD>:
> Corrupted record in index cache file
> /home/vas/Mail/.imap/dss/dovecot.index.cache: UID 56: Broken physical size in
> mailbox dss: read(/home/vas/Mail/dss) failed: Cached message size smaller
> than expected (8271 < 8282, box=dss, UID=56)
> Sep 28 23:01:35 admin dovecot: Error: imap(vas)<85536><6ITSkfB2JfNOjBOD>:
> Mailbox dss: UID=56: read(/home/vas/Mail/dss) failed: Cached message size
> smaller than expected (8271 < 8282, box=dss, UID=56) (FETCH BODY[2])
> Sep 28 23:01:35 admin dovecot: imap(vas)<85536><6ITSkfB2JfNOjBOD>: FETCH
> read() failed in=896 out=54723 deleted=0 expunged=0 trashed=0 hdr_count=15
> hdr_bytes= body_count=14 body_bytes=37063
> Sep 28 23:02:54 admin dovecot: imap-login: Login: user=, method=PLAIN,
> rip=78.140.19.131, lip=95.179.154.55, mpid=85560, TLS,
> session=
> Sep 28 23:02:55 admin dovecot: Error: imap(vas)<85560>:
> Corrupted record in index cache file
> /home/vas/Mail/.imap/dss/dovecot.index.cache: UID 27: Broken physical size in
> mailbox dss: read(/home/vas/Mail/dss) failed: Cached message size smaller
> than expected (4867 < 4878, box=dss, UID=27)
> Sep 28 23:02:55 admin dovecot: imap(vas)<85560>: Panic:
> file istream.c: line 315 (i_stream_read_memarea): assertion failed: (old_size
> <= _stream->pos - _stream->skip)
> Sep 28 23:02:55 admin dovecot: imap(vas)<85560>: Fatal:
> master: service(imap): child 85560 killed with signal 6 (core not dumped -
> https://dovecot.org/bugreport.html#coredumps - set service imap {
> drop_priv_before_exec=yes })
>
>
> Is this that corrupted index case? It even causes the panic in the
> daemon. Can I fix the index or somehow reindex the mailbox?
>
> --
> Victor Sudakov, VAS4-RIPE, VAS47-RIPN
> 2:5005/49@fidonet http://vas.tomsk.ru/
You can use doveadm force-resync -u vas "*"
Aki
Re: Local access to IMAP mailboxes
This time an Android IMAP client complains about the "unexpected end
of the network stream" while opening a mailbox named "dss", and
dovecot logs:
Sep 28 23:01:34 admin dovecot: imap-login: Login: user=, method=PLAIN,
rip=78.140.19.131, lip=95.179.154.55, mpid=85536, TLS,
session=<6ITSkfB2JfNOjBOD>
Sep 28 23:01:35 admin dovecot: Error: imap(vas)<85536><6ITSkfB2JfNOjBOD>:
Corrupted record in index cache file
/home/vas/Mail/.imap/dss/dovecot.index.cache: UID 56: Broken physical size in
mailbox dss: read(/home/vas/Mail/dss) failed: Cached message size smaller than
expected (8271 < 8282, box=dss, UID=56)
Sep 28 23:01:35 admin dovecot: Error: imap(vas)<85536><6ITSkfB2JfNOjBOD>:
Mailbox dss: UID=56: read(/home/vas/Mail/dss) failed: Cached message size
smaller than expected (8271 < 8282, box=dss, UID=56) (FETCH BODY[2])
Sep 28 23:01:35 admin dovecot: imap(vas)<85536><6ITSkfB2JfNOjBOD>: FETCH read()
failed in=896 out=54723 deleted=0 expunged=0 trashed=0 hdr_count=15
hdr_bytes= body_count=14 body_bytes=37063
Sep 28 23:02:54 admin dovecot: imap-login: Login: user=, method=PLAIN,
rip=78.140.19.131, lip=95.179.154.55, mpid=85560, TLS,
session=
Sep 28 23:02:55 admin dovecot: Error: imap(vas)<85560>:
Corrupted record in index cache file
/home/vas/Mail/.imap/dss/dovecot.index.cache: UID 27: Broken physical size in
mailbox dss: read(/home/vas/Mail/dss) failed: Cached message size smaller than
expected (4867 < 4878, box=dss, UID=27)
Sep 28 23:02:55 admin dovecot: imap(vas)<85560>: Panic: file
istream.c: line 315 (i_stream_read_memarea): assertion failed: (old_size <=
_stream->pos - _stream->skip)
Sep 28 23:02:55 admin dovecot: imap(vas)<85560>: Fatal:
master: service(imap): child 85560 killed with signal 6 (core not dumped -
https://dovecot.org/bugreport.html#coredumps - set service imap {
drop_priv_before_exec=yes })
Is this that corrupted index case? It even causes the panic in the
daemon. Can I fix the index or somehow reindex the mailbox?
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
Joseph Tam wrote:
[dd]
>
> Whether you use any of them is up to you. As I said, direct mailbox
> file access will work. However, if you deal with Gb size mailboxes,
> re-indexing and general mail operations will be painful.
>
> > My mail goes through procmail, so Dovecot will not be the only one to
> > touch the mboxes anyway.
>
> Not a deterrent: replace all mailbox recipies with a pipe to dovecot's LDA
>
> :0 ... pattern ...
> | /path/to/dovecot-lda -d {user} -m {mailbox}
>
> ...
>
> # End of file: deliver to inbox
> :0 w
> | /path/to/dovecot-lda -d {user}
>
This will clutter up ~/.procmailrc, but may be I'll try someday.
Now I have encountered a different problem. My Android IMAP client
(the Gmail map) shows lots of mails in my Inbox. However, I already
deleted those mails locally from /var/mai/vas. They are being cached
somewhere.
Is it not dovecot caching them?
Nor can I find a way to forcefully rescan an IMAP folder in the Gmail app.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
Joseph Tam wrote:
[dd]
>
> Whether you use any of them is up to you. As I said, direct mailbox
> file access will work. However, if you deal with Gb size mailboxes,
> re-indexing and general mail operations will be painful.
>
> > My mail goes through procmail, so Dovecot will not be the only one to
> > touch the mboxes anyway.
>
> Not a deterrent: replace all mailbox recipies with a pipe to dovecot's LDA
>
> :0 ... pattern ...
> | /path/to/dovecot-lda -d {user} -m {mailbox}
>
> ...
>
> # End of file: deliver to inbox
> :0 w
> | /path/to/dovecot-lda -d {user}
>
This will clutter up ~/.procmailrc, but may be I'll try someday.
Now I have encountered a different problem. My Android IMAP client
(the Gmail map) shows lots of mails in my Inbox. However, I already
deleted those mails locally from /var/mai/vas. They are being cached
somewhere.
Is it not dovecot caching them?
Nor can I find a way to forcefully rescan an IMAP folder in the Gmail app.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
Larry Rosenman wrote: > > Larry Rosenman wrote: > > I have my dovecot running with TLS, so the passwords are NOT > transmitted in > > the clear (starttls). > > The problem is not transmitting them in the clear but keeping them in > the clear in a local file like muttrc, or entering the password each > time you launch mutt. > > > Hence my gpg text file trick. ( Do you enter your gpg password each time you launch mutt, or do you use some kind of gpg agent? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
On Thu, 27 Sep 2018, Victor Sudakov wrote:
As others have written, you may see performance degradation as Dovecot
will have to rebuild indices, but if you have small mailboxes, this won't
be too bad. The only reason I use direct file access rather than IMAP
is that I'm too lazy to work out a passwordless access method. If this
doesn't bother you or you can configure this (e.g. Kerberos, keyring,
etc.), IMAP access is preferable since you won't pull the indices out
from Dovecot's feet.
What passwordless access methods does Dovecot support? I'm more or
less experienced with GSSAPI but for the present I would not like to
set up a KDC somewhere on a box exposed to the Internet.
Yes, Kerberos is one way. Or some password broker as Larry has done
but the client has to be able to work with it. You can also pipe
to imap worker process via ssh/pubkey if your server will allow it.
Perhaps dovecot supports SSL client certificate authentication as well --
I haven't invested a lot of time looking into the various schemes.
Whether you use any of them is up to you. As I said, direct mailbox
file access will work. However, if you deal with Gb size mailboxes,
re-indexing and general mail operations will be painful.
My mail goes through procmail, so Dovecot will not be the only one to
touch the mboxes anyway.
Not a deterrent: replace all mailbox recipies with a pipe to dovecot's LDA
:0 ... pattern ...
| /path/to/dovecot-lda -d {user} -m {mailbox}
...
# End of file: deliver to inbox
:0 w
| /path/to/dovecot-lda -d {user}
Joseph Tam
Re: Local access to IMAP mailboxes
On 9/27/18, 10:57 AM, "Victor Sudakov" wrote: Larry Rosenman wrote: > I have my dovecot running with TLS, so the passwords are NOT transmitted in > the clear (starttls). The problem is not transmitting them in the clear but keeping them in the clear in a local file like muttrc, or entering the password each time you launch mutt. Hence my gpg text file trick. (
Re: Local access to IMAP mailboxes
Larry Rosenman wrote: > I have my dovecot running with TLS, so the passwords are NOT transmitted in > the clear (starttls). The problem is not transmitting them in the clear but keeping them in the clear in a local file like muttrc, or entering the password each time you launch mutt. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
I have my dovecot running with TLS, so the passwords are NOT transmitted in the clear (starttls). On Thu, Sep 27, 2018 at 10:42 AM Victor Sudakov wrote: > Joseph Tam wrote: > > > > > However, I often read and modify the mailboxes locally with mutt (e.g. > > > append and delete mails). > > > > > > Should I expect any problems wit Dovecot indexes etc? What if I even > > > do "rm ~/Mail/some/mailbox", will Dovecot be mad at me? > > > > I do it all the time. Works fine. > > Great, thank you! > > > > > As others have written, you may see performance degradation as Dovecot > > will have to rebuild indices, but if you have small mailboxes, this won't > > be too bad. The only reason I use direct file access rather than IMAP > > is that I'm too lazy to work out a passwordless access method. If this > > doesn't bother you or you can configure this (e.g. Kerberos, keyring, > > etc.), IMAP access is preferable since you won't pull the indices out > > from Dovecot's feet. > > What passwordless access methods does Dovecot support? I'm more or > less experienced with GSSAPI but for the present I would not like to > set up a KDC somewhere on a box exposed to the Internet. > > > > > You'll also get a lot of innocuous griping in the log files about > > UIDVALIDITY and mailbox corruption, but they can be safely ignored. > > My mail goes through procmail, so Dovecot will not be the only one to > touch the mboxes anyway. > > -- > Victor Sudakov, VAS4-RIPE, VAS47-RIPN > 2:5005/49@fidonet http://vas.tomsk.ru/ > -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
Re: Local access to IMAP mailboxes
Larry Rosenman wrote: > What I do is keep my passwords in a file encrypted by GPG and have mutt do: > source "gpg -q --textmode -d ~/.neomutt/passwords.gpg |" > > that way they're not readable except by use of a passworded GPG key. > Very clever, I must admit. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
Joseph Tam wrote: > > > However, I often read and modify the mailboxes locally with mutt (e.g. > > append and delete mails). > > > > Should I expect any problems wit Dovecot indexes etc? What if I even > > do "rm ~/Mail/some/mailbox", will Dovecot be mad at me? > > I do it all the time. Works fine. Great, thank you! > > As others have written, you may see performance degradation as Dovecot > will have to rebuild indices, but if you have small mailboxes, this won't > be too bad. The only reason I use direct file access rather than IMAP > is that I'm too lazy to work out a passwordless access method. If this > doesn't bother you or you can configure this (e.g. Kerberos, keyring, > etc.), IMAP access is preferable since you won't pull the indices out > from Dovecot's feet. What passwordless access methods does Dovecot support? I'm more or less experienced with GSSAPI but for the present I would not like to set up a KDC somewhere on a box exposed to the Internet. > > You'll also get a lot of innocuous griping in the log files about > UIDVALIDITY and mailbox corruption, but they can be safely ignored. My mail goes through procmail, so Dovecot will not be the only one to touch the mboxes anyway. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
What I do is keep my passwords in a file encrypted by GPG and have mutt do: source "gpg -q --textmode -d ~/.neomutt/passwords.gpg |" that way they're not readable except by use of a passworded GPG key. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: larry...@gmail.com US Mail: 5708 Sabbia Drive, Round Rock, TX 78665-2106 On 9/27/18, 9:50 AM, "dovecot on behalf of Victor Sudakov" wrote: Thomas Leuxner wrote: > > > > >> However, I often read and modify the mailboxes locally with Mutt (e.g. > > > >> append and delete mails). > > Why not use Mutt's IMAP capabilities and keep the indexes nice and clean? Maybe the only reason is that I don't want to bother with authentication and don't want to keep my password in muttrc. Anyway, my mail is sorted and delivered to ~/Mail/* by procmail, so the mboxes will be touched by a process outside of dovecot. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
Thomas Leuxner wrote: > > > > >> However, I often read and modify the mailboxes locally with Mutt (e.g. > > > >> append and delete mails). > > Why not use Mutt's IMAP capabilities and keep the indexes nice and clean? Maybe the only reason is that I don't want to bother with authentication and don't want to keep my password in muttrc. Anyway, my mail is sorted and delivered to ~/Mail/* by procmail, so the mboxes will be touched by a process outside of dovecot. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
On Wed, 26 Sep 2018, Victor Sudakov wrote: However, I often read and modify the mailboxes locally with mutt (e.g. append and delete mails). Should I expect any problems wit Dovecot indexes etc? What if I even do "rm ~/Mail/some/mailbox", will Dovecot be mad at me? I do it all the time. Works fine. As others have written, you may see performance degradation as Dovecot will have to rebuild indices, but if you have small mailboxes, this won't be too bad. The only reason I use direct file access rather than IMAP is that I'm too lazy to work out a passwordless access method. If this doesn't bother you or you can configure this (e.g. Kerberos, keyring, etc.), IMAP access is preferable since you won't pull the indices out from Dovecot's feet. You'll also get a lot of innocuous griping in the log files about UIDVALIDITY and mailbox corruption, but they can be safely ignored. Joseph Tam
Re: Local access to IMAP mailboxes
* Victor Sudakov 2018.09.26 12:17: > > >> However, I often read and modify the mailboxes locally with Mutt (e.g. > > >> append and delete mails). Why not use Mutt's IMAP capabilities and keep the indexes nice and clean? Regards Thomas signature.asc Description: PGP signature
Re: Local access to IMAP mailboxes
Sami Ketola wrote: > > On 26 Sep 2018, at 13.07, Aki Tuomi wrote: > > > >> > >> I have made mailboxes in ~/Mail available via IMAP (Dovecot 2.3.2.1), > >> that is: "mail_location = mbox:~/Mail:INBOX=/var/mail/%u" > >> > >> However, I often read and modify the mailboxes locally with Mutt (e.g. > >> append and delete mails). > >> > >> Should I expect any problems wit Dovecot indexes etc? What if I even > >> do "rm ~/Mail/some/mailbox", will Dovecot be mad at me? > >> > > > > Dovecot is tolerant to changes with mbox and maildir. It will reindex if > > it detects someone changing them outside. That is good news! Thank you Aki! > > That is true but it however has performance impacts if the caching > is impaired but still does works. Ah, is this perhaps the reason why my Android IMAP client sometimes takes so long to list a folder's contents? I must have touched the folder locally by Mutt and Dovecot is reindexing, is that right? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Re: Local access to IMAP mailboxes
> On 26 Sep 2018, at 13.07, Aki Tuomi wrote: > > > > On 26.09.2018 12:22, Victor Sudakov wrote: >> Dear Colleagues, >> >> I have made mailboxes in ~/Mail available via IMAP (Dovecot 2.3.2.1), >> that is: "mail_location = mbox:~/Mail:INBOX=/var/mail/%u" >> >> However, I often read and modify the mailboxes locally with mutt (e.g. >> append and delete mails). >> >> Should I expect any problems wit Dovecot indexes etc? What if I even >> do "rm ~/Mail/some/mailbox", will Dovecot be mad at me? >> > > Dovecot is tolerant to changes with mbox and maildir. It will reindex if > it detects someone changing them outside. That is true but it however has performance impacts if the caching is impaired but still does works. Sami
Re: Local access to IMAP mailboxes
On 26.09.2018 12:22, Victor Sudakov wrote: > Dear Colleagues, > > I have made mailboxes in ~/Mail available via IMAP (Dovecot 2.3.2.1), > that is: "mail_location = mbox:~/Mail:INBOX=/var/mail/%u" > > However, I often read and modify the mailboxes locally with mutt (e.g. > append and delete mails). > > Should I expect any problems wit Dovecot indexes etc? What if I even > do "rm ~/Mail/some/mailbox", will Dovecot be mad at me? > Dovecot is tolerant to changes with mbox and maildir. It will reindex if it detects someone changing them outside. Aki
