Re: Incremental purge?

[Jesus Cea]

On 24/8/20 7:12, Fabien KOCIK wrote:
> First, extract user list using doveadm users '*'.
> Then process this list to fit your needs (seven parts for a week and a
> part processed by week day for example).
> Finally, use -u option of doveadm purge into a loop.

This is a good idea for installations with tons of users. Good point.

I would be personally interested in incremental purge per user. My
personal IMAP account, for instance, has at this moment about 40GB of
mdbox files. I keep most of my email for decades, but sure I could free
some space doing a purge, but I am scared of trying with 200.000 mdbox
files without knowing how long it would take.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - https://www.jcea.es/_/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

Incremental purge?

[Jesus Cea]

I am using mdbox storage format.

As far as I know, “doveadm purge” does a complete purge. The time and disk 
activity would be proportional to the number mdbox files on the storage. The 
purge time could be huge.

I wonder if there is some way of doing a kind of incremental purge, 
stopping/restarting purging, etc. Purging by file range would be nice too.

Advice?

Thanks


Enviado desde mi iPhone

Thunderbird, server side filtering (sieve) and new email not in "inbox"

[Jesus Cea]

Migrating my extensive thunderbird filtering rules to sieve has the
unfortunate effect that thunderbird doesn't realize that there is new
email in other mailboxes beside "inbox".

According to ancient documentation,
,
I could configure Thunderbird to check each mailbox periodically, but I
have >500 mailboxes and I rather prefer the server to notify the client
when new email is avaliable somewhere.

Reading between lines in that document, looks like Thunderbird would
support STATUS updates while in "IDLE". Is that the case?.

Do you know how could I configure dovecot to provide that kind of
information to thunderbird?. Maybe some extension?

I am aware of "notify", but thunderbird is not "notify" compatible.

I can write an IMAP4 proxy to "inject" replies to thunderbird
side-fetched via "notify", but I need to know what thunderbird expect.
Do you have any idea?.

Thanks for your time and expertise.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

Re: Supporting RFC 5466 (IMAP4 Extension for Named Searches (Filters))

[Jesus Cea via dovecot]

On 26/9/16 17:51, Aki Tuomi wrote:
> 
>> On September 26, 2016 at 4:14 PM Tanstaafl  wrote:
>>
>>
>> On 9/19/2016 11:26 AM, Tanstaafl  wrote:
>>> On 10/1/2014 3:21 PM, Stephan Bosch  wrote:
>>>> On 10/1/2014 2:42 PM, Jesus Cea wrote:
>>>>> I wonder if Dovecot  supports RFC 5466 (IMAP4 Extension for Named
>>>>> Searches (Filters)) or if there is any plan about it.
>>>
>>>> I have a partial implementation in my patch queue. I haven't worked on
>>>> it for a few months now due to other projects that took precedence. It
>>>> still may take quite a while until I can continue that effort.
>>>
>>> I don't know if it is a good idea to resurrect such an old thread, but...
>>>
>>> Any chance there has been movement on this?
>>>
>>> There is a Thunderbird bug opened for supporting this:
>>>
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=439047
>>>
>>> and it would be much easier to try to push it forward if there was
>>> actually a server that supported it already.
>>
>> Timo? Aki? Anyone?
> 
> Hi! 
> 
> We need to discuss this internally
> 
> Aki
Very old thread.

Any news about this?

Thanks.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

Re: ESEARCH is announced but it doesn't work

[Jesus Cea via dovecot]

On 28/11/19 5:58, @lbutlr via dovecot wrote:
> The MULTISEARCH capability (RFC 6237) builds upon the ESEARCH capability to 
> make it possible to search in another mailbox than the currently selected 
> one. The command syntax you're using requires MULTISEARCH.

Ah, understood. Sorry for wasting your time.

Thanks.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

ESEARCH is announced but it doesn't work

[Jesus Cea via dovecot]

I am using Dovecot 2.3.4. I could upgrade if necessary.

I see this capabilities after login in:

"""
a OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES
WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY LITERAL+
NOTIFY SPECIAL-USE] Logged in
"""

ESEARCH (RFC 7377) seems to be available, but the command is not recognized:

"""
tag1 ESEARCH IN (mailboxes "folder1" subtree "folder2") unseen
tag1 BAD Error in IMAP command ESEARCH: Unknown command (0.001 + 0.000
secs).
"""

What am I doing wrong?

Thanks for your time and expertise.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

"search return (min)" is very inefficient

[Jesus Cea]

RFC 4731: " IMAP4 Extension to SEARCH Command for Controlling
  What Kind of Information Is Returned"

https://tools.ietf.org/html/rfc4731

I do this (telnet session):

"""
a select proveedores/TUHS
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
Flags permitted.
* 8330 EXISTS
* 0 RECENT
* OK [UNSEEN 6085] First unseen.
* OK [UIDVALIDITY 1395814416] UIDs valid
* OK [UIDNEXT 8358] Predicted next UID
* OK [HIGHESTMODSEQ 3149] Highest
a OK [READ-WRITE] Select completed (0.264 + 0.000 + 0.263 secs).
a search return (min) since 01-Feb-2018
* ESEARCH (TAG "a") MIN 8156
a OK Search completed (59.451 + 0.000 + 59.450 secs).
"""

Note that it needs 59 seconds to locate the first message matching the
"since" criteria. This is really slow and unusable in practice. Things
are worse with bigger mailboxes, of course.

Am I missing some performance trick?.

I am trying to locate the first message received after a particular
date. I don't use the "date" header because it can be falsified and
because I care about the receiving date and not the actual sending date.

I guess dovecot is scanning every message in the folder (mailbox).
Wouldn't those messages be stored in ascending "received" order?. Could
that be used for bisection search to locate the right message?. Could
that break if I copy/move/edit messages?. Is that the reason for such a
awful performance?.

Suggestions?.

Thanks for your time and knowledge!.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

what user to use for SIS deduplication?

[Jesus Cea]

Running Dovecot 2.2.25.

I have the following line in my root crontab:

"/usr/local/bin/doveadm sis deduplicate
/home/dovecot/buzones/attachments /home/dovecot/buzones/attachments/queue"

This line is execute each hour and the plan is to deduplicate
attachments stored in the SIS system.

This works well most of the time, but from time to time cron complains
with this error:

"""
Error: mkdir(/home/dovecot/buzones/attachments/bb/f8/hashes) failed: No
such file or directory
"""

In those situations I just create the directory by hand and set the
right permissions.

Interestingly, most of the time everything works fine, dovecot creates
the needed directories and everything is fine. But I get that error
every few days and I must create the directory by hand.

Am I running the deduplication process correctly?. It is launched as
"root" and I would expect it to "setgid()" to the right user/group
(dovemail). Should I run it in the dovemail cron?. I have a faint memory
of getting the very same error in that case, a missing hash directory,
but I am not sure I am recalling correctly.

Any advice?. Thanks.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

SIS deduplication fails to create intermediate directories and funny permissions

[Jesus Cea]

I have configured my system to use SIS, sharing the same same storage
for all users, and offline deduplication.

I have this in my "root" cron:

"""
53 1 * * *  /usr/local/bin/doveadm purge -A
16 * * * *  /usr/local/bin/doveadm sis deduplicate
/home/dovecot/buzones/attachments /home/dovecot/buzones/attachments/queue
"""

Yes, it is launched as "root" but I would expect doveadm drop privileges
as soon as possible.

This is working great, but from time to time I get this error message:

"""
Your "cron" job on babylon5
/usr/local/bin/doveadm sis deduplicate /home/dovecot/buzones/attachments
/home/dovecot/buzones/attachments/queue

produced the following output:

Error: mkdir(/home/dovecot/buzones/attachments/dc/62/hashes) failed: No
such file or directory
"""

Looks like doveadm is trying to create the "hashes" directory but the
parent "62" doesn't exits yet. If I create that directory by hand and
set permissions correctly, everything (seems to) work.

Talking about this, I see something strange wit the permissions.

All files and directories in SIS are created with "dovemail" user and
group EXCEPT the "hashes" directory inside the newly created directory.

That is:

When I get the error message, I have this:

"""
drwx--   7 dovemail dovemail   7 Apr  1 13:28 .
drwx-- 237 dovemail dovemail 237 Mar 24 20:09 ..
drwx--   3 dovemail dovemail   3 Feb 29 02:30 1d
drwx--   3 dovemail dovemail   3 Mar  9 01:53 7f
drwx--   3 dovemail dovemail   3 Feb 29 02:30 88
drwx--   3 dovemail dovemail   3 Feb 29 02:30 8d
"""

I create the missing directory by hand and change permissions. Now I have:

"""
drwx--   7 dovemail dovemail   7 Apr  1 13:28 .
drwx-- 237 dovemail dovemail 237 Mar 24 20:09 ..
drwx--   3 dovemail dovemail   3 Feb 29 02:30 1d
drwx--   3 dovemail dovemail   3 Apr  1 13:30 62
drwx--   3 dovemail dovemail   3 Mar  9 01:53 7f
drwx--   3 dovemail dovemail   3 Feb 29 02:30 88
drwx--   3 dovemail dovemail   3 Feb 29 02:30 8d
"""

I run the deduplicator again (as root) and it creates a "hashes"
directory inside the "62" directory... with user/group root/root.

"""
drwx-- 2 root root 2 Apr  1 13:30 hashes
"""

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

How to delete Dovecot users when using Single Instance Storage

[Jesus Cea]

In case somebody find this useful:

https://blog.jcea.es/posts/20160211-delete_users_dovecot.html

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

Re: "doveadm backup" doesn't work anymore after upgrading to 2.2.18

[Jesus Cea]

On 20/05/15 21:53, Robert L Mathews wrote:
> On 5/19/15 6:19 AM, Jesus Cea wrote:
> 
>> dsync-local(jcea): Error: Remote command returned error 65: ssh csi
>> doveadm dsync-server dsync-server
>> """
>>
>> Now in 2.2.18 I get the second error and the backup is aborted. It
>> doesn't work anymore.
>>
>> I don't find any reference to error 65 anywhere.
> 
> Error 65 is likely the SSH process exit code. Pages like this suggest
> it's "Host Not Allowed To Connect":

It is not a SSH error because to get the first error (the sieve related
one) "doveadm" already connected to the remote backup server via SSH.

I have installed & configured sieve in the backup server, with an empty
filter set. I can now do the backup correctly.

Interestingly if I disable the sieve plugin in the backup server, the
backup keeps working. WTF!

My guess is that sieve has set the
"vendor/vendor.dovecot/pvt/server/sieve/default" attribute in the
mailbox and the backup process is happy now, even if I disable sieve
after the fact.

I consider this issue closed but it would be nice to be documented
somewhere. Thanks.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

"doveadm backup" doesn't work anymore after upgrading to 2.2.18

[Jesus Cea]

Until today I could do this to backup my primary IMAP4 server:

"""
doveadm backup ssh csi doveadm dsync-server
"""

It doesn't work anymore after upgrading to Dovecot 2.2.18:

"""
jcea@ubuntu:~$ doveadm backup ssh csi doveadm dsync-server
Enter passphrase for key '/home/jcea/.ssh/id_rsa':
dsync-remote(root): Error: Mailbox INBOX: Failed to get attribute
vendor/vendor.dovecot/pvt/server/sieve/default: Mailbox attributes not
enabled
dsync-local(jcea): Error: Remote command returned error 65: ssh csi
doveadm dsync-server dsync-server
"""

The first error is regular business: I don't have SIEVE in the backup,
since it is a backup. I have seen this error forever, no consequences.

Now in 2.2.18 I get the second error and the backup is aborted. It
doesn't work anymore.

I don't find any reference to error 65 anywhere.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

Re: [Dovecot] mdbox-files over 2 MB

[Jesus Cea]

I keep the quoted message complete because the original thread was
posted six months ago. I am top-posting, no comments inside the quoted text.

I plan to write a blog about mdbox oddities but in the meantime I can
tell you that if you store attachments "detached" in your mdbox storage
the issue is this:

1. You have a 200Kbytes MB m.* file you are currently targeting.

2. You have a new email to store. Lets say it has a 3MB attachment.

3. You have a 2MB mdbox max file size .

4. Dovecot notices that 0.2+3 > 2, so it will create a new m.* file.

5. When storing the message, it will store the attachment outside of the
mbox, so the new m.* file will be, lets say, 5Kbytes only.

6. Now you have 2 m.* files: one of 200 Kbytes and other of 5Kbytes.

7. Dovecot tries to reuse undersized m.* files, but it is quite
forgiving. I have some private patches to increase its agressiveness.

If you don't detach attachment, you probably have the same issue: when a
new message makes a m.* file to oversize, dovecot creates a new m.* file
and leave the old one alone, so it will be undersized.



On 06/05/14 03:05, Stan Hoeppner wrote:
> On 5/5/2014 2:23 PM, Patrick Domack wrote:
>> Quoting Reindl Harald :
>>
>>> Am 05.05.2014 16:10, schrieb Hardy Flor:
 Is there really no one with this problem?
>>>
>>> next time quote the problem instead demand
>>> others to seek for you in the archives
>>
>> Even searching for him in the archives, I have no idea what he thinks is
>> a problem.
>>
>> I know I have no issues using mdbox, with 2mb or 50mb rotate sizes.
> 
> 
> He's wondering why many of his mdbox files are much smaller than 2 MB.
> His original post:
> 
> 
> On 3/17/2014 1:29 AM, Hardy Flor wrote:
>> Hello,
>>
>> there are copies with different size in 3 mailboxes of the user
>> sequentially about 3800 emails.
>> why not something 2MB files?
>>
>> After the big file "m.0034" with 14MB follow very many small ...
>>
>> - doveconf:
>>
>> # 2.2.12: /etc/dovecot/dovecot.conf
>> # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4 ext4
>> mail_attachment_dir = /var/mail/attachments
>> mail_attachment_hash = %{md5}
>> mail_attachment_min_size = 16 k
>> mail_location = mdbox:/var/mail/user/%n
>> mdbox_rotate_interval = 1 weeks
>> mdbox_rotate_size = 2 M
>>
>> - directory:
>>
>> root@xxx:/var/mail/user/xxx/storage# ls -al
>> insgesamt 76532
>> drwx--S--- 2 vmail mail 4096 Mär 16 13:52 .
>> drwx--S--- 6 vmail mail 4096 Mär 16 12:10 ..
>> -rw--- 1 vmail mail74360 Mär 16 12:39 dovecot.map.index
>> -rw--- 1 vmail mail25204 Mär 16 13:52 dovecot.map.index.log
>> -rw--- 1 vmail mail63544 Mär 16 12:39 dovecot.map.index.log.2
>> -rw--- 1 vmail mail  2092921 Mär 16 12:15 m.0001
>> -rw--- 1 vmail mail  2086746 Mär 16 12:15 m.0002
>> -rw--- 1 vmail mail  2084182 Mär 16 12:15 m.0003
>> -rw--- 1 vmail mail  2096826 Mär 16 12:15 m.0004
>> -rw--- 1 vmail mail  1882666 Mär 16 12:15 m.0005
>> -rw--- 1 vmail mail  1903965 Mär 16 12:15 m.0006
>> -rw--- 1 vmail mail  2091169 Mär 16 12:15 m.0007
>> -rw--- 1 vmail mail  2086396 Mär 16 12:15 m.0008
>> -rw--- 1 vmail mail   507205 Mär 16 12:16 m.0009
>> -rw--- 1 vmail mail  2031456 Mär 16 12:17 m.0010
>> -rw--- 1 vmail mail  2095697 Mär 16 12:17 m.0011
>> -rw--- 1 vmail mail  1689071 Mär 16 12:17 m.0012
>> -rw--- 1 vmail mail  2092124 Mär 16 12:17 m.0013
>> -rw--- 1 vmail mail  1950602 Mär 16 12:17 m.0014
>> -rw--- 1 vmail mail  2092215 Mär 16 12:17 m.0015
>> -rw--- 1 vmail mail  2087463 Mär 16 12:17 m.0016
>> -rw--- 1 vmail mail  2079795 Mär 16 12:17 m.0017
>> -rw--- 1 vmail mail  2014121 Mär 16 12:17 m.0018
>> -rw--- 1 vmail mail  2081893 Mär 16 12:17 m.0019
>> -rw--- 1 vmail mail  2092088 Mär 16 12:17 m.0020
>> -rw--- 1 vmail mail  2090508 Mär 16 12:17 m.0021
>> -rw--- 1 vmail mail  1929296 Mär 16 12:17 m.0022
>> -rw--- 1 vmail mail  2067685 Mär 16 12:17 m.0023
>> -rw--- 1 vmail mail  1745743 Mär 16 12:20 m.0024
>> -rw--- 1 vmail mail   866452 Mär 16 12:20 m.0025
>> -rw--- 1 vmail mail   296379 Mär 16 12:20 m.0026
>> -rw--- 1 vmail mail   433541 Mär 16 12:39 m.0027
>> -rw--- 1 vmail mail92526 Mär 16 12:39 m.0028
>> -rw--- 1 vmail mail19094 Mär 16 12:39 m.0029
>> -rw--- 1 vmail mail   106837 Mär 16 12:39 m.0030
>> -rw--- 1 vmail mail   162012 Mär 16 12:39 m.0031
>> -rw--- 1 vmail mail   412080 Mär 16 13:51 m.0032
>> -rw--- 1 vmail mail  1715868 Mär 16 13:51 m.0033
>> -rw--- 1 vmail mail 14433784 Mär 16 13:51 m.0034
>> -rw--- 1 vmail mail 3280 Mär 16 13:51 m.0035
>> -rw--- 1 vmail mail 6308 Mär 16 13:51 m.0036
>> -rw--- 1 vmail mail   534274 Mär 16 13:51 m.0037
>> -rw--- 1 vmail mail   378719 Mär 

Supporting RFC 5466 (IMAP4 Extension for Named Searches (Filters))

[Jesus Cea]

I wonder if Dovecot  supports RFC 5466 (IMAP4 Extension for Named
Searches (Filters)) or if there is any plan about it.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

virtualfolders under mdbox storage backend

[Jesus Cea]

Reading docs in http://wiki2.dovecot.org/Plugins/Virtual I wonder how is
that used if I am using MDBOX as my storage backend. The only references
there are about Maildir.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

"doveadm sync/backup" should show some kind of progress meter

[Jesus Cea]

Using "-v" with those commands doesn't nothing, aparently.

Using "-D" shows complex debug info only useful for debugging.

But this comands take a lot of time and should show some kind of
progress meter/feedback like network speed, percent, ETA, etc.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

"doveadm sync/backup" and SSH should use "ssh-agent" if available

[Jesus Cea]

I use SSH-AGENT in my system, to cache SSH credentials and avoid typing
my key every time I connect to a remoto host.

But "doveadm sync/backup", when one of the ends is SSH accessed, doesn't
use the available SSH-AGENT and forces me to type my password.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

"doveadm backup/sync" don't follow message moving even with MDBOX

[Jesus Cea]

Using mdbox as the storage backend, moving a message from a mailbox to
another just update a reference. The message is not fisically moved on
the disk.

I would expect "dsync" to use this fact when source and destination are
mdbox. But it doesn't.

Anybody else can confirm?.

Any plan to implement it?.

Thanks!.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

Re: "doveadm backup/sync" are badly documented (SOLVED AND BUG REPORT!)

[Jesus Cea]

On 28/09/14 23:52, Jesus Cea wrote:
[..]
> This will "backup" this mailing list mailbox to "/tmp/aa". Good. Lets
> try thru SSH:
> 
> $ doveadm backup -u jcea  -m proveedores/dovecot ssh 127.0.0.1 dsync -u
> jcea mdbox:/tmp/aa/
> jcea@127.0.0.1's password:
> Fatal: Invalid parameter: mdbox:/tmp/aa/
> dsync-local(jcea): Error: read(remote) failed: EOF (version not received)
> 
> How shoudl I do it?.

Trying a few things I got here, that works:

$ doveadm -o mail_fsync=never -o mdbox_rotate_interval=99d -o
mdbox_rotate_size=10M -o mail_attachment_dir=/tmp/aa/attachments -o
mail_attachment_min_size=64k -o mail_attachment_hash=%{sha256} -o
mail_location=mdbox:/tmp/aa/:DIRNAME=XxX backup  -m 'varios/nosotros' -R
 ssh 127.0.0.1 doveadm dsync-server

Two things:

1. I am connecting to myself thru SSH. I want to do a copy of my mdbox
with a different configuration. I was not able to transfer that
configuration change to thru SSH, so I just reverse the data flow
direction (The "-R"): just connect to the regular mailbox_location and
bring data here to be stored with a different configuration. This was a
kind of enlightment.

2. You must do "ssh 127.0.0.1 doveadm dsync-server" instead of simply
"ssh 127.0.0.1" because you would get this error:

bash: dsync-server: command not found
dsync-local(jcea): Error: read(remote) failed: EOF (version not received)
dsync-local(jcea): Error: Remote command returned error 127: ssh
127.0.0.1 dsync-server

I guess dovecot is sending the legacy "dsync-server" instead of the
current "doveadm dsync-server".

LOOKS LIKE THAT IS A BUG!!.

> $ doveadm backup -h
> backup: invalid option -- 'h'
> doveadm backup [-u |-A] [-S ] [-dfR] [-l ] [-r
> ] [-m ] [-n  | -N] [-x ] [-s
> ] 
> 
> I can't find any explanation at all about the different parameters of
> "doveadm backup".

Please, document.

> Same thing with "doveadm sync".

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

Re: dsync full sync

[Jesus Cea]

On 01/09/14 14:41, Atıf CEYLAN wrote:
> Hi all,
> I have 2 question.
> 
> First:
> I use dovecot (version 2.2.9) with mdbox mail format. When I run dsync
> tool with "mirror" or "backup" parameters my source and destination
> directory synchronize correctly but if I delete some messages in user
> mailbox, deleted messages does not synced to destination.

Had you tried running "doveadm purge" in the destination?.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

"doveadm backup/sync" are badly documented

[Jesus Cea]

Most documents around there talk abour "dsync", but the modern way is
"doveadm backup". This command is not documented in the wiki and there
are a few details missing, like how to use it thru SSH.

I am currently doing some tests about how to backup my mdbox. I can do
tests in local using:

$ doveadm backup -u jcea -m proveedores/dovecot mdbox:/tmp/aa/

This will "backup" this mailing list mailbox to "/tmp/aa". Good. Lets
try thru SSH:

$ doveadm backup -u jcea  -m proveedores/dovecot ssh 127.0.0.1 dsync -u
jcea mdbox:/tmp/aa/
jcea@127.0.0.1's password:
Fatal: Invalid parameter: mdbox:/tmp/aa/
dsync-local(jcea): Error: read(remote) failed: EOF (version not received)

How shoudl I do it?.

$ doveadm backup -h
backup: invalid option -- 'h'
doveadm backup [-u |-A] [-S ] [-dfR] [-l ] [-r
] [-m ] [-n  | -N] [-x ] [-s
] 

I can't find any explanation at all about the different parameters of
"doveadm backup".

Same thing with "doveadm sync".

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

Re: "Corrupted dbox file [...] purging found mismatched offsets"

[Jesus Cea]

On 29/07/14 01:21, Daniel Parthey wrote:
> Are you using dovecot director and a shared filesystem like NFS?
> 
> You not only need to run imap and pop3 service via director,
> but also the doveadm service, otherwise different hosts might
> access the mdbox simultaneously, which will corrupt your index
> files and you will lose flags.

No, this is a dovecot server running in my own laptop. I guess some
computer crash left the mail in an inconsistent state, and I need to
assert the extend of the problem. A well designed system should not lose
email, even when crashes are involved.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz




signature.asc
Description: PGP signature


signature.asc
Description: OpenPGP digital signature

Re: "Corrupted dbox file [...] purging found mismatched offsets"

[Jesus Cea]

On 28/07/14 01:30, Jesus Cea wrote:
> Doing a "doveadm purge" today I got this:
> 
> """
> doveadm(jcea): Error: Corrupted dbox file

I am using Dovecot 2.2.13.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

"Corrupted dbox file [...] purging found mismatched offsets"

[Jesus Cea]

Doing a "doveadm purge" today I got this:

"""
doveadm(jcea): Error: Corrupted dbox file
/home/jcea/.thunderbird/dovecot/storage/m.686 (around offset=1385772):
purging found mismatched offsets (1385742 vs 1380664, 185/275)
doveadm(jcea): Warning: fscking index file
/home/jcea/.thunderbird/dovecot/storage/dovecot.map.index
doveadm(jcea): Warning: mdbox /home/jcea/.thunderbird/dovecot/storage:
rebuilding indexes
doveadm(jcea): Error: Purging namespace '' failed: Internal error
occurred. Refer to server log for more information. [2014-07-28 01:17:10]
"""

Checking the log I get this:

"""
Jul 28 01:17:18 ubuntu dovecot: imap(jcea): Error:
/home/jcea//.thunderbird/dovecot/mailboxes/proveedores/dovecot/DbOx-mAiLs/dovecot.index
reset, view is now inconsistent
Jul 28 01:17:19 ubuntu dovecot: imap(jcea): Error:
/home/jcea//.thunderbird/dovecot/mailboxes/varios/OpenStreetMap/osm-newbies/DbOx-mAiLs/dovecot.index
reset, view is now inconsistent
Jul 28 01:17:20 ubuntu dovecot: imap(jcea): Error:
/home/jcea//.thunderbird/dovecot/mailboxes/varios/esc&AOk-pticos/DbOx-mAiLs/dovecot.index
reset, view is now inconsistent
"""

Subsequent "doveadm purge" don't show anything.

What happened and why?.

Consequences?. Did I lose email?

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

mdbox "doveadm purge" strategy

[Jesus Cea]

Hi there.

I am using mdbox storage with great success but I am seeing something I
find strange and I would like to compare notes with you.

When doing "doveadm purge", "m.*" files with non deleted messages are
left alone, but "m.*" files with deleted messages copy the "non deleted"
messages to a newly created "m.*" file and delete the original.

Good so far.

What surprises me is that the newly created "m.*" files are not
fullsize. they can be "small" files, not using the 2MB max size I have
configured for mdbox storage. Moreover, those files stay in that size
forever, no new messages are appended ever.

I guess each old "m.*" file with deleted messages generates a new
independent "m.*" containing only the non deleted messages. I wonder if
that is the case and if that is a fundamental issue that can't be
changed. I would like my "m.*" files to use the full configured mdbox
size, instead of having quite a few of new SMALL "m.*" files.

Thanks for your time and attention!.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

Re: [Dovecot] Not backing up cache files

[Jesus Cea]

On 26/03/14 09:05, Jesus Cea wrote:
> I am using mdbox for dovecot storage.
> 
> I wonder if I could skip "dovecot.index.cache" files and, in general,
> "*.cache" files when doing mail backup. Those files are big and change
> frequently. What I feel from reading documentation [1][2] is that they
> could be dropped in the backups, to be recreated on the fly if needed.
> 
> But I would like expert confirmation.
> 
> [1] http://wiki2.dovecot.org/IndexFiles
> [2] http://wiki2.dovecot.org/Design/Indexes/Cache

1. I stopped "dovecot".

2. I did a ZFS snapshot (play safe!), and disabled incoming messages.

3. I deleted ALL "dovecot.index.cache" (one per mailbox, I have about
500 mailboxes) without any apparent effect.

4. I started dovecot.

5. I moved around with apparently no side-effects. Cache are recreated,
small size, growing as you download messages.

I still would love an authoritative answer.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

[Dovecot] Not backing up cache files

[Jesus Cea]

I am using mdbox for dovecot storage.

I wonder if I could skip "dovecot.index.cache" files and, in general,
"*.cache" files when doing mail backup. Those files are big and change
frequently. What I feel from reading documentation [1][2] is that they
could be dropped in the backups, to be recreated on the fly if needed.

But I would like expert confirmation.

[1] http://wiki2.dovecot.org/IndexFiles
[2] http://wiki2.dovecot.org/Design/Indexes/Cache

Thanks!.

-- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz



signature.asc
Description: OpenPGP digital signature

[Dovecot] Pluggable mailstore

[Jesus Cea]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Reading the archives I see messages from a long (years) time ago
discussing pluggable mailstores. No recent news, though.

Any progress?. Current status?

Regards

- -- 
Jesús Cea Avión _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
Twitter: @jcea_/_/_/_/  _/_/_/_/_/
jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJTEruXAAoJEGjgN61Khv5D0hMIAK8fH6Qu+fvliQXjpTFutJ9r
nw86mVM5OjPtNKw3O4/0/SfiM+/36rrT53rThbb/guDAIkNwt05NgfIbjdd74jRw
8U+CzoG/fVDe9jV2tzy5UThogVALKSxj1+EQrTk3PwepF68NZYb+s1ZDrfga7LKG
F0+wt+XGTga4hIIkPGHoEDCQHX5Z2VpZwItsPXIta1Dv+ObeJtVcv11xL6raf3R2
0FFF1fj0OcKuGFBNH9cKiFTq5jvHb3QzK2205VDpsv5rnNnYZT9pbFHrtZ3Qk6O9
nxKSQVjRjf2pGMo9mjD/qxq/LUQFUVkJFdeueN/Z9KWeqm6MC/n16GCYBYgWpXU=
=LrP4
-END PGP SIGNATURE-