Re: "client_id" in lua script

[Michael Slusarz via dovecot]

> On 10/25/2024 9:47 AM MDT ecdlguy--- via dovecot  wrote:
> 
> Hi,
> 
> I'm trying to block specific MUAs, like e.g. Outlook-App for iOS and Android 
> or Xiaomi Mail / MiMail using a lua script.
> While implementing this, I noticed that "client_id" is always nil in the lua 
> script or I am doing something wrong.
> This is part of the postfix config:

A note that this is explicitly not a supported IMAP ID use-case.

Per https://www.rfc-editor.org/rfc/rfc2971.html#section-3:

  Servers MUST NOT deny access to or refuse service for a client
  based on information from the ID command.  Clients MUST NOT refuse
  to operate or limit their operation with a server based on the ID
  response.

Client blocking is much better handled via other mechanisms, that may 
incorporate ID information as a part of the decision making process but does 
not make a determinative decision based on that string.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot replication

[Michael Slusarz via dovecot]

> On 07/12/2024 1:14 PM MDT James Cook via dovecot  wrote:
>
> On Fri, Jul 12, 2024 at 06:28:13PM GMT, John Fawcett via dovecot wrote:
> >Hi James
> >
> >I want to avoid the -1 parameter because it doesn't do deletes in the 
> >target.
> 
> -l, not -1.

No, it's -1 - as in one(1)-way sync.

-l (lowercase L) is for locking.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot imap_zlib

[Michael Slusarz via dovecot]

> On 07/09/2024 6:42 AM MDT Bjoern Franke via dovecot  
> wrote:
> 
> >> I tested teh git version of dovecot. It seems the IMAP Compress plugin
> >> (imap_zlib) has disappeared.
> > 
> > https://github.com/dovecot/core/commit/5f27e25c64555dcaae6cb00c479cd05bc2638081
> 
> so the zlib plugin is also deprecated and clients should run compress 
> themselves?

These are two distinct plugins.

imap-zlib = compression of the IMAP protocol stream
zlib = mailbox storage compression on the Dovecot server

For 2.4, "zlib" plugin has been renamed to "mail-compress".  
https://doc.dovecot.org/2.4/core/plugins/mail_compress.html

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot imap_zlib

[Michael Slusarz via dovecot]

> On 07/07/2024 8:09 AM MDT Joan Moreau via dovecot  wrote:
> 
> I tested teh git version of dovecot. It seems the IMAP Compress plugin 
> (imap_zlib) has disappeared.

https://github.com/dovecot/core/commit/5f27e25c64555dcaae6cb00c479cd05bc2638081

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: [EXT] Re: Dovecot community repositories

[Michael Slusarz via dovecot]

> On 06/13/2024 2:33 AM MDT MK via dovecot  wrote:
>  
> What is the reason that Debian 12/Ubuntu 22.04/RHEL 9 are not supported by CE 
> 2.3? 

OS-provided dependencies that won't work with 2.3 code (e.g., OpenSSL).

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Dovecot community repositories

[Michael Slusarz via dovecot]

> On 06/12/2024 5:37 AM MDT MK via dovecot  wrote:
> 
> just a short question to the dovecot people, maybe Aki or someone else can 
> answer this.
> Will there be an update to the Dovecot community repositories in the near 
> future? 
> The repositories are lagging behind the current distributions. Just as an 
> example: Debian 12 has been released in 06/2023, this is one year ago and 
> there are still no packages for it.
> Same for Ubutun 22.04, RHEL 9...  Is there still any interest from dovecot 
> side to continue to maintain the community repostitorys?

The community repositories continue to be maintained.

Debian 12/Ubuntu 22.04/RHEL 9 are not supported by CE 2.3 so we don't build 
packages for them.  They will be supported in CE 2.4.

Distros may have done their own work to modify Dovecot source to get 2.3 to 
build/package on these systems.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Log detective help

[Michael Slusarz via dovecot]

> On 06/05/2024 1:22 PM MDT GDS via dovecot  wrote:
>  
> Hello all, I am seeing hundreds of lines like the one below in my mail.log 
> from this specific IP address, which belongs to Google. Is there a way to 
> determine why this "deferred (delivery temporarily suspended)" is happening?
> 
> Jun  5 19:09:32 arthemis postfix/error[86771]: 5D9D148296D: 
> to=, orig_to=, relay=none, delay=4099, 
> delays=4099/0.02/0/0, dsn=4.4.1, status=deferred (delivery temporarily 
> suspended: connect to localhost.com[74.125.224.72]:25: Connection timed out)

"localhost.com" - you almost certainly are intending to connect to localhost 
(i.e. the local loopback address, 127.0.0.1) rather than the remote domain 
localhost.com.  So it looks like a configuration error.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: After user disconnect run the custom script

[Michael Slusarz via dovecot]

> On 05/20/2024 9:43 AM MDT Alexey Krylov via dovecot  
> wrote:
> 
> Please, send me the link, where I can find the info about configuring
> firing script after dovecot client is disconnected.
> 
> I found post-login scripting. Than's cool, but... I need to fire script
> a little bit later.

See 
https://doc.dovecot.org/admin_manual/list_of_events/#mail-user-session-finished

You will need to build a event listener for this event, and then do your 
scripting in there.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

RE: temporary auth errors

[Michael Slusarz via dovecot]

> On 05/02/2024 7:48 AM MDT Marc via dovecot  wrote:
> 
> > auth_failure_delay = 2 secs ?
> > 
> > That will still simply wait before *rejecting* the login, compared to
> > *dropping the connection*.
> > 
> > We are thus looking for three different behaviours:
> > 
> > 1.  If backend confrims auth, ACK auth + proceed (grant access) to email.
> > 
> > 2.  If backend confirm "no such user" or "invalid creds", wait for
> > auth_failure_delay and then *reject* the login.
> > 
> > 3.  If the backend fails (ie, can neither confirm nor deny), simply drop
> > the connection.
> > 
> > I hope this is more clear.
> > 
> 
> Yes that is more clear, but no idea (seems a little out of scope to support 
> by design)

In complicated, localized authentication scenarios, Lua auth is likely the best 
answer.  
https://doc.dovecot.org/configuration_manual/authentication/lua_based_authentication/

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Auth USER lookup failed

[Michael Slusarz via dovecot]

> On 02/05/2024 10:17 PM MST thecou...@gmail.com wrote:
> 
> I have dovecot working with PAM and Samba DC authentication for IMAP. However 
> when I'm attempting to pass emails from postfix via LMTP
> I don't actually need to authenticate LMTP I'm happy to check for valid users 
> upstream.
> 
> I'm getting the error:
> Feb 06 15:11:39 Debian-server postfix/local[178200]: ADF0E78713C: passing 
>  to transport=lmtp
> Feb 06 15:11:39 Debian-server dovecot[178075]: auth: Error: 
> static(dom.username): passdb doesn't support lookups, can't verify user's 
> existence
> Feb 06 15:11:39 Debian-server dovecot[178075]: 
> lmtp(dom.usern...@debian-server.sr.local)<178233>: 
> Error: auth-master: userdb lookup(dom.usern...@debian-server.sr.local): Auth 
> USER lookup failed

You need to define a userdb to return user information.  Since LMTP doesn't 
require auth, it can't use the passdb so that's what the error message is 
saying.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: The future of SIS

[Michael Slusarz via dovecot]

Aki is correct and is consistent with what I said in the video, although I 
could have phrased my explanation better.
 
"dsync" refers to the tool/utility (part of doveadm) that does mail 
synchronization between a source account to a destination account.  As Aki 
said, this is not going anywhere.  This is a necessary tool for any kind of 
migrations, for example.  dsync is under active maintenance, as we heavily use 
this tool internally.
 
What is being removed is the replicator plugin (that used dsync).  That's what 
is being referred to in the video.  Replicator hasn't been actively maintained 
for years now so this was dead code anyway.
 
To answer the OP: sis is also being removed and should not be used by any new 
installation. Code remains to read data written by the old plug-in so that 
these installations don't require a migration between 2.3 and 2.4.  This is 
another plugin that hasn't be actively maintained in years, and has all kinds 
of limitations that prevent it from running at scale.
 
Neither replicator nor sis is code that is moving from open to closed source. 
These plugins aren't used in Pro.  They are unmaintained so they are being 
removed, as happens with any kind of old code.
 
michael 


> On 10/13/2023 1:26 PM MDT Laura Smith via dovecot  wrote:
> 
>  
> FUD ? 
> 
> I knew someone would accuse me of that which is why I linked to the video 
> from the horse's mouth, I transcribe what the speaker said:
> 
> "there will be an open source version, but that open source version will be 
> maintained for single server use only. we are actually taking out anything 
> any actually kinda' involves multiple servers, dsync replication and err some 
> other stuff. so dovecot will be a fully-featured single node server"
> 
> 
> 
> 
> --- Original Message ---
> On Friday, October 13th, 2023 at 19:37, Aki Tuomi 
>  wrote:
> 
> 
> > Dear Laura, please don't spread FUD that you made up.
> > 
> > Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. 
> > There is not a trove of code going into Dovecot 3.0 that "never sees the 
> > daylight".
> > 
> > Thank you,
> > Aki
> > 
> > > On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org 
> > > wrote:
> > > 
> > > TL;DR If you are a Dovecot Community user, don't waste your time reading 
> > > the Dovecot Pro release notes.
> > > 
> > > To expand:
> > > 
> > > I think you have to understand that lots of things that are going into 
> > > Dovecot 3 (Pro) will never see the light of day in the community edition.
> > > 
> > > In addition, Dovecot have publicly quite plainly announced in public that 
> > > they are actively removing all multi-server related functionality from 
> > > Dovecot Community.
> > > 
> > > I don't think the community has quite yet grasped it. Things like dsync 
> > > will be GONE in the community version.
> > > 
> > > If you don't believe me, look at this video, about 15 minutes in:
> > > https://youtu.be/s-JYrjCKshA?feature=shared&t=912
> > > 
> > > --- Original Message ---
> > > On Friday, October 13th, 2023 at 17:15, Sebastian Marsching 
> > > sebast...@marsching.com wrote:
> > > 
> > > > Hi,
> > > > 
> > > > I am currently in the process of planning a new deployment of Dovecot. 
> > > > I was planning to use mdbox or sdbox with “mail_attachment_fs = sis 
> > > > posix”, but I stumbled across the following notice in the documentation 
> > > > for Dovecot 3.0
> > > > ___
> > > > dovecot mailing list -- dovecot@dovecot.org
> > > > To unsubscribe send an email to dovecot-le...@dovecot.org
> ___
> dovecot mailing list -- dovecot@dovecot.org
> To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Slusarz via dovecot]

> On 07/19/2023 2:54 PM MDT Michael Grimm via dovecot  
> wrote:
> 
> Michael Slusarz via dovecot  wrote:
> >> On 07/18/2023 9:00 AM MDT Gerald Galster  wrote:
> 
> >> While I understand it takes effort to maintain the replication plugin, 
> >> this is especially problematic for small active/active high-availability 
> >> deployments.
> > 
> > To clarify: replication absolutely does not provide "active/active".  
> > Replication was meant to copy data to a standby server, but you can't have 
> > concurrent mailbox access.  This is why directors existed.
> 
> That simply isn't true, and I am baffled that you don't know that replication 
> works with a two server active/active setup for years now! Two separate 
> instances (active/active) on two different continents are a completely 
> reliable failover scenario for years now.
> 
> Very irritating to read such a statement.

You may be irritated, but my statement is accurate.

Eventually consistent replication is *NOT* active/active.  active/active has a 
very specific meaning (and is not the same as master/master).

Quotas and shared mailboxes are two troublesome concepts with replicator.  
Inconsistent mailbox views are a call center driver.  Neither of these would be 
an issue in a true active/active setup.  Forcing a user to a single node at any 
given time will prevent some (but not all) issues.

Replicator's scaling issue can't really be worked around, and was a main driver 
why Dovecot Pro was developed (example: one Pro customer migrating from 
CE/replicators saw a 90% decrease in server count).

Your positive individual experience does not change the inherent 
characteristics, and limitations, of the design.  If your setup works for you, 
in your particular circumstances, great!  But it doesn't work for everyone.  
There is a reason Dovecot development moved on from replicator based 
architecture 10+ years ago.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [EXT] RE: Replication going away?

[Michael Slusarz via dovecot]

> On 07/19/2023 12:51 PM MDT Marc  wrote:
>
> > A 50-100 mailbox user server will run Dovecot CE just fine.  Pro would
> > be overkill.
> 
> What is overkill? I always thought it had a bit more features and support.

For Pro 2.3, you need (at minimum) 7 Dovecot nodes + HA authentication + HA 
storage + (minimum) 3 Cassandra nodes if using object storage.  This is per 
site; most of our customers require data center redundancy as well, so multiply 
as needed.  And this is only email retrieval; this doesn't even begin to touch 
upon email transfer.

Email high availability isn't cheap.  (I would argue that if you truly need 
this sort of carrier-grade HA for 50 users, it makes much more sense to use 
email as-a-service than trying to do it yourself these days.  Unless you have 
very specific reasons and a ton of cash.)

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Slusarz via dovecot]

> On 07/18/2023 9:00 AM MDT Gerald Galster  wrote:
>
> While I understand it takes effort to maintain the replication plugin, this 
> is especially problematic for small active/active high-availability 
> deployments.

To clarify: replication absolutely does not provide "active/active".  
Replication was meant to copy data to a standby server, but you can't have 
concurrent mailbox access.  This is why directors existed.


> I guess there are lots of servers that use replication for just 50 or 100 
> mailboxes. Cloudstorage (like S3) would be overkill for these.
> 
> Do you provide dovecot pro subscriptions for such small deployments?

A 50-100 mailbox user server will run Dovecot CE just fine.  Pro would be 
overkill.

All current Dovecot development assumes that storage is decoupled from the 
system.  Shared (as in network available) storage is what you need if you want 
high availability, whether in Pro or CE.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Slusarz via dovecot]

Hello all,

I want to provide a brief overview regarding various questions surrounding 
features that are being removed from Dovecot CE going forward.

We are currently working on providing updated/improved website info and 
documentation that will better explain exactly what is being maintained in CE.  
However, the desire to have unified messaging clashes with the Engineering 
Team's desire to continue to push code to the open source repository when it is 
ready...

So I want to educate on just a few points here, with the promise that further 
information will be provided in the future.

A reminder that Dovecot is commercial software, and has been since Timo made 
this decision 13 years ago.  Dovecot is not maintained by a community of 
volunteers.  We continue to be lucky that Timo remains Dovecot's Chief 
Architect today, but there are 20 dedicated Dovecot employees, plus additional 
Open-Xchange support staff, that are working on the software everyday.  This is 
carrier-grade software, which requires significant resources to maintain.

Dovecot CE is the open source version of this commercial product (currently, 
Dovecot Pro).  Dovecot CE is not a separate project - it is maintained as part 
of the day-to-day maintenance of Pro.

Every single person that works for Dovecot/OX is extremely proud and dedicated 
to releasing as much software as we can to open source.  CE is able to take 
advantage of this situation to provide features that would not be allowed in a 
purely voluntary project (for example, there are 5 full time QA people working 
on what is eventually released as Dovecot CE).

However, there remains a delicate balance of what we can openly release and 
what we need to be able to commercially provide in order to keep the lights on 
(which allows us to continue to provide open releases...).  This is a difficult 
juggling act, and is one that is always prone to recalibration in any open 
software product, not just Dovecot.

Dovecot CE has always been 100% open source, and will continue to be so.  
Nothing is changing in the future.  Dovecot CE has been, and will always 
continue to be, fully compliant with open source principles (see 
https://opensource.org/osd/).

For a variety of software, maintenance, and (yes) business reasons, there comes 
a time when decisions need to be made to move beyond existing software.  This 
is completely normal in software development, and there is no "open source" 
duty to continue to maintain software that is no longer useful (or, is broken 
or is unmaintained or is not longer best practices or is no longer commercially 
viable or is duplicative of other features that exist or )  That decision 
is what is being done for a selection of longstanding Dovecot features.  It is 
time to move on from them.  There are valid reasons to do so.

If you disagree: the software is open source.  You can continue to use the 
existing software, adapt it to your needs, move to a different solution, or 
whatever else.  

To focus development efforts, and to provide extreme clarity for users going 
forward, Dovecot CE for the first time has adopted a defined Vision Statement: 
"To provide the world's premier open source, standards compliant, 
full-featured, single node email backend server."  This vision formulation was 
made to ensure that CE users continue to receive world class, stable, tested, 
modern, secure email software going forward.  Maintaining features that have 
existed since the mid-2000s (replication, Directors), at the expense of moving 
the software forward to adapt to new paradigms (cloud, containers, 
storage-layer replication, statelessness) is not the proper choice.

These Dovecot CE feature decisions are mine.  If you are unhappy with them, I 
ask that you direct your vitriol directly (and privately) to me.  The Dovecot 
Team does fantastic work and has provided software, under open source 
principles, that runs millions of email servers around the world.  They 
continue to provide invaluable feedback internally in determining the proper 
balance between open and commercial considerations.  They deserve to be thanked 
by the community, not vilified.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Debian package for bookworm

[Michael Slusarz via dovecot]

> On 06/16/2023 9:59 AM MDT Claudio Corvino  wrote:
> 
> I updated to Debian 12 but I can't find repo for bookworm on 
> https://repo.dovecot.org/.
> When it will be released?

There will not be any releases of 2.3 for Debian 12. You will need to wait for 
2.4.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Inaccurate results while searching for a phrase in subject (fts-flatcurve)

[Michael Slusarz via dovecot]

See below.

> On 05/23/2023 2:14 AM MDT s...@fea.st wrote:
> 
> I had been using the lucene FTS plugin since a decade now and it has done me 
> well. Thought of upgrading to the new & current stuff and came across the 
> flatcurve plugin which seems very promising (xapian on the other hand was 
> creating indexes larger than my mailboxes themselves). I am using following 
> configuration in dovecot.conf:
> 
> fts = flatcurve
> fts_filters_en = lowercase english-possessive stopwords
> fts_languages = en
> fts_tokenizers = generic email-address

^^^ FTS input is being tokenized, so the phrase "/home/johndoe/render.php" will 
be indexed not as a full string but instead separately as "home", "johndoe", 
"render", and "php".

See: 
https://doc.dovecot.org/settings/plugin/fts-plugin/#plugin_setting-fts-fts_tokenizers

This has nothing to do with flatcurve (or any FTS driver) - Dovecot will never 
send the full "/home/johndoe/render.php" to the driver to be indexed.


> fts_autoindex = no
> fts_enforced = yes
> 
> A search command like this:
> 
> doveadm -D search -u j...@doe.com mailbox INBOX SUBJECT 
> "/home/johndoe/render.php"
> 
> should show the messages with subject: "CRON: /home/johndoe/render.php OK" 
> but produces a lot of extra undesired results and I think the second line in 
> this debug output indicates the reason:
> 
> May 23 07:44:13 doveadm(j...@doe.com): Debug: fts-flatcurve(INBOX): Query 
> (hdr_subject:/home/johndoe/render.php*) matches=0 uids=

This is correct, since "/home/johndoe/render.php" was not indexed so there 
should be zero results.


> May 23 07:44:13 doveadm(j...@doe.com): Debug: fts-flatcurve(INBOX): Query 
> (hdr_subject:php* AND hdr_subject:render* AND hdr_subject:johndoe* AND 
> hdr_subject:home*) matches=272 

And this is also correct, as the search phrase is attempted by searching both 
its full string and also all of its tokenized components.  (Both the original 
text and all search terms are processed through the tokenizer before passing to 
a FTS driver.)


> I tried rebuilding the indexes with "fts_flatcurve_substring_search = yes" 
> too but that didn't change anything. It works as expected with lucene plugin 
> because in that case header search is performed via dovecot indexes instead 
> of FTS. May be I am not doing something right in configuring this new FTS? 

I'm not a lucene expert... but with the old lucene plugin, you were almost 
certainly using it without Dovecot tokenization support, since the plugin 
predates it (I think) - using Dovecot tokenization would have required 
'use_libfts' to be present in the fts_lucene setting (which I doubt was ever 
documented).  I believe Dovecot was just doing simple white-space tokenization 
instead, so lucene code/library was likely receiving the full string and doing 
internal tokenization.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: RHEL9 Latest Repo?

[Michael Slusarz]

> On 07/27/2022 12:55 PM MDT dove...@ptld.com wrote:
>  
> Any plans or timeline for when there will be a latest repo for RHEL9?

There are no plans to provide RHEL9 packages in Dovecot CE 2.3.x.

RHEL9 packages will likely be provided for 2.4.  (Before it is asked, there is 
no timeline for 2.4 release.)

michael

Re: test-crypto.c - Assert failed

[Michael Slusarz]

> On 07/27/2022 12:50 AM MDT Tamsy  wrote:
> 
> On a new standard Ubuntu 22.04 LTS installation Dovecot's "configure &&
> make" runs through but "make check" fails.
> 
> Is dovecot-2.3.19.1 not yet compatible with openSSL 3.0.2 (openssl
> 3.0.2-0ubuntu1.6) or is this just happening here?

As has been discussed on this list previously, Dovecot 2.3.x is not (yet) fully 
compatible with openSSL 3.

michael

Re: large search indexer tasks, submitted to flatcurve+tika+tesseract backend for attachment scanning, timeout even with "fts_index_timeout = 0"; how to increase/remove timeouts?

[Michael Slusarz]

> On 07/23/2022 8:25 AM MDT PGNet Dev  wrote:
> 
> i'm running dovecot 2.3.19.1

[snip]

> when i exec large reindex jobs, i get occassional timeout errors on dovecot's 
> indexer-worker connection to tiks backend, e.g.,
> 
>   2022-07-23 09:54:43 
> indexer-worker(postmas...@example.com): Error: 
> fts_tika: PUT http://127.0.0.1:9998/tika/ failed: Request timed out (Request 
> queued 61.031 secs ago, 1 send attempts in 60.103 secs, 60.080 in http 
> ioloop, 0.000 in other ioloops, connected 60.103 secs ago)
>   2022-07-23 09:54:43 
> indexer-worker(postmas...@example.com): Error: 
> Mailbox Sent: Precache for UID=90782 failed: Internal error occurred. Refer 
> to server log for more information. [2022-07-23 09:54:43] (attempted to index 
> 2 messages between UIDs 90778..90782)
> 
> i don't see any fts timeout info @
> 
>   https://wiki.dovecot.org/Timeouts
> 
> here
> 
>   
> https://doc.dovecot.org/settings/plugin/fts-plugin/#plugin_setting-fts-fts_index_timeout
> 
>   "
>   fts_index_timeout
> 
>   Default: 0
> 
>   Values: Unsigned integer
> 
>   When the full text search backend detects that the index 
> isn’t up-to-date, the indexer is told to index the messages and is given this 
> much time to do so. If this time limit is reached, an error is returned, 
> indicating that the search timed out during waiting for the indexing to 
> complete: NO [INUSE] Timeout while waiting for indexing to finish
> 
>   A value of 0 means no timeout.
>   "

[snip]

> where do I set that timeout to not fail, as above, on large index tasks?

You need to change the source, as Tika has a hardcoded 60 second HTTP request 
limit.

https://github.com/dovecot/core/blob/release-2.3.19/src/plugins/fts/fts-parser-tika.c#L76

michael

Re: Is multi factor authentication practical/feasible?

[Michael Slusarz]

> On 07/07/2022 5:24 AM Aki Tuomi  wrote:
>
> FWIW I think OAuth2 is the modern way to do actually MFA authentication.  
> There is some progress in Mozilla world (and hopefully other mail clients) to 
> allow OAuth2 to work outside the "big three" circle. Mostly this is *client 
> development issue*, the server-side already mostly supports all the bits you 
> need to roll your own MFA with OAuth2 using off the shelf components. No need 
> to pay microsoft or google.
> 
> Alternate to OAuth2, which works pretty well today, is to use device 
> passwords.

A bit late to the game, but wanted to expand a bit on Aki's comments.

It's good that this topic is being discussed.  We've long felt that email 
authentication (and, related, client auto-configuration) is one of the biggest 
weaknesses of email as compared to more "modern" messaging technologies.

However, discussions around this topic tend to get sidetracked as everyone 
wants to try to design their own technical solutions.  However, all the 
necessary technologies exist and are standardized.  Token auth is handled by 
OAuth2.  MFA ,and more generally authentication UI, is handled by OpenID 
Connect.  At the mail protocol levels, token auth is handled by SASL.  
Additionally, SASL supports auto-discovery of authentication providers so there 
is no need to "hard-code" OAuth2 providers (the unfortunate way that some 
clients are currently handling OAuth2).  Dovecot supports all of these 
technologies already, so there's nothing left to do on the server side.  (Side 
note: client auto-configuration is also already fully supported using existing 
technologies as well.)

Instead, the issue is chicken/egg - all of this is worthless until 
clients/providers start implementing this.  That's where the focus of efforts 
need to be, not in trying to determine which technologies to use.

Admittedly, this not insignificant paradigm shift can be a bit confusing 
technically.  It's been a long-standing TODO to create some kind of 
implementation guide to help server/client/auth providers to understand what 
they need to do to create this new "modern email authentication" ecosystem.  
This is a classic example of a situation where necessary standards exist, but 
the education about these standards are lacking AND there remains open 
questions about how those standards should interact with each other in 
real-world scenarios.  Dynamic client registration in OpenID Connect, in 
particular, is a key component to make this work but is somewhat under 
documented and lesser known, so it will take community involvement, and likely 
trial and error, to determine best practices.

TL;DR from a Dovecot perspective: we feel we have all the necessary components 
needed to enable "modern email auth" in the current product, so we don't see 
any additional engineering efforts needed in Dovecot.  We instead are focusing 
our attention in building and supporting a broader community of client authors 
and authentication providers to push for implementation in order to accomplish 
the goal.

michael

p.s. As mentioned by Aki, app-specific/device passwords is a perfectly 
acceptable solution, although a bit of an end-user usability nightmare.  It's a 
hack to improve security today, but not a proper long-term answer.

Re: enable/control fts-tika debug logging in Dovecot 2.3.18 + Tika Server 2.4.0?

[Michael Slusarz]

> On 05/23/2022 5:27 PM PGNet Dev  wrote:
> 
> how to correctly turn on debug/verbose logging for fts-tika use in/by dovecot?

mail_debug = yes

This turns on HTTP debugging for the outgoing Tika requests.

Unfortunately, Tika has not yet been converted to events/categories with the 
ability to more granularly enable debugging just for this component.

It's probably easier to just look at tika's debugging logs.  The default log 
level (at least in Tika 2.3) will output an INFO line for every attachment 
indexed:

INFO  [qtp235162442-22] 16:15:19,905 
org.apache.tika.server.core.resource.TikaResource /tika (text/calendar)

michael

Re: Dovecot and RFC6856

[Michael Slusarz]

> On 05/17/2022 6:00 AM Tan Mientras  wrote:
> 
> Does dovecot also implement RFC8656? Will the same happen if we migrate to 
> dovecot? Any plans for future adoption of that RFC?
> 
EAI/UTF-8 support (RFC 6530 is the better, generalized document to point at) 
has been on the TODO list for a long time for Dovecot.  It simply has not yet 
risen to be a priority over other things we are working on.

michael

Re: JMAP Support Status

[Michael Slusarz]

> On 04/14/2022 11:31 AM Benny Pedersen  wrote:
> 
>  
> On 2022-04-14 19:11, Michael Slusarz wrote:
> >> On 04/13/2022 2:24 AM David Klingenberg 
> >> wrote:
> >> 
> >> has there been any development on JMAP support in Dovecot?
> > 
> >  JMAP is not currently being developed.
> 
> oh :)
> 
> cyrus-imapd have it

The cyrus developers created JMAP, so that makes sense.

michael

Re: JMAP Support Status

[Michael Slusarz]

> On 04/13/2022 2:24 AM David Klingenberg  wrote:
> 
> 
> has there been any development on JMAP support in Dovecot? JMAP is not 
> currently being developed.



michael

Re: resend whole inbox to user

[Michael Slusarz]

> On 04/06/2022 1:29 PM Marc  wrote:
> 
> I was wondering if there is some way to force an imap client to 're-download' 
> all the messages from the inbox. I can remember in the 'old days' that when 
> the connection was dropped during a pop download, the whole inbox was 
> re-downloaded, resulting in quite a lot of duplicates. I am looking for such 
> action.

You can change the UIDVALIDITY of the mailbox.  This invalidates the client 
cache which would cause a (well-behaving) client to rebuild that mailbox from 
current server state.

https://datatracker.ietf.org/doc/html/rfc3501#section-2.3.1.1

michael

Re: Dupliate-ish email search?

[Michael Slusarz]

> On 03/02/2022 12:00 PM @lbutlr  wrote:
> 
> I'm mulling over writing some code to find emails in a maildir that are 
> duplicates, ish. That is to say that sometimes the same message doesn't quite 
> show up as an exact match. Like some ad company may send you three identical 
> messages, except they aren't actually EXACTLY identical, the message-IDs are 
> different, and may the to address quoted part is different, so normal 
> duplicate finders fail to find them.
> 
> Before I start, is this a solved problem?

Besides the fact that you've pretty much described how modern AV/AS systems 
work? :)

Joking aside, isn't this what Bayesian classification is essentially doing?  
Comparing the similarities between text (via tokens) in messages and then using 
Bayesian probabilities to emphasize certain terms/relationships?  Although this 
requires training and is not comparing any messages directly...

Maybe some form of perceptual hashing (or similar idea) would work?  E.g. 
http://phash.org/

michael

Re: Lucene support for FTS - EOL date.

[Michael Slusarz]

> On 02/04/2022 4:03 AM Jacek Grabowski  wrote:
> 
> Actually on the https://doc.dovecot.org/configuration_manual/fts/lucene/ site 
> we can read about lucene fts plugin:
> 
> "This plugin is no longer maintained in Dovecot (as of 2.3+) and will be 
> removed in the future."
> 
> Does anyone know about approximate EOL date for lucene plugin support in 
> Dovecot?
> 
Realistically ... many years ago.  There has been no maintenance or testing of 
the code for years now.

michael

Re: S3 integration w Dovecot

[Michael Slusarz]

> On 01/21/2022 10:38 AM Martin Olsen  wrote:
> 
> As stated on documentation S3-compatible Storages — Dovecot documentation 
> https://doc.dovecot.org/configuration_manual/mail_location/obox/s3/ only AWS 
> S3 is officially supported.
> 
> However, we know that Scality has been successfully integrated at some larger 
> corporations.
> 
This is correct, Scality is a partner of Open-Xchange for OX Dovecot Pro 
installations, and is a supportable storage solution.


> Particularly curious if anyone has integrated with Pure S3 object storage 
> solution, or any other for that matter?
> 
This support is provided through Scality's sproxyd interface, not S3. See 
https://doc.dovecot.org/admin_manual/obox/scality_key_format/  
https://doc.dovecot.org/admin_manual/obox/scality_key_format/.  sproxyd allows 
us to provide a fully geo-redundant solution that can support Dovecot and the 
mail storage in multiple active data centers.  S3 support is limited to a 
single data center/region due to limitations of the S3 protocol.

For OX Dovecot Pro, we only support "AWS S3" for SLA purposes.  This is for 
several reasons:
* There is no "official" S3 spec. Thus, you have to certify against specific 
vendors - it is not possible to certify generally against the protocol.  Be 
very wary of a provider that claims they are "fully AWS S3" compliant - what 
does that mean if there is no formalized spec?
* Experience has shown us that many S3 installations cannot handle the mail 
loads in larger Dovecot installations.  After they reach specific loads, they 
can enter a death spiral and/or become unresponsive.
* It is impossible to support a customer that just installs S3 software 
themselves and creates their own storage array.  This is a custom/bespoke 
project.  The purpose of the OX Dovecot Pro product is to provide a full email 
system out of the box without the need for custom development.
michael

Re: FTS Tokenization filters normalizer-icu vs lowercase

[Michael Slusarz]

> On 01/20/2022 9:20 AM Alessio Cecchi  wrote:
> 
> I'm trying to setup fts-flatcurve with tokenization.
> 
> What are the differences/benefits with "fts_filters = normalizer-icu" vs 
> "fts_filters = lowercase"?
> 
> Reading the Doc I found about normalizer-icu "This is potentially very 
> resource intensive." and about lowercase "Supports UTF8, when compiled with 
> libicu".
> 
> So, using lowercase is almost the same that normalizer-icu but faster?
> 
No, these are 2 different actions.

Lowercase tries to use language rules to map characters to a "lowercase" 
equivalent, which is character/language dependent.

Normalization tries to take a string and reduce it to a unique, normalized 
form, that can be directly compared to other normalized strings.  UTF, for 
example, can have strings that display the same to the user but contain very 
different byte data.  For example, it is possible to create more complicated 
glyphs by either using a specific code-point (i.e., a 4 byte UTF element) or by 
using a combination of UTF sequences that, when combined, create an identical 
display of the character.

Normalization is a very complicated topic.  
https://en.wikipedia.org/wiki/Unicode_equivalence might help with further 
understanding.

The ICU library deals with general internationalization support, and these two 
filters are using different parts of that library to do different things.  They 
are not replacements for each other, they are complimentary - you could 
normalize a string and then lowercase it, for example.

michael


> 
> 
> FYI
> 
> for using fts-flatcurve with dovecot RPM packages from repo.dovecot.org you 
> have to rebuild with --with-icu --with-stemmer --with-textcat and related 
> library.
> 
> Thanks
> 
> --
> Alessio Cecchi
> Postmaster @ http://www.qboxmail.it
> https://www.linkedin.com/in/alessice
> 

Re: FTS flatcurve not index messages when SEARCH run on Virtual Mailboxes

[Michael Slusarz]

> On 01/17/2022 1:24 AM Aki Tuomi  wrote:
> 
> > On 17/01/2022 10:09 Alessio Cecchi  wrote:
> > So my question is, there is a specific command for switch from an FTS 
> > plugin to another or there is a bug in this procedure?
> > Thanks
> > 
> fts rescan actually just deletes fts indexes, it does not actually rescan 
> anything.
> 
> to do full rescan, you need to 
> 
> doveadm fts rescan -u user
> doveadm index -u user "*"

To be a bit more exact (at least for flatcurve), doveadm fts rescan deletes all 
messages above the lowest, non-indexed UID.

Unfortunately, the current Dovecot FTS API does not allow those deleted indices 
to be indexed programatically, so it requires a separate, external trigger to 
index the UIDs.  So, Aki's command chain is the necessary series of commands 
that are needed at the current time.

This brings up an old TODO I had in looking into what it would take to trigger 
indexing via the internal API.  Or whether that's even a good idea.

michael

Re: FTS flatcurve not index messages when SEARCH run on Virtual Mailboxes

[Michael Slusarz]

> On 01/11/2022 9:36 AM Alessio Cecchi  wrote:
> 
> 
> When I SEARCH on Virtual/All result are empty or contain only messages from 
> previously index mailboxes. I notice also that when SEARCH on Virtual/All 
> directory "fts-flatcurve" is created but is empty:
> 
Can you open a ticket at 
https://github.com/slusarz/dovecot-fts-flatcurve/issues?  FTS drivers don't 
require code to handle virtual mailboxes separately, but it is possible that 
this is hitting a code path in flatcurve that isn't touched with single mailbox 
queries so there could very well be a bug.

michael

Re: special-use missing in 2.3 imap connection

[Michael Slusarz]

> On 11/22/2021 1:10 PM Marc  wrote:
> 
> Should I specify special_use any where, it is missing from my telnet 
> imap.local 143
> 
> 2.2: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE 
> IDLE SPECIAL-USE STARTTLS AUTH=PLAIN] Dovecot ready.
> 
> 2.3: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE 
> IDLE STARTTLS AUTH=PLAIN] Dovecot ready.

This is pre-auth output correct?  SPECIAL-USE is only valid in 
authenticated/selected state, so there's no reason to list before that.

Looking at the code though, I don't see that this was ever output by Dovecot in 
the capability banner, even in 2.2.  Do you have 'imap_capability = 
+SPECIAL-USE' as a setting?

FYI: ENABLE and IDLE are only valid in authenticated/selected state, so you may 
ask why they are listed as well (at least I did).

IDLE is listed due to client issue.  From configure.ac:

dnl IDLE doesn't really belong to banner. It's there just to make Blackberries
dnl happy, because otherwise BIS server disables push email.

(heh, not sure how useful this is anymore)

ENABLE is listed since it can be pipelined with an auth/login.

michael

Re: full-text index and search

[Michael Slusarz]

If substring_search is true, the indexes can unfortunately be quite large (40%+ 
of mailbox data size) - this is because Xapian does not natively support 
substring searching so we have to hack/fake it by storing redundant data in the 
index.

If substring_search is false, storage size is generally < 10% of mailbox 
storage size.

There's some (older) benchmarking on this at 
https://github.com/slusarz/dovecot-fts-flatcurve#indexing-benchmark-with-substring-matching-enabled-default-configuration

Obviously, this is dependent on the local mix of message data you are indexing. 
 The amount of attachments, language, the media type of text parts (e.g. plain 
vs. html), etc. are all variables that may change storage size.

I don't know how storage compares with Solr.  flatcurve and Solr are two 
completely different use-cases however, so I'm not sure how useful that 
comparison is.

michael


> On 11/03/2021 2:26 PM Marc  wrote:
> 
> 
> 
> Is there some info on what to expect how big these indexes can get (% 
> mailbox)? Is there any differences between solr / xapian storage use?
> 
> https://github.com/slusarz/dovecot-fts-flatcurve/
> 

Re: Solr FTS - message deletes not working as expected

[Michael Slusarz]

> On 11/03/2021 12:56 PM Shawn Heisey  wrote:
> 
> > Thunderbird does NOT necessarily process expunges immediately.  Depends on 
> > what else it is doing in the background.  So you can't click delete in the 
> > UI and not immediately see anything on the backend and definitively 
> > correlate the two.
> 
> The message is deleted by dovecot immediately.  I double checked this by 
> purging a message on my Linux client and saw the message immediately 
> disappear on my Windows client.  It happened even faster than I would 
> have expected.  IMAP seems to be a very good protocol!  I can't say it 
> definitively without more evidence, but the problem *seems* to be in 
> FTS, not imap or core dovecot.

For Solr, there's a code path in the FTS expunge code that will silently toss 
expunge requests:

if (ctx->last_indexed_uid == 0 ||
uid > ctx->last_indexed_uid + 100) {
/* don't waste time asking Solr to expunge a message that is
   highly unlikely to be indexed at this time. */
return;
}

So it's possible you are running into that.

michael

Re: Solr FTS - message deletes not working as expected

[Michael Slusarz]

Have you tried another client?

Thunderbird does NOT necessarily process expunges immediately.  Depends on what 
else it is doing in the background.  So you can't click delete in the UI and 
not immediately see anything on the backend and definitively correlate the two.

Another option is to ensure debug logging is enabled for Dovecot so you can see 
what the FTS code is doing.  "log_debug = category=fts" and/or "mail_debug = 
yes" will help in that regard.

michael


> On 11/02/2021 9:16 AM Shawn Heisey  wrote:
> 
>  
> On 10/28/21 8:00 AM, Shawn Heisey wrote:
> > Also, when I send a message with Thunderbird, which deletes the 
> > message in Drafts and adds one to Sent, I am not seeing a delete 
> > request in the Solr log.  I do see the add. So this isn't limited to 
> > just the shift-delete workflow. 
> 
> 
> I have confirmed this with multiple attempts.
> 
> I start a new message in Thunderbird.  Then I wait around for that 
> message to be auto-saved to Drafts.  When that happens, I see an "add" 
> request in the solr log.
> 
> Then I send the message.  At that point, I see another add in Solr's 
> log.  Based on the message number in the add request, I know that this 
> time the add happens in the Sent folder.  But despite the fact that 
> Thunderbird deletes the message from Drafts, Solr never sees a delete 
> request.  My dovecot version has been updated since the last time I 
> indicated what version it is.  Now it is "2:2.3.17-3+ubuntu20.04" from 
> the dovecot repo, not the ubuntu repo.  The same thing happened with 2.3.16.
> 
> As I mentioned in the first message on this thread, when shift-delete is 
> used in Thunderbird to delete messages, that also never sends a delete 
> to Solr.
> 
> Something somewhere is misbehaving.  Is it Thunderbird accessing IMAP 
> incorrectly, or is it Dovecot?
> 
> I will do a packet capture to see if maybe dovecot is sending requests 
> that are not logged by Solr.  That seems unlikely -- even bad requests 
> should result in some kind of entry in the solr log.
> 
> Thanks,
> Shawn

Re: Duplicate plugins - FTS Xapian

[Michael Slusarz]

See below.

> On 08/31/2021 5:35 AM Aki Tuomi  wrote:
> 
> > On 31/08/2021 13:59 Tom Hendrikx  wrote:
> >   
> > On 31-08-2021 12:01, Aki Tuomi wrote:
> > > 
> > >> On 31/08/2021 10:56 Felix Zielcke  wrote:
> > >>
> > >> Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi:
> > >>>
> >  On 31/08/2021 00:11 Joan Moreau  wrote:
> > 
> >  Hi
> >  There seems to be 2 plugins doing the same thins
> >  - https://github.com/slusarz/dovecot-fts-flatcurve/
> >  - https://github.com/grosjo/fts-xapian/ (mine)

...

> >  Isn't there double work here ?

FYI, Joan also reached out to me on the Github project page, so you can see my 
response to this question there:

https://github.com/slusarz/dovecot-fts-flatcurve/issues/5

Short answer: I see the plugins as "different".  Yes, they both use the same 
underlying indexing technology (Xapian), but the design goals, features, and 
implementations are unique for each project.  Thus, I don't see them as 
duplicate work.


> > >> Is there somewhere a direct comparison of them?
> > >> I currenty use fts-xapian from Joan without problems.
> > >> But what would be the advantages of fts-flatcurve over fts-xapian?

I was asked a similar question previously.  See 
https://github.com/slusarz/dovecot-fts-flatcurve/issues/4#issuecomment-902425597


> > > fts_flatcurve does only full word searching, although you can use 
> > > fts_filters and fts_tokenizers settings to affect stemming and other 
> > > matching to make it work with plurals and such.

Aki's knowledge is a bit dated - fts_flatcurve now supports both prefix 
matching (this is what Xapian provides by default) AND full substring matching 
(this is what is technically required by the IMAP RFC).

Although the current fts_flatcurve default is for full substring matching, it 
is debatable whether that is the correct choice.  Users tend to think of search 
in terms of "prefix" matching - that's how Google works - and substring 
matching requires significant additional storage since Xapian doesn't support 
it natively.


> > I still think it's weird to see that Open-Xchange starts a FTS Xapian 
> > plugin with mostly the same basic functionality that is already 
> > available in an existing plugin maintained by someone in the community 

As Aki noted, Open-Xchange didn't "start" anything.  fts_flatcurve is entirely 
a project I have developed on my personal time.

As a Dovecot CE user, I had the same complaint that most people have - in 
current versions of Dovecot, there is no easy-to-use FTS option that works 
out-of-the-box.  Solr is the only search driver still actively supported, and 
it requires significant additional resources and configuration to setup.

Unfortunately, we don't have the resources within Open-Xchange to create a 
CE-only search option as our customers generally use our proprietary, object 
storage aware FTS system that is part of OX Dovecot Pro.  Without commercial 
demand, it is difficult to justify paid work on features that do not produce 
revenue in return.  (fts_flatcurve does use Dovecot core features that are 
maintained for the Pro FTS system, like stemming, so we can take advantage of 
trickle down benefits from the commercial code maintenance.)

My goal was to give back to the community by providing an easy-to-use, resource 
sensitive, no-maintenance option to address this current product limitation.

Hopefully, fts_flatcurve will fit the requirements for most smaller 
installations that run Dovecot that don't require the carrier/enterprise grade 
FTS solutions large customers demand.


> > Especially if that happens without any (apparent) communication with the 
> > existing plugin developer to find out whether fixing the issues that 
> > slusarz/Open-Xchange seem to have with the existing plugin, can be fixed.

There were fundamental disagreements about design goals, and Joan (or anybody, 
really) would likely not have appreciated someone coming in and demanding that 
the entire architecture and design of the project needed to be changed based on 
my personal opinions.

This is open source.  Everybody should be free to participate as they wish, as 
long as you are not negatively affecting someone else.  I don't think that has 
happened here.


> > Combining forces just seems a better way to spend scarce development 
> > resources than building something similar (but different) without any 
> > communication.

I disagree, but that's ok.  People can do that.

michael

Re: quota_grace

[Michael Slusarz]

> On 07/29/2021 1:26 PM dove...@ptld.com wrote:
> 
> How would you disable quota grace?
> quota_grace = 0
> quota_grace = 0%
> quota_grace = 0m
> quota_grace = -1

Either of the first three would work.  They all resolve to a value of 0 - 
quota_grace adjusts the existing root limits by whatever factor quota_grace is 
(either a percentage or a defined size).

michael

Re: Dovecot v2.3.15 released

[Michael Slusarz]

> On 06/22/2021 10:50 AM Alessio Cecchi  wrote:
> 
>  
> Il 21/06/21 13:18, Timo Sirainen ha scritto:
> >  + imap: Support official RFC8970 preview/snippet syntax. Old methods of
> >    retrieving preview information via IMAP commands ("SNIPPET and PREVIEW
> >    with explicit algorithm selection") have been deprecated.
> 
> Hi,
> 
> After upgrading dovecot from 2.3.14 to 2.3.15 I noticed a problem 
> parsing FETCH response of PREVIEW attribute.
> 
> Basically there is any space after the preview content and the rest of 
> the string which causes issues on parsing:
> 
> a UID FETCH 2539 (MODSEQ UID FLAGS INTERNALDATE PREVIEW 
> BODY.PEEK[HEADER.FIELDS (FROM TO SUBJECT DATE)])
> * 8 FETCH (UID 2539 MODSEQ (3) FLAGS (\Seen $HasNoAttachment) 
> INTERNALDATE "04-Mar-2021 12:18:02 +0100" PREVIEW 
> "test"BODY[HEADER.FIELDS (FROM TO SUBJECT DATE)] {151}
> 
> With dovecot 2.3.14 there was no problem:
> 
> a UID FETCH 2539 (MODSEQ UID FLAGS INTERNALDATE PREVIEW 
> BODY.PEEK[HEADER.FIELDS (FROM TO SUBJECT DATE)])
> * 8 FETCH (UID 2539 MODSEQ (3) FLAGS (\Seen $HasNoAttachment) 
> INTERNALDATE "04-Mar-2021 12:18:02 +0100" PREVIEW (FUZZY "test") 
> BODY[HEADER.FIELDS (FROM TO SUBJECT DATE)] {151}

Yes, this is an unfortunate regression if PREVIEW is not the last FETCH option 
listed in the command.

Fix is:

diff --git a/src/imap/imap-fetch-body.c b/src/imap/imap-fetch-body.c
index 59a9b24d99..39e66f093d 100644
--- a/src/imap/imap-fetch-body.c
+++ b/src/imap/imap-fetch-body.c
@@ -648,6 +648,7 @@ fetch_snippet(struct imap_fetch_context *ctx, struct mail 
*mail,
str_append(ctx->state.cur_str, "NIL");
if (preview->old_standard)
str_append(ctx->state.cur_str, ")");
+   str_append_c(ctx->state.cur_str, ' ');

return 1;
  }

michael

Re: auth module logging

[Michael Slusarz via dovecot]

> On August 2, 2019 9:35 PM AP via dovecot  wrote:
> 
> As per my discussion with cmouse on irc, I'm currently just using
> dovecot for its auth mechanism, etc capabilities (alas, Exim does
> not support argon directly). Tis a fun little side project. :)
> 
> I have the config pared down as far as I think is reasonable (see
> below) and all is working well except that dovecot is very silent.
> 
> Errors hit the logs but I would appreciate seeing successful auths
> happen for the additional piece of mind. Cmouse and I couldn't
> find a way to do it on irc and it appears that the capability is
> missing. Successul /logins/ can be logged but auths, by themselves,
> cannot.
> 
> I would appreciate if the ability was added.
> 
> Dovecot 2.3.7.1 is in use.

Events (using event exporter) is probably what you want, new in 2.3.7.

https://doc.dovecot.org/admin_manual/list_of_events/

michael

RE: Dovecot eBook

[Michael Slusarz via dovecot]

 
 
  
   You need to talk to Peer and/or the publisher.  We are not affiliated with that book.
   
  
  
   
  
  
   michael
   
  
  
   
  
  
   

   
   

   One can easily find some PDFs on the net. But all of them looks outdated.
   
   http://gen.lib.rus.ec/book/index.php?md5=AB28964F8CB3CC8BC319BBDE25C8B41A
   
   * Do not forget to support writers by buying their books.
   
   
   
On July 21, 2019 12:09:24 AM GMT+03:00, Peter Fraser via dovecot  wrote:

 
  Well, if there’s no other way, I guess I will have to.
  
  Thanks.
  
  Sent from Mail for Windows 10
  
  
   From: LuKreme via dovecotSent: Saturday, July 20, 2019 3:33 AMTo: Peter Fraser via dovecotSubject: Re: Dovecot eBook
  
  
  On Jul 19, 2019, at 19:29, Peter Fraser via dovecot  wrote:
  
   

 I have a strange question. I bought the Dovecot Book off Amazon. I can easily prove it with a picture and/or my receipt off Amazon. I still have it o my library but I don’t like to travel around with it. Is there a way for me to get a PDF copy? I just checked Amazon and there is still no PDF version available there.

   
  
  
   
  
  Tedious, but scan the book. I have done this with my iPhone and it resulted in a very good copy that was fully OCRed
  
   
  
  
  
 

   
   Eugene Bright
   IT-engineer
   Tel.: +7 925 728 96 22
   
   
 

Re: Getting login stats

[Michael Slusarz via dovecot]

> I'm trying to get some IMAP auth stats on a Dovecot 2.3.6 instance, but 
> whatever I declare in metric, it always show 0.

None of these auth_* requests exist in 2.3.6.


> I tried using the following metrics:
> 
> 
> 
> metric auth_request_finished {
> event_name = auth_request_finished
> }
> 
> metric auth_passdb_request_finished {
> event_name = auth_passdb_request_finished
> }
> 
> metric auth_userdb_request_finished {
> event_name = auth_userdb_request_finished
> }
> 
> metric auth_client_request_started {
>  event_name = auth_client_request_started
> }
> 
> metric auth_client_userdb_lookup_started {
> event_name = auth_client_userdb_lookup_started
> }
> 
> metric auth_client_passdb_lookup_started {
>  event_name = auth_client_passdb_lookup_started
> }
> 
> metric auth_client_cache_flush_started {
>  event_name = auth_client_cache_flush_started
> }
> 
> metric imap_command_finished {
> event_name = imap_command_finished
> filter {
> name = LOGIN
> }
> }
> 
> 
> But even after many successful logins, doveadm reports 0 for all events:
> 
> metric_name   fieldcount sum min max avg  median 
> stddev %95
>   
> auth_request_finished duration 0 0   0   0   0.00 0  0.00 
>   0   
>  
> auth_passdb_request_finished  duration 0 0   0   0   0.00 0  0.00 
>   0   
>  
> auth_userdb_request_finished  duration 0 0   0   0   0.00 0  0.00 
>   0   
>  
> auth_client_request_started   duration 0 0   0   0   0.00 0  0.00 
>   0   
>  
> auth_client_userdb_lookup_started duration 0 0   0   0   0.00 0  0.00 
>   0   
>  
> auth_client_passdb_lookup_started duration 0 0   0   0   0.00 0  0.00 
>   0   
>  
> auth_client_cache_flush_started   duration 0 0   0   0   0.00 0  0.00 
>   0   
>  
> imap_command_finished duration 0 0   0   0   0.00 0  0.00 
>   0

Re: Dovecot release v2.3.6

[Michael Slusarz via dovecot]

> On April 30, 2019 at 12:06 PM "@lbutlr via dovecot"  
> wrote:
> 
> pkg adult shows the following, not mentioned in the changes:
> 
> dovecot-2.3.5.1 is vulnerable:
> dovecot -- json encoder crash
> CVE: CVE-2019-10691
> WWW: 
> https://vuxml.FreeBSD.org/freebsd/a64aa22f-61ec-11e9-85b9-a4badb296695.html

https://dovecot.org/pipermail/dovecot-news/2019-April/000407.html

michael

Re: SMTPUTF8 support

[Michael Slusarz via dovecot]

Dovecot does not work as the "final mile" delivery host in a fully-compliant 
SMTPUTF8 chain.

This is on the (long) list of things todo.

michael


> On April 6, 2019 at 6:53 AM "@lbutlr via dovecot"  wrote:
> 
> 
> On 5 Apr 2019, at 13:47, André Rodier via dovecot  wrote:
> > 
> >> root@portal:/etc/postfix# postmap -q andré@homebox.space 
> >> ldap:/etc/postfix/ldap-aliases.cf 
> >> andre@homebox.space
> 
> You have a solution that properly maps UTF to a non-UTF namespace. As I 
> understand it, SMTPUTF8 uses a UTF8 namespace.
> 
> So, this would indicate SMTPUTF8
> 
> # postmap -q andré@homebox.space ldap:/etc/postfix/ldap-aliases.cf 
> andré@homebox.space
> 
> It's a hard problem to solve because you have to, as you've already 
> discovered, replace the entire chain with SMTPUTF8 compatible tools all at 
> once.

Re: SMTPUTF8 support

[Michael Slusarz via dovecot]

> On April 3, 2019 at 10:12 PM sylvhem--- via dovecot  
> wrote:
> 
> I'm currently trying to set up SMTPUTF8 on my mail stack, but I can't 
> find any information on Dovecot's RFC 6531 support. Has it been 
> implemented yet? 

No.

michael

Re: Repository / builds for Debian "buster"?

[Michael Slusarz via dovecot]

Support for Buster will be added once 10/buster is a stable release.

We only provide builds for the current stable version, and the previous 
version, so Debian 8/jessie support will be dropped when this happens.  But it 
is premature to do that now.

michael


> On March 31, 2019 at 6:04 AM Kostya Vasilyev via dovecot 
>  wrote:
> 
> 
> Speaking of repositories...
> 
> Debian "buster" (future version 10) has for some time been "stage 1 frozen" 
> (if I have the terminology right) and quite usable.
> 
> Perhaps it's starting to make sense to make a repo / start producing builds?
> 
> -- 
> Kostya Vasilyev
> k...@fastmail.com

Re: Dovecot crashing when attempting to search in virtual folder with fts_squat activated

[Michael Slusarz via dovecot]

fts_squat was deprecated in 2.1.  There's a high likelihood it is buggy in a 
variety of ways in any recent Dovecot release.

michael


> On March 20, 2019 at 1:21 PM Benjamin Godbersen via dovecot 
>  wrote:
> 
> Hi everyone,
> 
> I have now updated to dovecot 2.3.4.1 - unfortunately the issue still 
> persists. Can anyone help me figure out if this is due to a misconfiguration 
> on my part or another error?
> 
> Any help is greatly appreciated!
> 
> Cheers
> Benjamin
> 
> Am 14.03.2019 um 23:28 schrieb benja...@godbersen.info 
> mailto:benja...@godbersen.info :
> 
> > > Hi everyone,
> > 
> > I am running into a problem when trying to use fts_squat in a 
> > virtual folder. Without fts_squat plugin the search (from, subject...) 
> > works in all folders. With activated fts the search on the inbox folders 
> > works expectedly well but any attempt to search anything in any virtual 
> > folder leads to the following error. Similarly when attempting "doveadm fts 
> > lookup". I also noticed that no search index for the virtual folders gets 
> > build - is this expected behaviour?
> > 
> > > > > Mar 14 23:14:58 *** dovecot: service=imap, user=***, 
> > ip=[::1]. Panic: file mail-storage.c: line 1913 (mailbox_get_open_status): 
> > assertion failed: (box->opened)
> > > Mar 14 23:14:58 *** dovecot: service=imap, user=***, 
> > > ip=[::1]. Error: Raw backtrace: 
> > > /usr/lib/x86_64-linux-gnu/dovecot/libdovecot.so.0(+0xba731) 
> > > [0x7f553a7ff731] -> 
> > > /usr/lib/x86_64-linux-gnu/dovecot/libdovecot.so.0(+0xba7fa) 
> > > [0x7f553a7ff7fa] -> 
> > > /usr/lib/x86_64-linux-gnu/dovecot/libdovecot.so.0(i_fatal+0) 
> > > [0x7f553a771638] -> 
> > > /usr/lib/x86_64-linux-gnu/dovecot/libdovecot-storage.so.0(mailbox_get_open_status+0x68)
> > >  [0x7f553aae4a78] -> 
> > > /usr/lib/dovecot/modules/lib21_fts_squat_plugin.so(+0x3684) 
> > > [0x7f553677a684] -> 
> > > /usr/lib/dovecot/modules/lib21_fts_squat_plugin.so(+0x3820) 
> > > [0x7f553677a820] -> 
> > > /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_backend_lookup_multi+0x163)
> > >  [0x7f5539b016a3] -> 
> > > /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xd728) [0x7f5539b06728] -> 
> > > /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_search_lookup+0xeb) 
> > > [0x7f5539b06bbb] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xf8b8) 
> > > [0x7f5539b088b8] -> dovecot/imap(imap_search_start+0x6a) [0x5654cb5a0d6a
 ] -> dovecot/imap(cmd_sort+0x293) [0x5654cb593553] -> 
dovecot/imap(command_exec+0x64) [0x5654cb599874] -> dovecot/imap(+0x1bd22) 
[0x5654cb597d22] -> dovecot/imap(+0x1bdbc) [0x5654cb597dbc] -> 
dovecot/imap(client_handle_input+0x1b5) [0x5654cb5981c5] -> 
dovecot/imap(client_input+0xa4) [0x5654cb5987e4] -> 
/usr/lib/x86_64-linux-gnu/dovecot/libdovecot.so.0(io_loop_call_io+0x69) 
[0x7f553a8174a9] -> 
/usr/lib/x86_64-linux-gnu/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e)
 [0x7f553a818d1e] -> 
/usr/lib/x86_64-linux-gnu/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) 
[0x7f553a8175ac] -> 
/usr/lib/x86_64-linux-gnu/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7f553a8177b8] -> 
/usr/lib/x86_64-linux-gnu/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f553a7940a3] -> dovecot/imap(main+0x339) [0x5654cb58a539] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f553a375b97] -> 
dovecot/imap(_start+0x2a) [0x5654cb58a71a]
> > > Mar 14 23:14:58 *** dovecot: service=imap, user=***, 
> > > ip=[::1]. Fatal: master: service(imap): child 6436 killed with signal 6 
> > > (core dumps disabled)
> > > 
> > > > > This is my config:
> > 
> > > > > # 2.3.0.1 (ffd8a29): /etc/dovecot/dovecot.conf
> > > # Pigeonhole version 0.5.0.1 (d33dca20)
> > > # OS: Linux 4.15.0-46-generic x86_64 Ubuntu 18.04.2 LTS ext4
> > > auth_mechanisms = plain login digest-md5 cram-md5 apop
> > > auth_username_chars = 
> > > abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890&.-_@'
> > > default_vsz_limit = 8096 M
> > > disable_plaintext_auth = no
> > > first_valid_uid = 30
> > > imap_client_workarounds = delay-newmail
> > > imap_logout_format = rcvd=%i, sent=%o
> > > mail_home = /var/qmail/mailnames/%Ld/%Ln
> > > mail_location = maildir:/var/qmail/mailnames/%Ld/%Ln/Maildir
> > > mail_log_prefix = "service=%s, user=%u, ip=[%r]. "
> > > mail_max_userip_connections = 100
> > > mail_plugins = quota fts fts_squat virtual
> > > managesieve_logout_format = rcvd=%i, sent=%o
> > > managesieve_notify_capability = mailto
> > > managesieve_sieve_capability = fileinto reject envelope 
> > > encoded-character vacation subaddress comparator-i;ascii-numeri$
> > > namespace inbox {
> > >   inbox = yes
> > >   location =
> > >  

Re: index problems after update

[Michael Slusarz via dovecot]

> On February 15, 2019 at 3:35 AM Hajo Locke via dovecot  
> wrote:
> 
> Thanks for your reply, so iam not the only one. I think downgrading is 
> not an option for me. Ubuntu 18.04 comes with openssl 1.1.0 and it seems that 
> older versions like 2.2.22 not compile successful.
> Best way for me would be a patch from Ubuntu-side, but without any 
> errormessage or something its wasted time to file a bugreport.  Though Ubuntu 
> 18.04 should run for next 5 years.
> 
https://repo.dovecot.org/

michael

Re: Using SHA256/512 for SQL based password

[Michael Slusarz via dovecot]

> On February 12, 2019 at 4:33 PM Robert Moskowitz via dovecot 
>  wrote:
> 
> On 2/12/19 6:03 PM, Matthias Fechner via dovecot wrote:
> > Am 12.02.2019 um 17:05 schrieb Robert Moskowitz via dovecot:
> >> I have trying to find how to set the dovecot-sql.conf for using
> >> SHA256/512.  I am going to start clean with the stronger format, not
> >> migrate from the old MD5.  It seems all I need is:
> > you maybe would like to have a look to the hashing algo ARGON2I which is
> > currently recommended for new developments and deployments.
> 
> Recommended by whom?
> 
> Can you provide a link?

https://password-hashing.net/

michael

Re: Dovecot v2.2.36.1 released

[Michael Slusarz]

> On February 5, 2019 at 8:36 AM Eric Broch  wrote:
> 
> What's the difference between 2.2.x and 2.3.x version of Dovecot? And 
> why do you maintain both?

https://dovecot.org/pipermail/dovecot-news/2018-August/000386.html

michael

FOSDEM

[Michael Slusarz via dovecot]

Hello all,

Several Open-Xchange/Dovecot folks will be attending FOSDEM in Brussels on 2-3 
February 2019.

For those of you planning to attend: we would love to meet and chat with 
members of the Dovecot community!  Come find us wandering around the talks, or 
look for us in the evening.  Rumor is Brussels has some good beer...

Also, we'd love to invite everyone to a talk we are giving on the Chat Over 
IMAP initiative we have been working on.  It takes place Sunday at 1105 in the 
Real-Time Communications room:

https://fosdem.org/2019/schedule/event/chat_over_imap/

Hope to see some of you there!

michael

Re: Solr -> Xapian ?

[Michael Slusarz]

Maybe a dumb question (I admit I haven't followed this thread very closely)...

But why are you writing a new FTS driver?  If squat allegedly does everything 
you need it to do, why don't you just take that plugin and fix it up to do what 
you need?  That seems way easier than trying to create a FTS driver from 
scratch.

michael


> On January 7, 2019 at 7:05 AM Joan Moreau via dovecot  
> wrote:
> 
> 
> Hi
> 
> ANyone to answer specifically ?
> 
> Q1 : get_last_uid -> Is this the last UID indexed (which may be not the 
> greatest value), or the gratest value (which may not be the latest) (the code 
> of existing plugins is unclear about this, Solr looks for the greatest for 
> insance)
> 
> Q2 : WHen Indexing an email, the data is not passed by "build_key". Why 
> so ? What is the link with "build_more" ?
> 
> Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least 
> among "cc, to, from, subject, body") is not appearing in the 'struct' data. 
> WHere to find it ?
> 
> Q4 : Refresh : this is very unclear. How come there would not be the 
> "latest" view on index. What is the real meaning of this function ?
> 
> Q5 : Rescan : is it just a bout remonving all indexes for a specific 
> mailbox ?
> 
> Q6 : lokkup_multi : isn't the function the same for all plugnins (see 
> below) ?
> 

Re: QUERY string clarification

[Michael Slusarz]

> On December 9, 2018 at 3:02 AM Ralf Hildebrandt  
> wrote:
> 
> 
> The manpage
> https://wiki.dovecot.org/Tools/Doveadm/SearchQuery
> specifies:
> 
> BODY string
> Matches messages, which contain string in the body part.
> 
> and a bit further down:
> 
> TEXT string
> Matches messages, which contain string in the message body.
> 
> Where's the difference?

BODY = body part (meaning, the non-header part of the message)
TEXT = entire message (header + body)

michael

Re: BUG: sieve does not set seen-Flag

[Michael Slusarz]

> On December 4, 2018 at 8:48 AM Jakobus Schürz  
> wrote:
> 
> I have an additional information.
> 
> Moving a already seen messsage to another imap-folder (drag&drop in
> thunderbird) shows this email in the other folder as unseen and recent.
> 
> Is this normal behaviour?

Depends on what Thunderbird is doing.  If it is doing a COPY, flags will be 
preserved per IMAP spec.  If it is doing an APPEND, then Thunderbird would be 
responsible for transferring the flags (and clearing the recent flag) to match 
COPY semantics.

michael

Re: [2.3.4] Segmentation faults

[Michael Slusarz]

> On December 4, 2018 at 8:18 AM Aki Tuomi  wrote:
> 
> We don't consider it as "very early beta". We consider it production ready. 
> It is bit more work to set up though.

FWIW, Squat has been deprecated since 2.0, so none of this should come as a 
surprise.

https://wiki.dovecot.org/Plugins/FTS


> Aki
> 
> > On 04 December 2018 at 17:16 Joan Moreau  wrote:
> > 
> > 
> > Thanks for mySql 
> > 
> > For Squat vs Solr, Solr does not reach Squat by very far in terms of
> > results : If I setup Solr, and search (via the search in Roundbube or
> > Evolution) for a keyword or part of the keyword, the results are
> > complete non sense. The diference between "search in full body" and
> > "search in fields" does not even work. 
> > 
> > Solr with Dovecot seems very early beta isnt'it ? 
> > 
> > On 2018-12-04 15:57, Aki Tuomi wrote:
> > 
> > > On 04 December 2018 at 16:46 Joan Moreau via dovecot 
> > >  wrote:
> > > 
> > > Hi 
> > > 
> > > How to solve this ? 
> > > 
> > > So many similar segfaults 
> > > 
> > > Thank you 
> > > 
> > > On 2018-11-30 06:11, Joan Moreau wrote:
> > > 
> > > ANother (very very long) example : 
> > > 
> > > # gdb /usr/libexec/dovecot/indexer-worker 
> > > core.indexer-worker.0.3a33f56105e043de802a7dfcee265a07.21017.154353342400
> > > GNU gdb (GDB) 8.2
> > > Copyright (C) 2018 Free Software Foundation, Inc.
> > > License GPLv3+: GNU GPL version 3 or later 
> > > 
> > > This is free software: you are free to change and redistribute it.
> > > There is NO WARRANTY, to the extent permitted by law.
> > > Type "show copying" and "show warranty" for details.
> > > This GDB was configured as "x86_64-pc-linux-gnu".
> > > Type "show configuration" for configuration details.
> > > For bug reporting instructions, please see:
> > > .
> > > Find the GDB manual and other documentation resources online at:
> > > . 
> > > 
> > > For help, type "help".
> > > Type "apropos word" to search for commands related to "word"...
> > > Reading symbols from /usr/libexec/dovecot/indexer-worker...done.
> > > [New LWP 21017]
> > > [Thread debugging using libthread_db enabled]
> > > Using host libthread_db library "/usr/lib/libthread_db.so.1".
> > > Core was generated by `dovecot/indexer-worker'.
> > > Program terminated with signal SIGSEGV, Segmentation fault.
> > > #0 0x7f768b62b13e in file_lock_do (fd=18, path=0x564540376790 
> > > "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search",
> > >  lock_type=0, 
> > > lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, 
> > > error_r=0x7fff045010b0) at file-lock.c:173
> > > 173 {
> > > (gdb) bt full
> > > #0 0x7f768b62b13e in file_lock_do (fd=18, path=0x564540376790 
> > > "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search",
> > >  lock_type=0, 
> > > lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=60, 
> > > error_r=0x7fff045010b0) at file-lock.c:173
> > > lock_type_str = 
> > > started = 
> > > ret = 
> > > __func__ = "file_lock_do"
> > > #1 0x7f768b62b5b6 in file_wait_lock_error (fd=18, path=0x564540376790 
> > > "/data/mail/grosjo.net/admin/mailboxes/QoS/dbox-Mails/dovecot.index.search",
> > >  lock_type=0, 
> > > lock_method=FILE_LOCK_METHOD_FCNTL, timeout_secs=, 
> > > lock_r=0x7fff04501118, error_r=0x7fff045010b0) at file-lock.c:318
> > > lock = 
> > > ret = 
> > > #2 0x7f768b62b660 in file_wait_lock (fd=, 
> > > path=, lock_type=lock_type@entry=0, lock_method= > > out>, timeout_secs=timeout_secs@entry=60, 
> > > lock_r=lock_r@entry=0x7fff04501118) at file-lock.c:303
> > > error = 0x564540376490 ""
> > > ret = 
> > > #3 0x7f768a976c87 in squat_trie_lock (trie=0x564540376490, 
> > > lock_type=0, file_lock_r=0x7fff04501118, dotlock_r=0x7fff04501120) at 
> > > squat-trie.c:294
> > > ret = 
> > > dotlock_r = 0x7fff04501120
> > > file_lock_r = 0x7fff04501118
> > > trie = 0x564540376490
> > > ret = 
> > > __func__ = "squat_trie_lock"
> > > lock_type = 0
> > > ret = 
> > > __func__ = "squat_trie_lock"
> > > #4 0x7f768a978627 in squat_trie_map (trie=0x564540376490, 
> > > building=) at squat-trie.c:1487
> > > file_lock = 0x0
> > > dotlock = 0x0
> > > changed = 
> > > ret = 
> > > #5 0x7f768a97b19d in squat_uidlist_map_header 
> > > (uidlist=0x5645403767f0) at squat-uidlist.c:378
> > > No locals.
> > > #6 squat_uidlist_map (uidlist=0x5645403767f0) at squat-uidlist.c:477
> > > mmap_hdr = 
> > > ret = 
> > > #7 0x7f768a97b432 in squat_uidlist_open (uidlist=0x5645403767f0) at 
> > > squat-uidlist.c:546
> > > No locals.
> > > #8 0x7f768a97b5aa in squat_uidlist_refresh (uidlist=) 
> > > at squat-uidlist.c:569
> > > No locals.
> > > #9 0x7f768a9787c2 in squat_trie_map (trie=0x564540376490, 
> > > building=) at squat-trie.c:1526
> > > file_lock = 0x56454210c850
> > > dotlock = 0x0
> > > changed = 
> > > ret = 
> > > #10 0x7f768a97b19d in squat_uidlist_map_header 
>

Re: Event 0x2b1a5f270bd0 leaked (parent=(nil)): auth-client-connection.c:338

[Michael Slusarz]

> On November 23, 2018 at 7:06 AM Mart Pirita  wrote:
>
> FYI, this is not fixed in v2.3.4:

Correct.  This is not a critical issue, so it will be fixed eventually but was 
not a priority for this release.

michael

> Nov 23 16:03:24 xxx dovecot: master: Dovecot v2.3.4 (0ecbaf23d) starting
> up for imap, pop3, lmtp (core dumps disabled)
> Nov 23 16:03:53 xxx dovecot: auth: Warning: Event 0x80c2f28 leaked
> (parent=(nil)): auth-client-connection.c:338
> Nov 23 16:03:53 xxx dovecot: auth: Warning: Event 0x80d71e0 leaked
> (parent=(nil)): auth-client-connection.c:338
> Nov 23 16:03:53 xxx dovecot: auth: Warning: Event 0x80c3220 leaked
> (parent=(nil)): auth-client-connection.c:338
> 
> ...
> 
> 
> Aki Tuomi wrote:
> > It will be fixed.
> >
> > Aki
> >
> > On 6.11.2018 8.57, Mart Pirita wrote:
> >> Hi,
> >>
> >>
> >> I'm not using rsyslog and instead of hiding, this event leak should be
> >> fixed.
> >>
> >>
> >>
> >> Michael Slusarz wrote:
> >>>> On November 3, 2018 at 9:41 AM Mart Pirita  
> >>>> wrote:
> >>>>
> >>>>
> >>>> Hi,
> >>>>
> >>>>
> >>>> But this harmless is spamming logs, so how to disable it:
> >>>>
> >>>> grep auth-client-connection.c:338 maillog | wc -l
> >>>>    1259
> >>> If using something like rsyslog, it is trivial to filter out unwanted 
> >>> entries.
> >>>
> >>> michael
> >>>
> >>>
> >>>> Aki Tuomi wrote:
> >>>>>> On 03 November 2018 at 12:12 Mart Pirita < sysad...@e-positive.ee 
> >>>>>> <mailto:sysad...@e-positive.ee>> wrote:
> >>>>>>
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>
> >>>>>> Noticed with latest v2.3.3 some new warning in logs, for example:
> >>>>>>
> >>>>>> dovecot: auth: Warning: Event 0x80a6fc0 leaked (parent=(nil)):
> >>>>>> auth-client-connection.c:338: 1 Time(s)
> >>>>>> dovecot: auth: Warning: Event 0x80aa1c8 leaked (parent=(nil)):
> >>>>>> auth-client-connection.c:338: 1 Time(s)
> >>>>>> dovecot: auth: Warning: Event 0x80aa718 leaked (parent=(nil)):
> >>>>>> auth-client-connection.c:338: 1 Time(s)
> >>>>>> dovecot: auth: Warning: Event 0x80adac0 leaked (parent=(nil)):
> >>>>>> auth-client-connection.c:338: 1 Time(s)
> >>>>>> dovecot: auth: Warning: Event 0x80b6c38 leaked (parent=(nil)):
> >>>>>> auth-client-connection.c:338: 1 Time(s)
> >>>>>> dovecot: auth: Warning: Event 0x80c0e00 leaked (parent=(nil)):
> >>>>>> auth-client-connection.c:338: 1 Time(s)
> >>>>>> dovecot: auth: Warning: auth client 0 disconnected with 1 pending
> >>>>>> requests: EOF: 12 Time(s)
> >>>>>>
> >>>>>>
> >>>>>> What are they?
> >>>>>>
> >>>>>>
> >>>>>> -- 
> >>>>>> Mart
> >>>>> Hi! It's harmless event leak. This is a known issue to us.
> 
> 
> -- 
> Mart
> 
>

Re: different TLS protocols on different ports

[Michael Slusarz]

> On November 14, 2018 at 12:46 PM "A. Schulze"  wrote:
<
> I stumbled upon RFC 8314 *) and I found it a welcome option to enforce more 
> modern protocols/ciphers.
> IMAPS/SUBMISSIONS aren't used widely (at least to my knowlege, many 
> postmaster used to configure IMAP+SUBMISSION and STARTTLS)

"IMAPS" has been used forever.  Every installation I can think of supports 993.

Same with submission.  465/587 has been a standard port for awhile now.

In fact, these are the only ports someone like a Google will allow you to 
connect to.
https://support.google.com/mail/answer/7126229?hl=en


> Switching Clients to complete new ports is a chance to separate and dry out 
> legacy MUA's

There is no switch to do.  These ports are well-known and well used.


> I just tried this but that's no valid syntax tough:
> 
>   service imap-login {
> inet_listener imap {
>   port = 143
> # using default protocols and ciphers...
> }
> inet_listener imaps {
>   port = 993
>   ssl_protocols = TLSv1.2 TLSv1.3
> ssl_cipher_list = ...
> 
> }
>   }
> 
> 
> Postfix let me easily define different TLS protocols on different ports.
> For that it would be cool if dovecot could assist on such migrations, too.
> 
> Andreas
> 
> *) see https://tools.ietf.org/html/rfc8314
>as well as the draft 
> https://tools.ietf.org/html/draft-lvelvindron-tls-for-email-02 to deprecate 
> TLSv1.1

Re: Event 0x2b1a5f270bd0 leaked (parent=(nil)): auth-client-connection.c:338

[Michael Slusarz]

> On November 3, 2018 at 9:41 AM Mart Pirita  wrote:
> 
> 
> Hi,
> 
> 
> But this harmless is spamming logs, so how to disable it:
> 
> grep auth-client-connection.c:338 maillog | wc -l
>    1259

If using something like rsyslog, it is trivial to filter out unwanted entries.

michael


> Aki Tuomi wrote:
> > > On 03 November 2018 at 12:12 Mart Pirita < sysad...@e-positive.ee 
> > > > wrote:
> > >
> > >
> > > Hi,
> > >
> > >
> > > Noticed with latest v2.3.3 some new warning in logs, for example:
> > >
> > > dovecot: auth: Warning: Event 0x80a6fc0 leaked (parent=(nil)):
> > > auth-client-connection.c:338: 1 Time(s)
> > > dovecot: auth: Warning: Event 0x80aa1c8 leaked (parent=(nil)):
> > > auth-client-connection.c:338: 1 Time(s)
> > > dovecot: auth: Warning: Event 0x80aa718 leaked (parent=(nil)):
> > > auth-client-connection.c:338: 1 Time(s)
> > > dovecot: auth: Warning: Event 0x80adac0 leaked (parent=(nil)):
> > > auth-client-connection.c:338: 1 Time(s)
> > > dovecot: auth: Warning: Event 0x80b6c38 leaked (parent=(nil)):
> > > auth-client-connection.c:338: 1 Time(s)
> > > dovecot: auth: Warning: Event 0x80c0e00 leaked (parent=(nil)):
> > > auth-client-connection.c:338: 1 Time(s)
> > > dovecot: auth: Warning: auth client 0 disconnected with 1 pending
> > > requests: EOF: 12 Time(s)
> > >
> > >
> > > What are they?
> > >
> > >
> > > -- 
> > > Mart
> >
> > Hi! It's harmless event leak. This is a known issue to us.

Re: split auth from other logging

[Michael Slusarz]

For later versions of 2.3.x, it will eventually be possible to use log 
filtering to accomplish this entirely within Dovecot.  We are not quite there 
yet though.

michael


> On September 26, 2018 at 1:21 AM Kai Schaetzl  wrote:
> 
> 
> Is there a way to split the auth logging (logins and failed logins) from 
> the other logging that goes to 
> info_log_path = /var/log/dovecot/dovecot.log
> ?
> This log gets a lot of other info as well, most notably the lmtp 
> notifications about every filed mail (with no level stamping, btw).
> This makes it really hard to find authentication errors quickly and 
> comfortably.
> It would be nice to be able to split at least the lmtp messages away.
> 
> Kai
> 
> 
>

Re: Best way to move mail from one server to another

[Michael Slusarz]

Sami is correct.  imapsync loses data.

Source -> imapsync -> destination : the mailboxes are different to an IMAP 
client
Source -> dsync -> destination (running Dovecot w/doveadm): the mailboxes are 
the same

You may not care that your users w/100,000 message mailboxes that have been 
cached on their client now have to invalidate that entire cache (and most 
likely re-download all of those messages).

So there is zero argument that, due to IMAP metadata loss, this is something 
that doesn't happen if dsync (or rsync) is used.

michael


> On September 4, 2018 at 9:45 AM Rick Romero  wrote:
> 
> 
> Quoting Robert Schetterer mailto:r...@sys4.de >:
> 
> > > 
> > Am 04.09.2018 um 17:18 schrieb Sami Ketola:
> > 
> > > > > 
> > > > > > > 
> > > > On 4 Sep 2018, at 18.00, Robert Schetterer 
> > > > mailto:r...@sys4.de > wrote:
> > > > 
> > > > Am 04.09.2018 um 16:52 schrieb Sami Ketola:
> > > > 
> > > > > > > > > 
> > > > > > > > > > > 
> > > > > > On 4 Sep 2018, at 17.47, Robert Schetterer 
> > > > > > mailto:r...@sys4.de > wrote:
> > > > > > 
> > > > > > Am 04.09.2018 um 09:41 schrieb Sami Ketola:
> > > > > > 
> > > > > > > > > > > > > 
> > > > > > > imapsync always loses data
> > > > > > > 
> > > > > > > > > > > > > never saw this, be carefull 
> > > > > > > by anounce such myths
> > > > > > 
> > > > > > > > > > > It is a fact. Imapsync works over 
> > > > > > IMAP only and IMAP protocol does not even support transferring all 
> > > > > > data. At most at least UID numbering will be lost and end users 
> > > > > > need to invalidate their local caches.
> > > > > 
> > > > > > > > > but using "looses data" might others think it 
> > > > > also may fail with imap,
> > > > so be more detailed next time
> > > > 
> > > > > > > UID number is also data that is saved on the IMAP 
> > > > backend. If that is lost then it's "lost data".
> > > 
> > > Sami
> > > 
> > > > > Sorry i migrated terrabytes of mail with imapsync and never 
> > > had a
> > problem, it works as designed, also with maildir rsync did a good 
> > job,
> > what never worked as it should was dsync ,cause of bugs ,that may
> > changed now
> > 
> > so this is my answer to topic
> > 
> > "Best way to move mail from one server to another"
> >  
> > 
> > > 
> https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt
> 
> Seems to use UIDs so that 'data' isn't lost.
> 
> 

Re: Doveadm protocol; dovecot v2.2.10

[Michael Slusarz]

> On July 30, 2018 at 2:07 PM David White  wrote:
>
> I’m currently on v2.2.10 and it appears the doveadm protocol command set is 
> limited to just the “mailbox” commands and the http protocol hasn’t been 
> implemented.
> 
> Is the doveadm http protocol still experimental in v2.3.2?

No.

michael

Re: 100% Dovecot MTA Replacement Setup

[Michael Slusarz]

> On July 19, 2018 at 9:41 AM David Favor  wrote:
> 
> 
> Working out upgrade of old 1.x + 2.x Dovecot installs to
> latest 2.3.2.1 out of Ubuntu Bionic repositories.
> ___
> 
> https://wiki.dovecot.org/Services suggests Dovecot can
> be used for full mail infrastructure, avoiding the
> complexity of exim4 or other MTA setup/management.
> 
> Let me know if I understand this correctly.
> 
> 1) https://wiki.dovecot.org/Submission can be used to
> listen on port 25 + port 587 (with auth).

This is a proxy only.  For any non-local delivery, you still need a submission 
server to deliver (and queue) mail remotely.  That requires a full SMTP service 
running somewhere.


> 2) https://wiki.dovecot.org/Pigeonhole can be used for
> filtering, including forwarding email to other machines.
> 
> 3) https://wiki.dovecot.org/Pigeonhole/ManageSieve/Configuration
> can be used to forward off machine email, for example flowing
> through a relay service like MailGun, using a simple script
> or service like ESMTPD.
> 
> 4) https://wiki.dovecot.org/LMTP can be used to deposit messages
> into filesystem as Maildir or dbox or mdbox backing stores.

michael

Re: Storing Messages in the cloud

[Michael Slusarz]

> On July 10, 2018 at 12:37 PM "M. Balridge"  wrote:

[snip]

> I also imagine the hard-working crew at Timo, Inc. have such things for sale
> to facilitate this with security and good performance, but they won't be
> inexpensive.
> 
> See: http://www.dovecot.fi/products/

More up-to-date link is:
https://www.open-xchange.com/portfolio/dovecot-pro/

It's fairly easy to write support for storing message blobs in object storage.  
In fact, we are working on open sourcing the generic Dovecot object storage 
interface in the near future as part of core that would allow this kind of 
message storage.

But:

1) if you have to store indexes in a separate block storage system, that kind 
of defeats the purpose of having a single storage layer and, alternately
2) using object storage in a block-storage like mode for indexes does not scale 
at all - block storage access patterns do not work well at object storage 
latencies and I/O capacity.

There are several other components on top of generic object storage access that 
are used by Dovecot Pro to address these concerns.  This allows Dovecot Pro's 
object storage support to store all data (messages, indexes, FTS) in a single 
object storage bucket while remaining performant and scalable.

These additional components are expensive to develop and maintain, and mainly 
useful for very large installations, so they will remain part of the commercial 
Dovecot Pro package when the basic object storage support is released.

michael

Re: IMAP_ID_LOG

[Michael Slusarz]

ID is an IMAP specific protocol command (RFC 2971).  There is no POP or SMTP 
equivalent.


> On April 9, 2018 at 10:19 AM "j.emerlik"  wrote:
> 
> Hi,
> there in configuration is possibile to set imap_id_log for logging Client 
> ID e.g.:  ID sent: name=Thunderbird, version=52.7.0.
> 
> Is possible to set same but for POP3 or SMTP Auth ?
> 
> Ideally, it would be possible to log in to the database e.g. using 
> postlogin script.
> 
> Regards,
> MattX
> 
> 
> 

Re: `mail_crypt` Doesn't Appear to be Working

[Michael Slusarz]

> On February 16, 2018 at 7:31 PM "jorda...@startmail.com 
> mailto:jorda...@startmail.com " wrote:
> 
> 
> Dovecot version: 2.2.22 (fe789d2)
> 

>From the mail_crypt page:

“This feature is available in v2.2.27+”


> I generated an EC key from the page https://wiki2.dovecot.org/Plugins/. 
> For
> reference here's my /etc/dovecot/conf.d/10-mail-crypt.conf file:
> 
> 
> mail_plugins = $mail_plugins mail_crypt
> 
> plugin {
> # mail_crypt_global_private_key =  mail_crypt_global_public_key =  mail_crypt_save_version = 2
> }
> 
> 
> I saw in a previous message on this mailing list that messages can be
> encrypted without the private key, so it's stored elsewhere. :-)
> 
> After restarting dovecot and sending myself a message I found that the 
> message
> was still unencrypted. Nothing in /var/log/dovecot/*.log files or syslog
> indicated any problem loading the plugin (I even made some mistakes with 
> the
> filename beforehand, and errors indicated it was trying to find the 
> plugin).
> 
> In case in matters dovecot was installed as part of the automated iredmail
> install (https://iredmail.org). I didn't modify the dovecot.conf file 
> after
> installation (except for thinking I needed to add `mail_crypt` to 
> `plugins`,
> then figuring I didn't).
> 
> Any help is appreciated.
> 
> Thank you.
> 

Re: Recommended tool for migrating IMAP servers

[Michael Slusarz]

> On December 4, 2017 at 8:46 AM Paolo  wrote:
> 
> 
> Il 04/12/2017 14:33, x9p ha scritto:
> >
> >> Can I use this tool even if I do not know the other remote server
> >> typology?
> >>
> > sure. just need both IMAP ports reachable and valid user/pass for both
> > servers.
> I think Davide was asking about dsync. If so, the answer is no: dsync 
> works only when both servers are Dovecot and needs some additional 
> configuration to work through the network (see 
> https://wiki2.dovecot.org/Replication).

This is entirely incorrect.  The source platform for dsync can be ANY IMAP/POP 
server.

The recommended tool for migrating into Dovecot is dsync.  You don't need any 
other tool, and other tools aren't going to preserve state so they are pretty 
much worthless for a real-world in-place migration.

michael

Re: Plugin virtual, Horde BAD IMAP QRESYNC not enabled

[Michael Slusarz]

Probably better to get the full protocol debug log from the Horde/IMP side.  
Instructions on how to enable that can be found in imp/config/backends.php. See:
https://github.com/horde/imp/blob/master/config/backends.php#L288

rawlog output below only shows what Dovecot is returning - would be more useful 
to see the entire client round-trip transactions however.


> On November 14, 2017 at 4:15 PM Jakob Schürz  
> wrote:
> 
> 
> Hi there!
> 
> I'm new on this list, so hello to all!
> First i want say, that dovecot is a really great mailserver. I use it
> since many years for my private usecase. I'm planning to setup a new
> Mailserver, so i'm fiddeling around to understand dovecot better and get
> a "perfect" Mailserver. (Linux and Win10) Outlook, k9mail and the
> gmail-app on android.
> 
> I want to leave google behind me, so i'm searching for a good server for
> cloudbased  and selfhosted calendar- and addressbook-server.
> On some discussion in debianforum.de i got the hint, to use horde.
> 
> So i installed horde from pear. Got ActiveSync working with the Android
> and Outlook. Mails, Calendar and Addressbook... very fine.
> 
> But then i run into a gotcha.
> 
> There is a problem  with some mails in dovecots virtual mailboxes.
> I tried a lot to find out, how to debug this. First i thought, it's a
> problem from horde, because the problem is only with horde-client.
> 
> Some Emails are not shown in horde. Exactly, they are not shown in horde
> on dovecots virtual mailboxes (plugin virtual in a separate virtual
> namespace)
> If i try to open this email in its original directory, it gets opened
> without any problem.
> 
> So, when i try to open it from within the virtual mailbox, i get this error:
> ---8<--
> Nov 14 23:29:52 aldebaran dovecot[2495]: imap(jakob): Panic: file
> index-mail-binary.c: line 586 (index_mail_get_binary_stream): assertion
> failed: (mail->data.stream != NULL)
> Nov 14 23:29:52 aldebaran dovecot[2495]: imap(jakob): Error: Raw
> backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95272) [0x7f671137e272]
> -> /usr/lib/dovecot/libdovecot.so.0(+0x9536d) [0x7f671137e36d] ->
> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f6711314951] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_binary_stream+0x238)
> [0x7f67116b3de8] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_binary_stream+0x60)
> [0x7f6711640fa0] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x139)
> [0x7f67116f40f9] -> dovecot/imap [jakob 192.168.0.14 UID
> FETCH](+0x1ebbc) [0x55bef3095bbc] -> dovecot/imap [jakob 192.168.0.14
> UID FETCH](+0x1cfb6) [0x55bef3093fb6] -> dovecot/imap [jakob
> 192.168.0.14 UID FETCH](imap_fetch_more+0x39) [0x55bef30950e9] ->
> dovecot/imap [jakob 192.168.0.14 UID FETCH](cmd_fetch+0x33b)
> [0x55bef3086d9b] -> dovecot/imap [jakob 192.168.0.14 UID
> FETCH](command_exec+0xa5) [0x55bef3092735] -> dovecot/imap [jakob
> 192.168.0.14 UID FETCH](+0x199c2) [0x55bef30909c2] -> dovecot/imap
> [jakob 192.168.0.14 UID FETCH](+0x19a4c) [0x55bef3090a4c] ->
> dovecot/imap [jakob 192.168.0.14 UID FETCH](client_handle_input+0x1b5)
> [0x55bef3090e55] -> dovecot/imap [jakob 192.168.0.14 UID
> FETCH](client_input+0x86) [0x55bef30913c6] ->
> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f6711392dd2]
> -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109)
> [0x7f6711394409] ->
> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
> [0x7f6711392e6c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
> [0x7f6711393018] ->
> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13)
> [0x7f671131ae93] -> dovecot/imap [jakob 192.168.0.14 UID
> FETCH](main+0x328) [0x55bef3083e68] ->
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f6710f6a2b1]
> -> dovecot/imap [jakob 192.168.0.14 UID FETCH](_start+0x2a) [0x55bef3083fea]
> Nov 14 23:29:52 aldebaran dovecot[2495]: imap(jakob): Fatal: master:
> service(imap): child 2704 killed with signal 6 (core dumps disabled)
> ---8<--
> 
> So i activated rawlog as after-login executeable. The IMAP-dialog when
> opening this email is:
> ---8<--
> 9 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
> IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS
> THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN
> NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH
> ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE XDOVECOT
> SPECIAL-USE QUOTA ACL RIGHTS=texk] Logged in
> 10 BAD Error in IMAP command SELECT: QRESYNC not enabled (0.000 + 0.000
> secs).
> 11 OK No quota (0.000 + 0.000 secs).
> ---8<--
> 
> If found only one Report for this problem, but on regular imap-folders,
> and there was not really a solution. This report is from 2015.
> On regular folders, i have no problem with this emails, only on virtual
> folders.
> So i think, it is a problem of the virtual-plugin.
> 
> Here is one of this failing emails:
> 
> ---8<-

Re: IMAP From sort uses only local-part

[Michael Slusarz]

> On August 7, 2017 at 3:09 PM Paul wrote:
> 
> As far as I can tell, when I ask Dovecot to sort an IMAP mailbox by the 
> From address, it ignores the part of the address after the '@'. So, for 
> instance, mails from i...@abc.org, i...@xyz.org, and i...@ghi.org are 
> all mingled together in arrival order.
> 
> Is there a way to change this? Thanks!

This is the RFC 5256 compliant behavior.  From is sorted on addr-mailbox.  If 
those are equal:

  If two or more messages exactly match according to the sorting
  criteria, these messages are sorted according to the order in
  which they appear in the mailbox.  In other words, there is an
  implicit sort criterion of "sequence number".

RFC 5256, Section 3.

michael

Re: Conditionally Filter Mails During IMAP Fetch

[Michael Slusarz]

> On May 16, 2017 at 1:05 AM Ory Drilon  wrote:
> 
> 
> Hi, is there a way to arbitrarily filter out mails at the moment the
> IMAP request is made, based on properties of the IMAP session and the
> mail itself?
> 
> All the filtering methods I find occur during delivery time.

imapsieve: https://tools.ietf.org/html/rfc6785

Although obviously triggering filtering based on a FETCH itself doesn't make 
much sense and isn't supported by imapsieve (triggering based on flag change 
*caused* by FETCH is supported however).

michael

Re: javamail imap listing

[Michael Slusarz]

"A non-empty reference name argument is the name of a mailbox or a level of 
mailbox hierarchy, and indicates the context in which the mailbox name is 
interpreted."

If Dovecot is using "." as the separator in the base level of hierarchy, then 
it correctly is returning nothing for 'LIST / "*"' - since no mailboxes in 
Dovecot live under that base level of hierarchy.

Gmail returns data because it is using "/" as a separator.

If you want Dovecot to return LIST entries for 'LIST "/" "*"', then configure 
the base namespace to use "/" as a separator.  Of course then 'LIST "." "*"' 
would return nothing...

michael

> On April 18, 2017 at 2:38 AM Matthew Broadhead 
>  wrote:
> 
> 
> Thanks Michael.  I have forwarded that link to the Geronimo JavaMail 
> team in case they think of anything their end.
> 
> But in the documentation you sent it seems that Dovecot should respond 
> in the case of
> LIST / "*"
> if my understanding is correct
> 
> Is there any way to adjust this behaviour in the settings?  I am 
> struggling to search for a solution to the problem given the limited 
> information.
> 
> On 18/04/2017 05:46, Michael Slusarz wrote:
> > You probably want to read the description of "reference name argument" to 
> > understand what is happening here.
> >
> > https://tools.ietf.org/html/rfc3501#section-6.3.8
> >
> > michael
> >
> >> On April 17, 2017 at 8:33 AM Matthew Broadhead 
> >>  wrote:
> >>
> >> Hi,
> >>
> >> i am using dovecot-2.2.10-7.el7.x86_64 on
> >> centos-release-7-3.1611.el7.centos.x86_64.
> >>
> >> if i follow this tutorial
> >> https://delog.wordpress.com/2011/05/10/access-imap-server-from-the-command-line-using-openssl/
> >> i can login to my server and successfully list the folders using
> >> tag LIST "" "*"
> >>
> >> *   LIST (\HasNoChildren) "." INBOX
> >> tag OK List completed.
> >>
> >> no folders are listed using
> >> tag LIST / "*"
> >> tag OK List completed.
> >>
> >> when i request a folder listing using JavaMail it similarly sends the
> >> following command and no folders are listed
> >> a97 LIST / "*"
> >> a97 OK List completed.
> >>
> >> if i follow the tutorial again and use my gmail account instead of my
> >> dovecot installation i can successfully list folders using both methods
> >> tag LIST / "*"
> >>
> >> *   LIST (\HasNoChildren) "/" "youtube"
> >> tag OK List completed.
> >>
> >> is there some way to make dovecot list folders when it receives the command
> >> LIST / "*"
> >>
> >> i note that in the listings my dovecot installation has "." in the
> >> middle and gmail has a "/"

Re: javamail imap listing

[Michael Slusarz]

You probably want to read the description of "reference name argument" to 
understand what is happening here.

https://tools.ietf.org/html/rfc3501#section-6.3.8

michael

> On April 17, 2017 at 8:33 AM Matthew Broadhead 
>  wrote:
> 
> Hi,
> 
> i am using dovecot-2.2.10-7.el7.x86_64 on
> centos-release-7-3.1611.el7.centos.x86_64.
> 
> if i follow this tutorial
> https://delog.wordpress.com/2011/05/10/access-imap-server-from-the-command-line-using-openssl/
> i can login to my server and successfully list the folders using
> tag LIST "" "*"
> 
> *   LIST (\HasNoChildren) "." INBOX
> tag OK List completed.
> 
> no folders are listed using
> tag LIST / "*"
> tag OK List completed.
> 
> when i request a folder listing using JavaMail it similarly sends the
> following command and no folders are listed
> a97 LIST / "*"
> a97 OK List completed.
> 
> if i follow the tutorial again and use my gmail account instead of my
> dovecot installation i can successfully list folders using both methods
> tag LIST / "*"
> 
> *   LIST (\HasNoChildren) "/" "youtube"
> tag OK List completed.
> 
> is there some way to make dovecot list folders when it receives the command
> LIST / "*"
> 
> i note that in the listings my dovecot installation has "." in the
> middle and gmail has a "/"

Re: How to migration my mails from another server ?

[Michael Slusarz]

impasync doesn't preserve IMAP state if Dovecot is the destination server.  You 
should use dsync instead.


> 
> On April 8, 2017 at 3:11 AM Adrian Minta  wrote:
> 
> Use imapsync: https://imapsync.lamiral.info/
> 
> On 04/08/2017 12:02 PM, Mik J wrote:
> 
> > > 
> > Hello,
> > 
> > I would like to host my mails and I would like to retrieve my 
> > existing mailbox content from my ISP
> > 
> > My ISP
> > ==
> > I don't have root access on the server
> > The server seem to use Dovecot according to the banner.
> > 
> > My machine
> > ==
> > I have root access on the server
> > I use Dovecot 2.2.21
> > I use Maildir
> > 
> > I read this page multiple times but it's not clear to me
> > https://wiki2.dovecot.org/Migration/Dsync
> > 
> > "Set up configuration for the IMAP server you wish to migrate from"
> > Is it my ISP's server or my server ?
> > In which configuration file should I do this ?
> > 
> > The same for this option
> > dsync_features = empty-header-workaroundwhere is it supposed to be 
> > configured ?
> > 
> > My most important case is quite "simple": the mailboxes (there are 
> > 2 or 3) I want to migrate don't exist on my machine, so there is no such 
> > problems as merge existing mails.
> > Will this migration use imap and retrieve the content of the mails 
> > ? Will it use port 143 or 993 ?
> > I'll use the backup option
> > 
> >     I have another case which there are mails on both servers. So I 
> > would like to keep existing mails on my machine.
> > I'll use the sync -1 option ?
> > 
> > Thank you
> > 
> > > 
> --
> Best regards,
> Adrian Minta
> 


--

kind regards,

Michael Slusarz
Senior Sales Engineer


Phone:  +1 (303) 888-1090
Skype:  michael.m.slusarz
Email:  michael.slus...@open-xchange.com mailto:michael.slus...@open-xchange.com
 
Twitter: @openexchange http://twitter.com/openexchange - Facebook: OpenXchange 
https://www.facebook.com/OpenXchange - Web: www.open-xchange.com 
http://www.open-xchange.com
Open-Xchange AG, Rollnerstr. 14, 90408 Nuremberg, District Court Nuremberg HRB 
24738
Managing Board: Rafael Laguna de la Vera, Carsten Dirks, Uwe Reumuth
Chairman of the Board: Richard Seibt

European Office:
Open-Xchange GmbH, Olper Huette 5f, D-57462 Olpe, Germany, District Court 
Siegen, HRB 8718
Managing Directors: Frank Hoberg, Martin Kauss

US Office:
Open-Xchange. Inc., 530 Lytton Avenue, Palo Alto, CA 94301, USA
 
Confidentiality Warning: This message and any attachments are intended only for 
the use of the intended recipient(s), are confidential, and may be privileged. 
If you are not the intended recipient, you are hereby notified that any review, 
retransmission, conversion to hard copy, copying, circulation or other use of 
this message and any attachments is strictly prohibited. If you are not the 
intended recipient, please notify the sender immediately by return e-mail, and 
delete this message and any attachments from your system.

Re: Problem with Dovecot and BlackBerry

[Michael Slusarz]

> On April 4, 2017 at 5:07 AM Luca Bertoncello  wrote:
> 
> Hi all,
> 
> i've got a strange behaviour with a BlackBerry Classic Phone (BBOS
> 10.3.2.2876) in combination with Dovecot 2.2.13 while trying to fetch
> mails.
> 
> Before burying myself into debugging sessions, i try to get an
> understanding if the following is a Client- or a Server-specific error
> in the behaviour:
> 
> CIGA8 UID FETCH 10009:10035 (UID FLAGS) (CHANGEDSINCE NOMODSEQ)
> CIGA8 BAD Error in IMAP command UID FETCH: Invalid CHANGEDSINCE modseq.

That's a broken client.

https://tools.ietf.org/html/rfc7162#section-3.1.4.1

"CHANGEDSINCE :  The CHANGEDSINCE FETCH modifier allows
   the client to further subset the list of messages described by the
   sequence set.  The information described by message data items is
   only returned for messages that have a mod-sequence bigger than
   ."

michael

Re: Replacement for antispam plugin

[Michael Slusarz]

> On February 10, 2017 at 12:13 PM Ralph Seichter wrote:
> 
> On 10.02.17 18:34, Michael Slusarz wrote:
> > > Can we add an exception for the Trash folder?
> > This is handled in the sieve script. E.g.:
> > 
> > require "environment";
> > if environment "imap.mailbox" "Trash" {
> >  stop;
> > }
> 
> This does not work for me, and I don't really expect it to work either.
> https://tools.ietf.org/html/rfc6785#section-4.4 states:
> 
>  The implementation MUST set the Environment [RFC5183] item "imap.mailbox"
>  to the name of the mailbox that the affected message is in, in the
>  case of existing messages, or is targeted to be stored into, in the
>  case of new messages.
> 
> The message already exists in the Spam folder, hence imap.mailbox should
> be "Spam" instead of "Trash", correct?

Incorrect.

When you move a message to a new mailbox, that is a "new message" event (a new 
UID in the target mailbox is created; the message count increases).  So 
imap.mailbox is set to the name of the *target* mailbox.

> Is there perhaps another way to ensure that manually deleted spam is not
> erroneously learned as ham?
> 
> -Ralph

Re: Replacement for antispam plugin

[Michael Slusarz]

> On February 10, 2017 at 9:25 AM George Kontostanos  
> wrote:

[snip]

> I think that this needs some change:
> 
>  # From Spam folder to elsewhere
>   imapsieve_mailbox2_name = *
>   imapsieve_mailbox2_from = Spam
>   imapsieve_mailbox2_causes = COPY
>   imapsieve_mailbox2_before = file:/usr/lib/dovecot/sieve/report-ham.sieve
> 
> When a message from Spam is moved to Trash then the report-ham.sieve is
> being executed.
> 
> Can we add an exception for the Trash folder?

This is handled in the sieve script.  E.g.:

require "environment";
if environment "imap.mailbox" "Trash" {
stop;
}

michael

Re: Simple way to get mailbox size by folder.

[Michael Slusarz]

On 12/3/2016 2:53 PM, Aki Tuomi wrote:



On December 3, 2016 at 11:16 PM Vijay Sarvepalli  wrote:


Is there a simple way to get each mailbox size using dovecot's IMAP
interface?  The GETQUOTAROOT and GETQUOTA seems to work with the full
maildir size and not the individual folders.

I am finding a simple way for the webmail client to show usage of each
folder.

Thanks
Vijay


$ doveadm mailbox status -u cmouse all INBOX
INBOX messages=9206 recent=0 uidnext=33115 uidvalidity=1451655531 unseen=7 
highestmodseq=39954  vsize=538582181 guid=d595a62d65818656f72c74be03de

$ doveadm -fjson mailbox status -u cmo...@cmouse.fi all INBOX
[{"mailbox":"INBOX","messages":"9206","recent":"0","uidnext":"33115","uidvalidity":"1451655531","unseen":"7","highestmodseq":"39954","vsize":"538582181","guid":"d595a62d65818656f72c74be03de"}]


If you want an IMAP specific method (rather than doveadm), you are 
stuck with "FETCH 1:* (RFC822.SIZE)".  With the nasty requirement - if 
you want 100% full accuracy - that you need to track responses to ensure 
that each size is only counted once.  Practically speaking this 
shouldn't happen by any reasonable server, but the warning needs to be made.


michael

Re: "anvil: Error: connect limit: disconnection for unknown pid 17659 + ident lmtp/backup@backup.invalid"

[Michael Slusarz]

> On July 7, 2016 at 7:45 AM Frank Elsner  wrote:
> 
> On Thu, 7 Jul 2016 14:31:04 +0200 Ralf Hildebrandt wrote:
> 
> > I updated to 2.3.0 today, and now I'm getting these entries in my log:
> >  ^
> 
> Hey, what's that? Typo or secret version?

Development (git master) version.

michael

Re: FTS search used / useful on an IMAP proxy?

[Michael Slusarz]

> 
> On June 28, 2016 at 7:07 AM Luca Lesinigo  wrote:
> 
> We are preparing an IMAP proxy based on dovecot-2.2.22, basic proxy 
> functionality is already working and I’m trying to understand if having the 
> FTS service configured on the dovecot *proxy* would be of any use.
> 
> I do suspect it would be useless, I guess dovecot in imap proxy mode just 
> forwards any command to the backend and does not bother to do anything about 
> it, but I’m failing to find a definitive answer in the documentation. If I am 
> guessing correctly, an fts service would only be useful if configured and 
> working on the actual backend.
> 

FTS only makes sense on backend, where the search would be executed.

michael

Re: password expire warning for dovecot users in IMAP/POP login

[Michael Slusarz]

The correct way to handle this IMAP-wise would be to return the EXPIRED 
response code (https://tools.ietf.org/html/rfc5530#section-3).  But this 
requires client support to report to the end user.  (And also requires that 
Dovecot would be able to determine from authentication source that the 
credentials are expired, as opposed to incorrect.)

michael

> On June 8, 2016 at 2:51 AM "A.L.E.C"  wrote:
> 
> On 06/08/2016 10:39 AM, mkaw...@redhat.com wrote:
> 
> > To make it happen, no need to add any other configurations on LDAP end
> > once possword policy is correctly set?
> 
> You've got me wrong. I just responded to Aki's question. ALERT feature
> could be used to send the message to the client, but there's no code to
> handle such LDAP password policies/notices yet.
> 
> --
> Aleksander 'A.L.E.C' Machniak
> Kolab Groupware Developer [http://kolab.org]
> 
> Roundcube Webmail Developer [http://roundcube.net]
> 
> ---
> PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl

Re: New feature: HTTP API

[Michael Slusarz]

> 
> On May 24, 2016 at 12:20 PM Peer Heinlein 
>  wrote:
> 
> Thanks for the new HTTP API.
> 
> I get the API with some commands up and running, but I'm still not able
> to create folder.
> 
> curl -k -H "Content-Type: application/json" -H "Authorization: Basic
> secret" -d
> 
> '[["mailboxCreate",{"user":"u...@example.org","mailbox":["INBOX/TEST"]},c01]]'
> https://xxx.xxx.xxx.xxx.:8080/doveadm/v1
> 

The identifier is a JSON string (c01) and needs to be in quotes, at a minimum.

> 
> and also
> 
> curl -k -H "Content-Type: application/json" -H "Authorization: Basic
> secret" -d
> 
> '[["mailboxList",{"user":"u...@example.org","mailboxMask":["INBOX/TEST"]},c01]]'
> https://xxx.xxx.xxx.xxx.:8080/doveadm/v1
> 
> doesn't work.
> 
> It /should/ create INBOX/TEST -- but doveadm always says it's "Invalid
> JSON input".
> 
> What am I doing wrong?
> 

michael

Re: Kernel panic in dovecot-ee-lmtp on Debian 8

[Michael Slusarz]

Fixed -- Publicly accessible link to commit:

https://github.com/dovecot/pigeonhole/commit/a95b0579b89c13fb3ee5700e76cbe6a4a3e898e0


> On April 25, 2016 at 10:46 AM Stephan Bosch  wrote:
> 
> 
> 
> 
> Op 25-4-2016 om 18:01 schreef Timo Sirainen:
> > On 25 Apr 2016, at 13:48, Tobi  wrote:
> >> Hi list
> >>
> >> I just realized that that I don not receive all mails in my mailbox
> >> (running dovecot-ee 2.2.23.1-1 on Debian 8). On my frontend servers
> >> (running postfix) the queue fills up with mails that cannot be delivered
> >> via lmtp to my backend servers. The error message on the frontend is
> >> "lost connection with backend while sending data"
> >> When I checked the logs on the backend server I found upon every
> >> delivery attempt kernel panics
> >>
> >> Apr 25 12:33:36 mbox1 dovecot: lmtp(REDACTED): Panic: epoll_ctl(del, 18)
> >> failed: Bad file descriptor
> >> Apr 25 12:33:36 mbox1 dovecot: lmtp(REDACTED): Error: Raw backtrace:
> > Can you get a core dump and gdb backtrace? See 
> > http://dovecot.org/bugreport.html#coredumps
> >
> >> /usr/lib/dovecot/libdovecot-sieve.so.0(program_client_run+0xe8)
> >> [0x7faa17d3b308] ->
> >> /usr/lib/dovecot/modules/sieve/lib90_sieve_extprograms_plugin.so(+0x45b8) 
> >> [0x7faa15d855b8]
> >> ->
> >> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_continue+0x7c)
> >> [0x7faa17cfa2ac] ->
> >>
> >> the weird thing is that this is not concerning all incoming mails. From
> >> time to time the frontends can deliver messages to my backends. I just
> >> tested it with a mail from work. It arrived via a frontend that has
> >> other mails in queue that could not be delivered so far.
> >>
> >> Does anyone have an idea where I could look for the root cause of this
> >> panic messages?
> > Looks like somehow caused by sieve extprograms.
> 
> Probably this bug:
> 
> https://git.dovecot.net/pigeonhole/core/commit/a95b0579b89c13fb3ee5700e76cbe6a4a3e898e0
> 
> Fixed for upcoming release.
> 
> Regards,
> 
> Stephan.

Re: Content-Enconding

[Michael Slusarz]

On 12/4/2015 8:04 AM, Leander Schäfer wrote:

Thank you for your quick feedback. I added it to my plugins in the
config. How can I make sure my mailclients are using it? Is there a way
to check this like I can check headers with additional Firefox plug-in
you may recomment?


I will be used automatically if the client supports it (and Dovecot is 
configured to enable compression).


You'd have to look at raw protocol data to determine whether it is being 
used or not. Like using a packet sniffer.


michael

Re: Interpreting keywords

[Michael Slusarz]

On 12/3/2015 12:30 AM, Steffen Kaiser wrote:


On Wed, 2 Dec 2015, Mark Foley wrote:


I've marked several messages in Thunderbird using tags. Tags used are:

0 Important




The messages so tagged appear to have the flag fields set in the IMAP
Maildir:

cur/1449002162.8993_0.mail:2,Sb
cur/1449001929.28087_0.mail:2,Sad

I've looked in dovecot-keywords and find:

$ more dovecot-keywords
0 $label1

I assume these "$label" values are macros that possibly refer to
"Important", "Work", etc., but
where are these $label's defined? Are they defined in the dovecot
configs somewhere or does the
mail client just "know" what these correspond to?


The latter, see
http://superuser.com/questions/983401/import-export-or-retrieve-thunderbird-tags-from-imap-server


IMAP flags are annoying since they are limited to a subset of ASCII 
(e.g. no spaces), thus the need to do flag mappings on the client side.


michael

Re: Content-Enconding

[Michael Slusarz]

On 12/4/2015 6:11 AM, Leander Schäfer wrote:


With Apache one may use "mod_deflate" in order to reduce bandwidth by
using e.g. gzip to compress the http traffic. I would like to use
something similar for email traffic between mail clients to dovecote and
postfix. My questions are:


https://tools.ietf.org/html/rfc4978

michael

Re: ESEARCH multiple folders

[Michael Slusarz]

On 7/7/2015 12:20 AM, Peter Chiochetti wrote:

FTS-Solr is blazingly fast; yet searching e.g. an archive with
subfolders still is slow; I learned, that this is a feature/bug of the
imap protocol. So I'd like to ask:

Does or will dovecot support the "IMAP4 Multimailbox SEARCH Extension"
specified in RFC 7377?


The speed (or lack thereof) has very little to do with the IMAP protocol 
limitations. RFC 7377 is not going to appreciably speed this process up; 
you'll probably get almost identical results pipelining all the 
subfolder searches.


This sounds more like an indexing issue.

michael