Re: Dovecot 2.3.6 on Solaris10: build issues, segfaults

[Joseph Tam via dovecot]

On Wed, 10 Jul 2019, Timo Sirainen wrote:


This is unexpected.  But I don't see how it's a Dovecot bug.  It seems
as if your compiler doesn't do padding correctly and then crashes
because it didn't do it correctly.  I guess you're compiling this as
32bit?


Yes, 32-bit.


Is size_t 32bit or 64bit?
...
Can you try with the below small test program if it prints the same 20?
...


(Output without compile flags -m64)
offset = 20
size = 4

(Output with compile flags -m64)
offset = 40
size = 8

Same output whether I use gcc 9.1.0 or Solaris c99.

I'll try building 64bit executables but I think that means I'll have to
build 64bit versions of third-party libraries.

Joseph Tam 

Re: Dovecot behind Load Balancer

[Tanstaafl via dovecot]

On 7/10/2019, 4:27:32 AM, Sami Ketola via dovecot 
wrote:
> There is a limit on thunderbird config that controls this.

And the default is 5 if memory serves...

Re: solr

[Shawn Heisey via dovecot]

On 7/10/2019 2:49 AM, Maciej Milaszewski IQ PL via dovecot wrote:

On the other hand solr replication is quite complicated process and
rollback or master-slave switch in this case is non-trivial task, that
may have result in whole dataset inconsistency.

Do you have any experience in such cases ? Maby load-balance in HAProxy
colud do the thing ? Something like:

.
server search1 192.168.1.1:8983 check port 8983 inter 20s fastinter 2
server search2 192.168.1.2:8983 backup
.


If you are using master-slave replication, you do not want to do this. 
Say the master went down, and it took you a while to get it back up. 
Unless you change the replication config, any data indexed on the slave 
would be gone as soon as the master was started back up, because the 
index would be replicated from the master.  You are quite correct that 
switching replication roles is non-trivial.


Using a load balancer in this manner is a great option if Solr is 
running in SolrCloud mode.  SolrCloud is a true cluster -- no masters, 
no slaves.  You can send indexing or queries to any system in the cloud 
and everything works.  The only real downside is that ZooKeeper (which 
is what turns Solr into SolrCloud) requires three servers for high 
availability, not two.  The third server could have much lower specs 
than the other two, as ZooKeeper's system requirements are typically 
quite modest.


Thanks,
Shawn

Re: Some questions

[Jean-Daniel Dupas via dovecot]

> Le 10 juil. 2019 à 14:06, Bardot Jérôme via dovecot  a 
> écrit :
> 
> Le 09/07/2019 à 17:28, Daniel Miller via dovecot a écrit :
>> 
>> On 7/9/2019 6:17 AM, Jérôme Bardot via dovecot wrote:
>>> Hello,
>>> 
>>> This is my first email here.
>>> I want to understand well how dovecot is integrate with ldap in a
>>> postfix/dovecot/ldap setup.
>>> I use a debian server.
>> 
>> Perfectly!
>> 
>>> 
>>> More specifically what dovecot need in ldap to work.
>>> I saw we can use several "mode" related to virtual domain, etc. For
>>> "start" i only need one domain with several address.
>>> I currently use fusiondirectory for manage my ldap users. i guess i
>>> can use that schema to auto create users email
>>> (name.firstn...@domain.tld for ie) ?
>>> I also want to setup some aliases and share directory based on ldap
>>> group/role can i do it ?
>>> 
>>> An other question is can we have two domain name for imap.domain.tld
>>> && smtp.domain.tld ?
>> 
>> Yes.
> There is some documentation somewhere on it ?
>> 
>> Dovecot & Postfix have no "hard" schema, or database definition, or
>> particular fields. You need to create map files which tell each server
>> how to use the information from LDAP (or any other database). Each
>> server (Postfix & Dovecot) have their own configuration which is
>> separate from each other. So you need to start with one or the other.
>> Postfix questions should be asked on the Postfix list.
>> 
>> Everything you asked for above is easily doable - just start with one
>> step at a time. Ask specific questions when you get stuck.
> 
> The map part stuck me at this time. Can i found somewhere a list of
> field should/can be map ? I think i’m ok with postfix conf if i
> understand well i can delegate pretty all stuff to dovecot/ldap.
> 
> 
> An other question is :
> 
> For all vitual stuff i always use a new user (system) with a custom
> home, all stuff i read are not clear for me about this point. There is
> some diagram with technical stuff about dovecot ?
> 

You should start by reading https://wiki2.dovecot.org/AuthDatabase/LDAP/Userdb

The main point if you use a single user is:

"If you're using a single UID and GID for all the users, you can specify them 
globally with mail_uid and mail_gid settings instead of returning them from 
LDAP."

Re: Some questions

[Bardot Jérôme via dovecot]

Le 09/07/2019 à 17:28, Daniel Miller via dovecot a écrit :
>
> On 7/9/2019 6:17 AM, Jérôme Bardot via dovecot wrote:
>> Hello,
>>
>> This is my first email here.
>> I want to understand well how dovecot is integrate with ldap in a
>> postfix/dovecot/ldap setup.
>> I use a debian server.
>
> Perfectly!
>
>>
>> More specifically what dovecot need in ldap to work.
>> I saw we can use several "mode" related to virtual domain, etc. For
>> "start" i only need one domain with several address.
>> I currently use fusiondirectory for manage my ldap users. i guess i
>> can use that schema to auto create users email
>> (name.firstn...@domain.tld for ie) ?
>> I also want to setup some aliases and share directory based on ldap
>> group/role can i do it ?
>>
>> An other question is can we have two domain name for imap.domain.tld
>> && smtp.domain.tld ?
>
> Yes.
There is some documentation somewhere on it ?
>
> Dovecot & Postfix have no "hard" schema, or database definition, or
> particular fields. You need to create map files which tell each server
> how to use the information from LDAP (or any other database). Each
> server (Postfix & Dovecot) have their own configuration which is
> separate from each other. So you need to start with one or the other.
> Postfix questions should be asked on the Postfix list.
>
> Everything you asked for above is easily doable - just start with one
> step at a time. Ask specific questions when you get stuck.

The map part stuck me at this time. Can i found somewhere a list of
field should/can be map ? I think i’m ok with postfix conf if i
understand well i can delegate pretty all stuff to dovecot/ldap.


An other question is :

For all vitual stuff i always use a new user (system) with a custom
home, all stuff i read are not clear for me about this point. There is
some diagram with technical stuff about dovecot ?


Thx

>
> -- 
> Daniel
>



0x053A41EF03878A98.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Hello Aki,

Thank you for your assurance.

It is really important to me to see that things are done well in the open 
source community. As there are many vested interests that would seek to have it 
otherwise.

And there are certain parts of the world, such as the place of my birth, 
Africa. Where Open Source really does hold the key to freedom from the worst 
forms of oppression and suffering known to man.

Sorry Adam, this will be my last cc on this topic. Just thought it was 
important to get this point to the Debian Release Management Team.

Warmest regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.




Understood Adam,

My thinking is that this is a package released with Debian 10. And so
has everything to do with the release.

I am new to the community, and have not yet studied the guidelines.

Regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.

-Original Message-
From: Adam D. Barratt 
To: arnoldo...@parallaxict.com
Cc: t...@sirainen.com, Dovecot Mailing List , 
debian-rele...@lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Wed, 10 Jul 2019 12:24:44 +0100

On 2019-07-10 12:01, Arnold Opio Oree wrote:

Please stop CCing this thread to the Debian Release Management mailing 
list.

If there are issues with the Dovecot packages in Debian then they
should 
be reported as bugs against the packages.

Thanks.

Adam
(Debian Release Team member)


-Original Message-
From: Aki Tuomi 
To: arnoldo...@parallaxict.com, t...@sirainen.com, Dovecot Mailing List

Cc: debian-rele...@lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Wed, 10 Jul 2019 14:13:53 +0300

Your report is most appreciated, though it lacked quite a lot of detail
that could've expedited the investigation why it happened. With the
further information we can hopefully figure out why you are
experiencing
the problems you reported. There is no offense intended.

Aki

On 10.7.2019 14.01, Arnold Opio Oree via dovecot wrote:
> Hello Timo,
> 
> I have scheduled some time to carry through the test requested by
> Aki.
> 
> I will add responding to you as part of that process.
> 
> Not sure if I have said or done anything to offend you, as your tone
> appears quite abrasive, even where you do not have an actual response
> to an observation.
> 
> It is my view that it is a privilege to provide an open source
> platform, and therefore a responsibility wherever feasible not only
> to properly maintain and support it, but also to properly document
> it; both of which from my experience and findings to date, Dovecot
> have failed to do.
> 
> Kind regards,
> 
> Arnold Opio Oree
> Chief Executive Officer
> Parallax Digital Technologies
> 
> arnoldo...@parallaxdt.com
> 
> 
> http://www.parallaxdt.com
> 
> 
> tel : +44 (0) 333 577 8587
> fax : +44 (0) 20 8711 2477
> 
> Parallax Digital Technologies is a trading name of Parallax Global
> Limited. U.K. Co. No. 08836288
> 
> The contents of this e-mail are confidential. If you are not the
> intended recipient you are to delete this e-mail immediately,
> disregard
> its contents and disclose them to no other persons.
> 
> 
> -Original Message-
> From: Timo Sirainen via dovecot <
> dovecot@dovecot.org
> >
> Reply-To: Timo Sirainen <
> t...@sirainen.com
> >
> To: Arnold Opio Oree <
> arnold.o...@parallaxict.com
> >
> Cc: Dovecot Mailing List <
> dovecot@dovecot.org
> >
> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
> archive - BUG REPORTS!
> Date: Wed, 10 Jul 2019 11:09:57 +0300
> 
> On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot <
> dovecot@dovecot.org
> > wrote:
> > > Dovecot Team,
> > > 
> > > I'd like to report a number of bugs, that are to my view all
> > > critical.
> 
> It would help to get your doveconf -n, example command lines causing
> the problems and the error messages it outputs or what the wrong
> behavior looks like in filesystem. It's now rather difficult to guess
> what exactly you tried and what happened.
> 
> Also what kind of output does readpst make? I'm not sure why 

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Understood Adam,
My thinking is that this is a package released with Debian 10. And so
has everything to do with the release.
I am new to the community, and have not yet studied the guidelines.
Regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.
-Original Message-
From: Adam D. Barratt 
To: arnoldo...@parallaxict.com
Cc: t...@sirainen.com, Dovecot Mailing List , 
debian-rele...@lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Wed, 10 Jul 2019 12:24:44 +0100

On 2019-07-10 12:01, Arnold Opio Oree wrote:
Please stop CCing this thread to the Debian Release Management mailing
list.
If there are issues with the Dovecot packages in Debian then they
should be reported as bugs against the packages.
Thanks.
Adam(Debian Release Team member)

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Adam D. Barratt via dovecot]

On 2019-07-10 12:01, Arnold Opio Oree wrote:

Please stop CCing this thread to the Debian Release Management mailing 
list.


If there are issues with the Dovecot packages in Debian then they should 
be reported as bugs against the packages.


Thanks.

Adam
(Debian Release Team member)

solr

[Maciej Milaszewski IQ PL via dovecot]

Hi
I have set up SOLR in accordance with documentation and it runs well.
I use solr like:
...
fts = solr
fts_solr = debug url=http://IP:8983/solr/ (solr in external machine)
..

Is replication of this system really essential ?? Due to my tests,
rollback of the solr server on external machine lasts less than minute
and it is nearly non-visible from client side.

On the other hand solr replication is quite complicated process and
rollback or master-slave switch in this case is non-trivial task, that
may have result in whole dataset inconsistency.

Do you have any experience in such cases ? Maby load-balance in HAProxy
colud do the thing ? Something like:

.
server search1 192.168.1.1:8983 check port 8983 inter 20s fastinter 2
server search2 192.168.1.2:8983 backup
.

Best Regards

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Aki Tuomi via dovecot]

Your report is most appreciated, though it lacked quite a lot of detail
that could've expedited the investigation why it happened. With the
further information we can hopefully figure out why you are experiencing
the problems you reported. There is no offense intended.

Aki

On 10.7.2019 14.01, Arnold Opio Oree via dovecot wrote:
> Hello Timo,
>
> I have scheduled some time to carry through the test requested by Aki.
>
> I will add responding to you as part of that process.
>
> Not sure if I have said or done anything to offend you, as your tone
> appears quite abrasive, even where you do not have an actual response
> to an observation.
>
> It is my view that it is a privilege to provide an open source
> platform, and therefore a responsibility wherever feasible not only to 
> properly maintain and support it, but also to properly document it; both of 
> which from my experience and findings to date, Dovecot have failed to do.
>
> Kind regards,
>
> Arnold Opio Oree
> Chief Executive Officer
> Parallax Digital Technologies
>
> arnoldo...@parallaxdt.com
>
> http://www.parallaxdt.com
>
> tel : +44 (0) 333 577 8587
> fax : +44 (0) 20 8711 2477
>
> Parallax Digital Technologies is a trading name of Parallax Global
> Limited. U.K. Co. No. 08836288
>
> The contents of this e-mail are confidential. If you are not the
> intended recipient you are to delete this e-mail immediately, disregard
> its contents and disclose them to no other persons.
>
>
> -Original Message-
> From: Timo Sirainen via dovecot 
> Reply-To: Timo Sirainen 
> To: Arnold Opio Oree 
> Cc: Dovecot Mailing List 
> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
> archive - BUG REPORTS!
> Date: Wed, 10 Jul 2019 11:09:57 +0300
>
> On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot <
> dovecot@dovecot.org> wrote:
>>> Dovecot Team,
>>>
>>> I'd like to report a number of bugs, that are to my view all
>>> critical.
> It would help to get your doveconf -n, example command lines causing
> the problems and the error messages it outputs or what the wrong
> behavior looks like in filesystem. It's now rather difficult to guess
> what exactly you tried and what happened.
>
> Also what kind of output does readpst make? I'm not sure why you're
> using DIRNAMEs here.
>
>>> doveadm-sync -1/general
>>>
>>> 1) If DIRNAMEs are not different between command line and
>>> mail_location doveadm sync will fail, saying that the source and
>>> destination directories are the same
> This sounds very strange. I'm not sure what exactly you did, and I
> couldn't reproduce with a small test.
>
>>> 2) The -n / -N flags do not work, and a sync will fail strangely if
>>> location is specified in the namespace definition
> Again, sounds strange.
>
>>> 3) Adds mbox to path name under mailbox directory (where syncing
>> from
>>> an mbox source)
> Probably with different parameters you could avoid it.
>
>>> 4) Not having the mailboxes at source named the same as those at
>>> destination causes errors and partial sync 
>>>
>>> 5) Not having the target mailboxes formatted to receive the sync
>>> (//DIRNAME/) will cause sync errors.
> I don't understand these. Target mailboxes are supposed to be empty
> initially, and after the initial sync they should be in the expected
> format. Why would they be different?
>
>>> doveadm-sync
>>>
>>> 1) With large synchronizations UIDs are corrupted where multiple
>>> syncs are executed and the program can no longer synchronize
> What exactly is the error message?
>
>>> dovecot
>>>
>>> 1) Panics and fails to expand ~ to user home: observed cases are
>>> where multiple namespaces are being used
> Panic message and more details would also help.
>
>>> With regards to the last error that I requested help on i.e.
>>> \Noselect. This has been resolved more-or-less by the workarounds
>>> that I have implemented for the bugs reported above.
>>>
>>> I have seen a number of threads whilst researching the \Noselect
>>> issue where people have been very confused. My finding was that
>>> \Noselect is a function of the IMAP specification server-side
>>> implementation RFC3501 (
>>> https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the
>>> server was returning directories with \Noselect because the
>> mailboxes
>>> were malformed on account of dovadm-sync errors. In order to fix
>> this
>>> I formed a bash command to transverse the mailbox hierarchy and
>>> create the missing folders critical to the sdbox format, namely
>>> DIRNAME.
> Nowadays we have also an option to disable creation of \Noselect
> folders, because they confuse people. Using mail_location = ...:NO-
> NOSELECT - It won't affect existing folders immediately though.
>
> -Original Message-
> From: Arnold Opio Oree 
> Reply-To: arnoldo...@parallaxict.com
> To: Aki Tuomi , Dovecot Mailing List <
> dovecot@dovecot.org>
> Cc: debian-rele...@lists.debian.org
> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
> archive - BUG REPORTS!
> 

Error: last_login_dict: Failed to write value: dict-server returned failure: sql dict: commit failed: Field 'password' doesn't have a default value

[Lefteris Tsintjelis via dovecot]

Hi, after upgrading my database I kept on getting those errors. It seems 
like later versions of databases have strict mode enabled by default and 
it must either be disabled or change the dovecot database schema. 
Besides the password, would anyone know what other fields must be 
changed to the dovecot schema and have a default value? Is there any 
other dovecot schema for strict mode?


Lefteris

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree via dovecot]

Hello Timo,

I have scheduled some time to carry through the test requested by Aki.

I will add responding to you as part of that process.

Not sure if I have said or done anything to offend you, as your tone
appears quite abrasive, even where you do not have an actual response
to an observation.

It is my view that it is a privilege to provide an open source
platform, and therefore a responsibility wherever feasible not only to properly 
maintain and support it, but also to properly document it; both of which from 
my experience and findings to date, Dovecot have failed to do.

Kind regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldo...@parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.


-Original Message-
From: Timo Sirainen via dovecot 
Reply-To: Timo Sirainen 
To: Arnold Opio Oree 
Cc: Dovecot Mailing List 
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Wed, 10 Jul 2019 11:09:57 +0300

On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot <
dovecot@dovecot.org> wrote:
> > Dovecot Team,
> > 
> > I'd like to report a number of bugs, that are to my view all
> > critical.

It would help to get your doveconf -n, example command lines causing
the problems and the error messages it outputs or what the wrong
behavior looks like in filesystem. It's now rather difficult to guess
what exactly you tried and what happened.

Also what kind of output does readpst make? I'm not sure why you're
using DIRNAMEs here.

> > doveadm-sync -1/general
> > 
> > 1) If DIRNAMEs are not different between command line and
> > mail_location doveadm sync will fail, saying that the source and
> > destination directories are the same

This sounds very strange. I'm not sure what exactly you did, and I
couldn't reproduce with a small test.

> > 2) The -n / -N flags do not work, and a sync will fail strangely if
> > location is specified in the namespace definition

Again, sounds strange.

> > 3) Adds mbox to path name under mailbox directory (where syncing
> from
> > an mbox source)

Probably with different parameters you could avoid it.

> > 4) Not having the mailboxes at source named the same as those at
> > destination causes errors and partial sync 
> > 
> > 5) Not having the target mailboxes formatted to receive the sync
> > (//DIRNAME/) will cause sync errors.

I don't understand these. Target mailboxes are supposed to be empty
initially, and after the initial sync they should be in the expected
format. Why would they be different?

> > doveadm-sync
> > 
> > 1) With large synchronizations UIDs are corrupted where multiple
> > syncs are executed and the program can no longer synchronize

What exactly is the error message?

> > dovecot
> > 
> > 1) Panics and fails to expand ~ to user home: observed cases are
> > where multiple namespaces are being used

Panic message and more details would also help.

> > With regards to the last error that I requested help on i.e.
> > \Noselect. This has been resolved more-or-less by the workarounds
> > that I have implemented for the bugs reported above.
> > 
> > I have seen a number of threads whilst researching the \Noselect
> > issue where people have been very confused. My finding was that
> > \Noselect is a function of the IMAP specification server-side
> > implementation RFC3501 (
> > https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the
> > server was returning directories with \Noselect because the
> mailboxes
> > were malformed on account of dovadm-sync errors. In order to fix
> this
> > I formed a bash command to transverse the mailbox hierarchy and
> > create the missing folders critical to the sdbox format, namely
> > DIRNAME.

Nowadays we have also an option to disable creation of \Noselect
folders, because they confuse people. Using mail_location = ...:NO-
NOSELECT - It won't affect existing folders immediately though.

-Original Message-
From: Arnold Opio Oree 
Reply-To: arnoldo...@parallaxict.com
To: Aki Tuomi , Dovecot Mailing List <
dovecot@dovecot.org>
Cc: debian-rele...@lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Mon, 08 Jul 2019 13:21:51 +0100

Hello Aki,

Thanks for looking into these.

I will as requested attempt the relevant procedures under Dovecot
2.3.6.

To make the test fair, I will need to fork the relevant production
groupware stack (which is now stable and in operation, with our
enterprise (email) data successfully migrated from Microsoft Exchange)
to a new staging server; given that the current staging server is now
of a materially different configuration 

Re: Dovecot behind Load Balancer

[Jean-Daniel Dupas via dovecot]

> Le 10 juil. 2019 à 11:46, Paolo Daniele  a écrit :
> 
> 
> 
> Il 10/07/19 11:44, Jean-Daniel Dupas ha scritto:
>> 
>>> Le 10 juil. 2019 à 10:24, Paolo Daniele via dovecot  a 
>>> écrit :
>>> 
>>> 
>>> Il 10/07/19 10:20, Aki Tuomi ha scritto:
> On 12/06/2019 20:02 Paolo Daniele via dovecot  wrote:
> 
> 
>  Hi,
>  i've a question for you.
>  I've two dovecot imap/pop server behind a zen load balancer.
>  Load balancing is made by lx4nat so the public ip address of my load 
> balancer contact directly the dovecot servers.
>  Since few months i've a message from thunderbird that i've reached the 
> imap limit login for ip.
>  I've triend to increase the max user ip parameter but sometimes i've the 
> same problem.
>  It's a strange things that actually i'm able to mitigate by reduce the 
> number of cached connections in Thunderbird but it's not normal.
>  What do you think about that?
>  Maybe there's some tuning that you can suggest.
>  Thank you,
>  Paolo
> 
 Have you ensured, by checking logs, that the connections are seen by 
 dovecot to come from public IP addresses?
 
 Also, thunderbird is known to open lots of concurrent connections.
 
 Aki
>>> Yes,
>>> connections are coming from the ip address of load balancer (also checked 
>>> with a netstat -an)
>> If connection are seen as coming from the IP address of the load balancer, 
>> isn't it normal that dovecot complains ?
>> That means that dovecot sees all connections as coming from a single client, 
>> which would explain why you reach that limit.
>> 
>> 
> Yeah of course, but i've checked that i haven't reach the max_user_per_ip 
> limit by counting dovecot process coming from that ip address.
> So the strange and the reason why i'm writing to you :)

Don't know if this is still relevant in your dovecot version, but did you see 
this:

https://serverfault.com/questions/385187/dovecot-ignoring-maximum-number-of-imap-connections
 


People had some issue by using the mail_max_userip_connections in the imap 
section and had to set it in the global section instead.

Re: Dovecot behind Load Balancer

[Paolo Daniele via dovecot]

Il 10/07/19 11:44, Jean-Daniel Dupas ha scritto:



Le 10 juil. 2019 à 10:24, Paolo Daniele via dovecot  a 
écrit :


Il 10/07/19 10:20, Aki Tuomi ha scritto:

On 12/06/2019 20:02 Paolo Daniele via dovecot  wrote:


  Hi,
  i've a question for you.
  I've two dovecot imap/pop server behind a zen load balancer.
  Load balancing is made by lx4nat so the public ip address of my load balancer 
contact directly the dovecot servers.
  Since few months i've a message from thunderbird that i've reached the imap 
limit login for ip.
  I've triend to increase the max user ip parameter but sometimes i've the same 
problem.
  It's a strange things that actually i'm able to mitigate by reduce the number 
of cached connections in Thunderbird but it's not normal.
  What do you think about that?
  Maybe there's some tuning that you can suggest.
  Thank you,
  Paolo


Have you ensured, by checking logs, that the connections are seen by dovecot to 
come from public IP addresses?

Also, thunderbird is known to open lots of concurrent connections.

Aki

Yes,
connections are coming from the ip address of load balancer (also checked with 
a netstat -an)

If connection are seen as coming from the IP address of the load balancer, 
isn't it normal that dovecot complains ?
That means that dovecot sees all connections as coming from a single client, 
which would explain why you reach that limit.


Yeah of course, but i've checked that i haven't reach the 
max_user_per_ip limit by counting dovecot process coming from that ip 
address.

So the strange and the reason why i'm writing to you :)

--
Paolo Daniele E-mail: paolo.dani...@tlcweb.it
Intendo s.r.l.Mobile: +39 388 9887357
Business Unit TLCWEB  Tel.:   +39 0968 430786
Via Coschi, 11Skype:  paolo_daniele1
88046 - Lamezia Terme, Cz Web:http://www.tlcweb.it
ItaliaWeb:http://www.intendo.it

Re: Dovecot behind Load Balancer

[Jean-Daniel Dupas via dovecot]

> Le 10 juil. 2019 à 10:24, Paolo Daniele via dovecot  a 
> écrit :
> 
> 
> Il 10/07/19 10:20, Aki Tuomi ha scritto:
>>> On 12/06/2019 20:02 Paolo Daniele via dovecot  wrote:
>>> 
>>> 
>>>  Hi,
>>>  i've a question for you.
>>>  I've two dovecot imap/pop server behind a zen load balancer.
>>>  Load balancing is made by lx4nat so the public ip address of my load 
>>> balancer contact directly the dovecot servers.
>>>  Since few months i've a message from thunderbird that i've reached the 
>>> imap limit login for ip.
>>>  I've triend to increase the max user ip parameter but sometimes i've the 
>>> same problem.
>>>  It's a strange things that actually i'm able to mitigate by reduce the 
>>> number of cached connections in Thunderbird but it's not normal.
>>>  What do you think about that?
>>>  Maybe there's some tuning that you can suggest.
>>>  Thank you,
>>>  Paolo
>>> 
>> Have you ensured, by checking logs, that the connections are seen by dovecot 
>> to come from public IP addresses?
>> 
>> Also, thunderbird is known to open lots of concurrent connections.
>> 
>> Aki
> Yes,
> connections are coming from the ip address of load balancer (also checked 
> with a netstat -an)

If connection are seen as coming from the IP address of the load balancer, 
isn't it normal that dovecot complains ? 
That means that dovecot sees all connections as coming from a single client, 
which would explain why you reach that limit.

Re: Dovecot behind Load Balancer

[Paolo Daniele via dovecot]

Il 10/07/19 10:27, Sami Ketola ha scritto:



On 10 Jul 2019, at 11.24, Paolo Daniele via dovecot  wrote:

connections are coming from the ip address of load balancer (also checked with 
a netstat -an)
I know that thunderbird has this problem and i haven't problem since the 
numbers of email domain increase.
I'm not sure the impact of rising the max user ip to an higher value in terms 
of ram consumption and cpu
Any suggestion?

At some point in time thunderbird opened 1 connection per email folder on 
destination. So if you have 20 folders
thunderbird would open 20 connections. Not sure if it is still true with 
current thunderbird versions.

There is a limit on thunderbird config that controls this.

Sami


I've checked the limitation on thunderbid and i've mitigated on our pcs 
(which have multiple email configured on the same servers) by telling 
thunderbird to store only one connection at the same time.
But i cannot "force" or contact every customer to check what client they 
use and change :)
This also create problems with Roundcube webmail which sometimes give me 
the same error.

Paolo

--
Paolo Daniele E-mail: paolo.dani...@tlcweb.it
Intendo s.r.l.Mobile: +39 388 9887357
Business Unit TLCWEB  Tel.:   +39 0968 430786
Via Coschi, 11Skype:  paolo_daniele1
88046 - Lamezia Terme, Cz Web:http://www.tlcweb.it
ItaliaWeb:http://www.intendo.it

Re: Dovecot behind Load Balancer

[Paolo Daniele via dovecot]

Il 10/07/19 10:27, Aki Tuomi ha scritto:

On 10/07/2019 11:24 Paolo Daniele via dovecot  wrote:

  
Il 10/07/19 10:20, Aki Tuomi ha scritto:

On 12/06/2019 20:02 Paolo Daniele via dovecot  wrote:


   Hi,
   i've a question for you.
   I've two dovecot imap/pop server behind a zen load balancer.
   Load balancing is made by lx4nat so the public ip address of my load 
balancer contact directly the dovecot servers.
   Since few months i've a message from thunderbird that i've reached the imap 
limit login for ip.
   I've triend to increase the max user ip parameter but sometimes i've the 
same problem.
   It's a strange things that actually i'm able to mitigate by reduce the 
number of cached connections in Thunderbird but it's not normal.
   What do you think about that?
   Maybe there's some tuning that you can suggest.
   Thank you,
   Paolo


Have you ensured, by checking logs, that the connections are seen by dovecot to 
come from public IP addresses?

Also, thunderbird is known to open lots of concurrent connections.

Aki

Yes,
connections are coming from the ip address of load balancer (also
checked with a netstat -an)
I know that thunderbird has this problem and i haven't problem since the
numbers of email domain increase.
I'm not sure the impact of rising the max user ip to an higher value in
terms of ram consumption and cpu
Any suggestion?
Thanks for your answer!
Paolo


What's your current limit? The default value is 100.

Aki

Actually:

mail_max_userip_connections = 500

Thanks,
Paolo

--
Paolo Daniele E-mail: paolo.dani...@tlcweb.it
Intendo s.r.l.Mobile: +39 388 9887357
Business Unit TLCWEB  Tel.:   +39 0968 430786
Via Coschi, 11Skype:  paolo_daniele1
88046 - Lamezia Terme, Cz Web:http://www.tlcweb.it
ItaliaWeb:http://www.intendo.it

Re: Dovecot behind Load Balancer

[Sami Ketola via dovecot]

> On 10 Jul 2019, at 11.24, Paolo Daniele via dovecot  
> wrote:
> 
> connections are coming from the ip address of load balancer (also checked 
> with a netstat -an)
> I know that thunderbird has this problem and i haven't problem since the 
> numbers of email domain increase.
> I'm not sure the impact of rising the max user ip to an higher value in terms 
> of ram consumption and cpu
> Any suggestion?

At some point in time thunderbird opened 1 connection per email folder on 
destination. So if you have 20 folders
thunderbird would open 20 connections. Not sure if it is still true with 
current thunderbird versions.

There is a limit on thunderbird config that controls this.

Sami

Re: Dovecot behind Load Balancer

[Aki Tuomi via dovecot]

> On 10/07/2019 11:24 Paolo Daniele via dovecot  wrote:
> 
>  
> Il 10/07/19 10:20, Aki Tuomi ha scritto:
> >> On 12/06/2019 20:02 Paolo Daniele via dovecot  wrote:
> >>
> >>
> >>   Hi,
> >>   i've a question for you.
> >>   I've two dovecot imap/pop server behind a zen load balancer.
> >>   Load balancing is made by lx4nat so the public ip address of my load 
> >> balancer contact directly the dovecot servers.
> >>   Since few months i've a message from thunderbird that i've reached the 
> >> imap limit login for ip.
> >>   I've triend to increase the max user ip parameter but sometimes i've the 
> >> same problem.
> >>   It's a strange things that actually i'm able to mitigate by reduce the 
> >> number of cached connections in Thunderbird but it's not normal.
> >>   What do you think about that?
> >>   Maybe there's some tuning that you can suggest.
> >>   Thank you,
> >>   Paolo
> >>
> > Have you ensured, by checking logs, that the connections are seen by 
> > dovecot to come from public IP addresses?
> >
> > Also, thunderbird is known to open lots of concurrent connections.
> >
> > Aki
> Yes,
> connections are coming from the ip address of load balancer (also 
> checked with a netstat -an)
> I know that thunderbird has this problem and i haven't problem since the 
> numbers of email domain increase.
> I'm not sure the impact of rising the max user ip to an higher value in 
> terms of ram consumption and cpu
> Any suggestion?
> Thanks for your answer!
> Paolo
> 

What's your current limit? The default value is 100.

Aki

Re: Dovecot behind Load Balancer

[Paolo Daniele via dovecot]

Il 10/07/19 10:20, Aki Tuomi ha scritto:

On 12/06/2019 20:02 Paolo Daniele via dovecot  wrote:


  Hi,
  i've a question for you.
  I've two dovecot imap/pop server behind a zen load balancer.
  Load balancing is made by lx4nat so the public ip address of my load balancer 
contact directly the dovecot servers.
  Since few months i've a message from thunderbird that i've reached the imap 
limit login for ip.
  I've triend to increase the max user ip parameter but sometimes i've the same 
problem.
  It's a strange things that actually i'm able to mitigate by reduce the number 
of cached connections in Thunderbird but it's not normal.
  What do you think about that?
  Maybe there's some tuning that you can suggest.
  Thank you,
  Paolo


Have you ensured, by checking logs, that the connections are seen by dovecot to 
come from public IP addresses?

Also, thunderbird is known to open lots of concurrent connections.

Aki

Yes,
connections are coming from the ip address of load balancer (also 
checked with a netstat -an)
I know that thunderbird has this problem and i haven't problem since the 
numbers of email domain increase.
I'm not sure the impact of rising the max user ip to an higher value in 
terms of ram consumption and cpu

Any suggestion?
Thanks for your answer!
Paolo

--
Paolo Daniele E-mail: paolo.dani...@tlcweb.it
Intendo s.r.l.Mobile: +39 388 9887357
Business Unit TLCWEB  Tel.:   +39 0968 430786
Via Coschi, 11Skype:  paolo_daniele1
88046 - Lamezia Terme, Cz Web:http://www.tlcweb.it
ItaliaWeb:http://www.intendo.it

Re: Dovecot behind Load Balancer

[Aki Tuomi via dovecot]

> On 12/06/2019 20:02 Paolo Daniele via dovecot  wrote:
> 
> 
>  Hi,
>  i've a question for you.
>  I've two dovecot imap/pop server behind a zen load balancer.
>  Load balancing is made by lx4nat so the public ip address of my load 
> balancer contact directly the dovecot servers.
>  Since few months i've a message from thunderbird that i've reached the imap 
> limit login for ip.
>  I've triend to increase the max user ip parameter but sometimes i've the 
> same problem.
>  It's a strange things that actually i'm able to mitigate by reduce the 
> number of cached connections in Thunderbird but it's not normal.
>  What do you think about that?
>  Maybe there's some tuning that you can suggest.
>  Thank you,
>  Paolo
>

Have you ensured, by checking logs, that the connections are seen by dovecot to 
come from public IP addresses?

Also, thunderbird is known to open lots of concurrent connections.

Aki

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Timo Sirainen via dovecot]

On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot  
wrote:
> 
> Dovecot Team,
> 
> I'd like to report a number of bugs, that are to my view all critical.

It would help to get your doveconf -n, example command lines causing the 
problems and the error messages it outputs or what the wrong behavior looks 
like in filesystem. It's now rather difficult to guess what exactly you tried 
and what happened.

Also what kind of output does readpst make? I'm not sure why you're using 
DIRNAMEs here.

> doveadm-sync -1/general
> 
> 1) If DIRNAMEs are not different between command line and mail_location 
> doveadm sync will fail, saying that the source and destination directories 
> are the same

This sounds very strange. I'm not sure what exactly you did, and I couldn't 
reproduce with a small test.

> 2) The -n / -N flags do not work, and a sync will fail strangely if location 
> is specified in the namespace definition

Again, sounds strange.

> 3) Adds mbox to path name under mailbox directory (where syncing from an mbox 
> source)

Probably with different parameters you could avoid it.

> 4) Not having the mailboxes at source named the same as those at destination 
> causes errors and partial sync 
> 
> 5) Not having the target mailboxes formatted to receive the sync 
> (//DIRNAME/) will cause sync errors.

I don't understand these. Target mailboxes are supposed to be empty initially, 
and after the initial sync they should be in the expected format. Why would 
they be different?

> doveadm-sync
> 
> 1) With large synchronizations UIDs are corrupted where multiple syncs are 
> executed and the program can no longer synchronize

What exactly is the error message?

> dovecot
> 
> 1) Panics and fails to expand ~ to user home: observed cases are where 
> multiple namespaces are being used

Panic message and more details would also help.

> With regards to the last error that I requested help on i.e. \Noselect. This 
> has been resolved more-or-less by the workarounds that I have implemented for 
> the bugs reported above.
> 
> I have seen a number of threads whilst researching the \Noselect issue where 
> people have been very confused. My finding was that \Noselect is a function 
> of the IMAP specification server-side implementation RFC3501 ( 
> https://tools.ietf.org/html/rfc3501#section-6.3.
>  
> 6).
>  And for me the server was returning directories with \Noselect because the 
> mailboxes were malformed on account of dovadm-sync errors. In order to fix 
> this I formed a bash command to transverse the mailbox hierarchy and create 
> the missing folders critical to the sdbox format, namely DIRNAME.

Nowadays we have also an option to disable creation of \Noselect folders, 
because they confuse people. Using mail_location = ...:NO-NOSELECT - It won't 
affect existing folders immediately though.

Re: Dovecot 2.3.6 on Solaris10: build issues, segfaults

[James via dovecot]

On 09/07/2019 06:35, Aki Tuomi via dovecot wrote:


 As for 1st, isn't gcc 3 rather old?


As is Solaris 10.  Solaris 10 predates gcc4 and comes/came with gcc3. No 
one says we have keep using the system gcc3 exclusively. 9.1.0 works too.


Solaris 10 03/05 = March 2005, release January 31, 2005 [1]
gcc 4.0.0: April 20, 2005 [2]


$ pkginfo -l SUNWgcc
   PKGINST:  SUNWgcc
  NAME:  gcc - The GNU C compiler
  CATEGORY:  system
  ARCH:  i386
   VERSION:  11.10.0,REV=2005.01.08.01.09
   BASEDIR:  /
VENDOR:  Oracle Corporation
  DESC:  GNU C - The GNU C compiler 3.4.3
PSTAMP:  sfw10-patch-x20121120060015
...



1. 
https://en.wikipedia.org/wiki/Solaris_%28operating_system%29#Version_history

"supported until 2021"

2. https://www.gnu.org/software/gcc/releases.html

Re: Dovecot 2.3.6 on Solaris10: build issues, segfaults

[James via dovecot]

On 09/07/2019 01:02, Joseph Tam via dovecot wrote:


Issue 2) Cannot build with --enable-hardening

Using gcc 9.1.0, "configure" step fails because fd passing was
broken, but the real problem was a compilation failure when
"--enable-hardening" is used.  Demonstration:


See:
https://dovecot.org/pipermail/dovecot/2019-January/114121.html



Issue 3) dovecot/doveconf segfaults on startup

It crashes here while processing dovecot.conf, as does "doveconf"

(settings-parser.c:1519 in setting_copy())
*dest_size = *src_size;

It appears *src_size is not an 8-byte address aligned (0x5597c).
It inherits this value from the calling routine as the sum of
"set" (8-byte aligned) + "def->offset"=20 => misaligned address.


32 bit or 64bit? cflags? I use 32 bit cc 12.6 and have no problem.  My 
test with gcc 9.1.0 didn't dump core either.

Re: Dovecot 2.3.6 on Solaris10: build issues, segfaults

[Timo Sirainen via dovecot]

On 9 Jul 2019, at 3.02, Joseph Tam via dovecot  wrote:
> 
> Issue 3) dovecot/doveconf segfaults on startup
> 
>   It crashes here while processing dovecot.conf, as does "doveconf"
> 
>   (settings-parser.c:1519 in setting_copy())
>   *dest_size = *src_size;

This is correct code.

>   It appears *src_size is not an 8-byte address aligned (0x5597c).
>   It inherits this value from the calling routine as the sum of
>   "set" (8-byte aligned) + "def->offset"=20 => misaligned address.
> 
>   (settings-parser.c:1597 in settings_dup_full())
>   src = CONST_PTR_OFFSET(set, def->offset);
> 
>   (gdb) p set
>   $2 = (const void *) 0x55968
>   (gdb) p *def
>   $3 = {type = SET_SIZE, key = 0x2d548 
> "submission_max_mail_size", offset = 20, list_info = 0x0}

This is unexpected. But I don't see how it's a Dovecot bug. It seems as if your 
compiler doesn't do padding correctly and then crashes because it didn't do it 
correctly. I guess you're compiling this as 32bit? Is size_t 32bit or 64bit?

Can you try with the below small test program if it prints the same 20?

#include 
#include 
#include 

#define in_port_t unsigned short

struct submission_settings {
bool verbose_proctitle;
const char *rawlog_dir;

const char *hostname;

const char *login_greeting;
const char *login_trusted_networks;

/* submission: */
size_t submission_max_mail_size;
unsigned int submission_max_recipients;
const char *submission_client_workarounds;
const char *submission_logout_format;

/* submission backend: */
const char *submission_backend_capabilities;

/* submission relay: */
const char *submission_relay_host;
in_port_t submission_relay_port;
bool submission_relay_trusted;

const char *submission_relay_user;
const char *submission_relay_master_user;
const char *submission_relay_password;

const char *submission_relay_ssl;
bool submission_relay_ssl_verify;

const char *submission_relay_rawlog_dir;
unsigned int submission_relay_max_idle_time;

unsigned int submission_relay_connect_timeout;
unsigned int submission_relay_command_timeout;

/* imap urlauth: */
const char *imap_urlauth_host;
in_port_t imap_urlauth_port;

int parsed_workarounds;
};

int main(void)
{
struct submission_settings set;

printf("offset = %ld\n", offsetof(struct submission_settings, 
submission_max_mail_size));
printf("size = %ld\n", sizeof(set.submission_max_mail_size));
return 0;
}