Re: Infinity loop when run "doveadm quota get -A" from Dovecot Director with 500 users
Thanks for the patch, we'll take a look at it.
Aki
> On 29/01/2021 04:49 Duc Anh Do wrote:
>
>
> Hi all,
>
> Because I think this is a race condition so instead of using only one
> current_ioloop:
> * I create ioloop for each Backend in the list of Director
> * I think connections from aDirector to same Backend are synchronous so I
> don't create ioloop for each connection
> * I think ioloop must be destroyed in correct order so I use a linked list
> to manage them, create then push and pop then destroy (sorry, I don't know
> any existing structure in source of Dovecot that I can re-use)
> I tested my patch with both "doveadm quota get -A" and "doveadm quota get -u
> xxx" many times. No error occurs (timeout leak, segment fault... etc).
> If you are interested in my patch, any comment is highly appreciated. I
> modified source files which might be shared with other doveadm commands so
> I'm not sure it's safe 100%.
>
> Thanks,
> Anh Do
>
>
> On Wed, 27 Jan 2021 at 16:20, Duc Anh Do wrote:
> > Hi all,
> >
> > I have one Dovecot Director, two Dovecot Backends and one LDAP server with
> > about 500 users. I would like to run doveadm quota get -A from the Director.
> > In each Backend, there is no problem when run the command:
> > # doveadm quota get -A
> > user1 User quota STORAGE 0 10485760 0
> > user1 User quota MESSAGE 0 - 0
> > …
> > user500 User quota STORAGE 0 10485760 0
> > user500 User quota MESSAGE 0 - 0
> >
> > However, when I run from the Director, the command might stuck in an
> > infinity loop (I have to terminate to quit):
> > # doveadm quota get -A
> > user1 User quota STORAGE 0 10485760 0
> > user1 User quota MESSAGE 0 - 0
> > …
> > user49 User quota STORAGE 0 10485760 0
> > user49 User quota MESSAGE 0 - 0
> > user66 User quota STORAGE 0 10485760 0
> > user66 User quota MESSAGE 0 - 0
> > ^Cdoveadm(user86): Error: doveadm server failure
> > doveadm: Error: Failed to iterate through some users
> > doveadm: Error: backend2.local:24245: Command quota get failed for user53:
> > EOF
> > doveadm: Error: backend1.local:24245: Command quota get failed for user66:
> > EOF
> > doveadm: Error: Aborted
> >
> > This problem occurs in both Dovecot 2.2.36 and Dovecot 2.3.11, 2.3.13 (I
> > build Dovecot from source). It's ok for me to get quota of one user from
> > the Director:
> > # doveadm quota get -u user1
> > Quota name Type Value Limit %
> > User quota STORAGE 0 10485760 0
> > User quota MESSAGE 0 - 0
> > And if there's only one Backend,doveadm quota get -A from the Director
> > works well too.
> >
> > After investigating, I found the infinity loop:
> > File src/doveadm/doveadm-mail-server.c:
> > static void doveadm_server_flush_one(struct doveadm_server *server)
> > {
> > unsigned int count = array_count(>queue);
> >
> > do {
> > io_loop_run(current_ioloop);
> > } while (array_count(>queue) == count &&
> > doveadm_server_have_used_connections(server) &&
> > !DOVEADM_MAIL_SERVER_FAILED());
> > }
> >
> > In case there're many Backends, I see only global variable current_ioloop
> > is used to notify in the callback function. Might this be a race condition?
> > I understand there's a workaround to do my work:
> >
> > * Run doveadm user '*' to get all users
> > * Loop through all users and run doveadm quota get -u xxx
> >
> > Thanks,
> > Anh Do
>
>
> --
>
> Thanks,
> Duc Anh
>
> Email: doducanh2...@gmail.com
> Skype: ducanh.do88
> Mobile: +84975730526
Re: Infinity loop when run "doveadm quota get -A" from Dovecot Director with 500 users
Hi all,
Because I think this is a race condition so instead of using only one
*current_ioloop*:
- I create ioloop for each Backend in the list of Director
- I think connections from a Director to same Backend are synchronous so
I don't create ioloop for each connection
- I think ioloop must be destroyed in correct order so I use a linked
list to manage them, create then push and pop then destroy (sorry, I don't
know any existing structure in source of Dovecot that I can re-use)
I tested my patch with both "doveadm quota get -A" and "doveadm quota get
-u xxx" many times. No error occurs (timeout leak, segment fault... etc).
If you are interested in my patch, any comment is highly appreciated. I
modified source files which might be shared with other doveadm commands so
I'm not sure it's safe 100%.
Thanks,
Anh Do
On Wed, 27 Jan 2021 at 16:20, Duc Anh Do wrote:
> Hi all,
>
> I have one Dovecot Director, two Dovecot Backends and one LDAP server with
> about 500 users. I would like to run *doveadm quota get -A* from the
> Director.
> In each Backend, there is no problem when run the command:
> # doveadm quota get -A
> user1 User quota STORAGE 0 10485760
>0
> user1 User quota MESSAGE 0-
>0
> …
> user500 User quota STORAGE 0 10485760
>0
> user500 User quota MESSAGE 0-
>0
>
> However, when I run from the Director, the command might stuck in an
> infinity loop (I have to terminate to quit):
> # doveadm quota get -A
> user1 User quota STORAGE 0 10485760
>0
> user1 User quota MESSAGE 0-
>0
> …
> user49User quota STORAGE 0 10485760 0
> user49User quota MESSAGE 0- 0
> user66User quota STORAGE 0 10485760 0
> user66User quota MESSAGE 0- 0
> ^Cdoveadm(user86): Error: doveadm server failure
> doveadm: Error: Failed to iterate through some users
> doveadm: Error: backend2.local:24245: Command quota get failed for user53:
> EOF
> doveadm: Error: backend1.local:24245: Command quota get failed for user66:
> EOF
> doveadm: Error: Aborted
>
> This problem occurs in both Dovecot 2.2.36 and Dovecot 2.3.11, 2.3.13 (I
> build Dovecot from source). It's ok for me to get quota of one user from
> the Director:
> # doveadm quota get -u user1
> Quota name TypeValueLimit%
> User quota STORAGE 0 104857600
> User quota MESSAGE 0-0
> And if there's only one Backend, *doveadm quota get -A* from the Director
> works well too.
>
> After investigating, I found the infinity loop:
> File src/doveadm/doveadm-mail-server.c:
> static void doveadm_server_flush_one(struct doveadm_server *server)
> {
>unsigned int count = array_count(>queue);
>
>do {
> io_loop_run(current_ioloop);
>} while (array_count(>queue) == count &&
> doveadm_server_have_used_connections(server) &&
> !DOVEADM_MAIL_SERVER_FAILED());
> }
>
> In case there're many Backends, I see only global variable
> *current_ioloop* is used to notify in the callback function. Might this
> be a race condition?
> I understand there's a workaround to do my work:
>
>- Run *doveadm user '*'* to get all users
>- Loop through all users and run *doveadm quota get -u xxx*
>
>
> Thanks,
> Anh Do
>
--
Thanks,
Duc Anh
Email: doducanh2...@gmail.com
Skype: ducanh.do88
Mobile: +84975730526
fix-doveadm-quota-director-20210128-2055.patch
Description: Binary data
Infinity loop when run "doveadm quota get -A" from Dovecot Director with 500 users
Hi all,
I have one Dovecot Director, two Dovecot Backends and one LDAP server with
about 500 users. I would like to run *doveadm quota get -A* from the
Director.
In each Backend, there is no problem when run the command:
# doveadm quota get -A
user1 User quota STORAGE 0 10485760
0
user1 User quota MESSAGE 0-
0
…
user500 User quota STORAGE 0 10485760
0
user500 User quota MESSAGE 0-
0
However, when I run from the Director, the command might stuck in an
infinity loop (I have to terminate to quit):
# doveadm quota get -A
user1 User quota STORAGE 0 10485760
0
user1 User quota MESSAGE 0-
0
…
user49User quota STORAGE 0 10485760 0
user49User quota MESSAGE 0- 0
user66User quota STORAGE 0 10485760 0
user66User quota MESSAGE 0- 0
^Cdoveadm(user86): Error: doveadm server failure
doveadm: Error: Failed to iterate through some users
doveadm: Error: backend2.local:24245: Command quota get failed for user53:
EOF
doveadm: Error: backend1.local:24245: Command quota get failed for user66:
EOF
doveadm: Error: Aborted
This problem occurs in both Dovecot 2.2.36 and Dovecot 2.3.11, 2.3.13 (I
build Dovecot from source). It's ok for me to get quota of one user from
the Director:
# doveadm quota get -u user1
Quota name TypeValueLimit%
User quota STORAGE 0 104857600
User quota MESSAGE 0-0
And if there's only one Backend, *doveadm quota get -A* from the Director
works well too.
After investigating, I found the infinity loop:
File src/doveadm/doveadm-mail-server.c:
static void doveadm_server_flush_one(struct doveadm_server *server)
{
unsigned int count = array_count(>queue);
do {
io_loop_run(current_ioloop);
} while (array_count(>queue) == count &&
doveadm_server_have_used_connections(server) &&
!DOVEADM_MAIL_SERVER_FAILED());
}
In case there're many Backends, I see only global variable *current_ioloop*
is used to notify in the callback function. Might this be a race condition?
I understand there's a workaround to do my work:
- Run *doveadm user '*'* to get all users
- Loop through all users and run *doveadm quota get -u xxx*
Thanks,
Anh Do
