Re: UID THREAD REFS US-ASCII ALL slow / stalls forever on one folder.
Hi Thanks for the debugging hints! client_timeout sat to 30M and the UID THREAD REFS US-ASCII ALL actually completes. But first after ~10 mins on a CPU: Intel(R) Celeron(R) CPU N2930 @ 1.83GHz (1833.38-MHz K8-class CPU) and after 69.484 secs on a CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (The same jail, moved to faster hardware) 63994 messages in around 40k threads. What could cause the much spent time building the thread data? version is still 3.2.2, but 3.0.13 shows the same behavior. The thread output of the slow folder is here FWIW https://pastebin.com/PpiJ8X2E Regards /Jesper On 03.07.2020 01.52, ellie timoney wrote: Hi, I think I would do something like: 0) set client_timeout to a big value (see below) 1) let the imapd start normally 2) connect to it with a minimal imap client (like imtest or telnet) 3) check logs to see which imapd process id your client is connected to (if there's more than one) 4) use the "gdb /path/to/binary pid" invocation to attach gdb to the running imap process 5) repeat #4 in a few different ways if it's unable to find symbols 6) set a breakpoint on the cmd_thread function (since that's what handles the THREAD command) and then continue 7) back in your imap client, send the UID THREAD REFS US-ASCII ALL" command and step through to see what happens Note that these are not exhaustive steps, more of a "get started, and then react accordingly, depending on what you see" I would also try variations of that THREAD command -- if you add/remove options, does it start working? Can you isolate the problem to a specific combination of options? Does it only happen for some mailboxes? You probably want to set client_timeout to a big value. The default is 10 seconds, so the server will probably throw your client off while you're reading output from gdb, and then you'll wind up debugging the "disconnect an unresponsive client code" instead. I usually set this to be 30 minutes or so for debugging. For 3.2 and newer, you can spell this as "30m". For older versions, the value is in seconds, so you want "1800". Permissions may be awkward. You will need to be the cyrus user (or root) to connect gdb to the running imapd, but you will also need permission to read the source that it was built from, which is probably not owned by the cyrus user. In my case it's under my user account's home directory, so I add the cyrus user to my user account's group, and make sure the path to the source directory is g+rx (because I don't like running gdb as root). Cheers, ellie On Fri, Jul 3, 2020, at 5:23 AM, Jesper Schmitz Mouridsen via Info-cyrus wrote: Hi. I recently upgraded Cyrus to 3.2.2 from 3.0.13. Now threading "UID THREAD REFS US-ASCII ALL" on one specific folder with >50K mails, makes imapd process use 100 CPU% without any progress. truss[1] or dtrace shows no output. The process seems totally stuck. I installed in a FreeBSD 12.1 jail. Any hints beside the online docs of how to use gdb to see what is going one. I could not make imapd run under gdb even with the -D option and debug_command setting or reading https://www.cyrusimap.org/imap/developer/developer-testing.html?highlight=debug It is fast enough on other folders also with ~50k mails < 3 secs. Any hints, highly appreciated :-) [1] https://www.freebsd.org/cgi/man.cgi?truss Regards Jesper Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
UID THREAD REFS US-ASCII ALL slow / stalls forever on one folder.
Hi. I recently upgraded Cyrus to 3.2.2 from 3.0.13. Now threading "UID THREAD REFS US-ASCII ALL" on one specific folder with >50K mails, makes imapd process use 100 CPU% without any progress. truss[1] or dtrace shows no output. The process seems totally stuck. I installed in a FreeBSD 12.1 jail. Any hints beside the online docs of how to use gdb to see what is going one. I could not make imapd run under gdb even with the -D option and debug_command setting or reading https://www.cyrusimap.org/imap/developer/developer-testing.html?highlight=debug It is fast enough on other folders also with ~50k mails < 3 secs. Any hints, highly appreciated :-) [1] https://www.freebsd.org/cgi/man.cgi?truss Regards Jesper Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: django-cyradm
On 12.12.2019 21.26, Niels Dettenbach wrote: Am Donnerstag, 12. Dezember 2019, 21:06:46 CET schrieb Jesper Schmitz Mouridsen via Info-cyrus: It has self service support. The system has 3 roles "admin","domain admin" and "account user". If you are interested take a look at sounds very interesting - i planned to write a similiar open source product (after the very flexible web-cyradm was very outdated) - but with exim / pam_mysql (should be similiar) and reserved some spare time over christmas and january for it... After planning with django years ago, my current plan was to use flask or similiar more light, because installation of django (compared to the old LAMP based web-cyradm) is more difficult for many users / impossible in some typical mass hosting environments and easier to maintain in regards of security over longer timespan (django typically needs security updates several times a year from what i experienced with it in other projects). And with flask it is (at least by my intention so far...) more easy to use it on i.e. embedded hardware or "small email servers" with a DevOps like software / "firmware" management. I very like the very intelligent (because it allows very flexible email configs as MTA integrations) database design of web_cyradm. While i've not found a way to easily "reuse" that existing database structure (or "rebuilt" it with the django DB subsystem) in flask is has to be done by hand - so "no prob" at this edge for me. But will definitely check it - and (if interested) contribute (where it may makes sense). it may possibly still is a nice base or working solution for what i'm looking for (saving me time to write something complete byself) and if flask or so is still an option, the similiar concept should allow to "reuse" at least the more fiddely python code from a django project. is there any public "main" repo of the project? https://github.com/jsm222/django-cyradm/ many thanks for your time and posting, niels. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
django-cyradm
Hi cyrus list. (This email contains self promotion of a cyrus/imap related piece of software..) I wrote some years ago (and recently updated) a small django application for cyrus and postfix, with database lookup tables. It has self service support. The system has 3 roles "admin","domain admin" and "account user". If you are interested take a look at https://djcyradm.schmitz.computer Kind regards Jesper Schmitz Mouridsen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
imclient libcyrus and ssl port 993
Hi. I'm trying to make the imclient_* functions work over imaps. imclient_startls works on port 143. Is there a way with libcyrus to connect, the same way as openssl s_client -connect localhost:993 e.g imaps. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: imclient_authenticate wrong prompt order.
mandag den 26. september 2016 15.51.45 CEST skrev Dan White:
On 09/24/16 17:28 +0200, jesper--- via Info-cyrus wrote:
The following sample prompts for entering the password after ones actually
did enter the password. 1) Why is that e.g why does this sample write
"please enter your password: " after the password is entered, and then
exits?
The authentication works. Only the prompting is a problem. ...
Have a look at doc/programming.html#callbacks_interactions within the cyrus
sasl source.
Thanks for the hint.
Can you provide an example which includes callbacks that is
not working as expected?
No I could not.
For reference I came up with the following:
#include
#include
#include
#include
static int getsecret_func(sasl_conn_t *conn,
void *context __attribute__((unused)),
int id,
sasl_secret_t **psecret);
static struct imclient *imclient;
static int getauthname_func(void *context,
int id,
const char **result,
unsigned *len);
void fatal(const char *s, int code)
{
fprintf(stderr, "Fatal error: %s\n", s);
}
/* callbacks we support. This is a global variable at the
top of the program */
static sasl_callback_t callbacks[] = {
{
SASL_CB_GETREALM, NULL, NULL/* we'll just use an interaction if this
comes up */
}, {
SASL_CB_USER, NULL, NULL /* we'll just use an interaction if
this comes up */
}, {
SASL_CB_AUTHNAME, _func, NULL /*A mechanism should call
getauthname_func if it
needs the authentication name */
},{
SASL_CB_PASS, _func, NULL /* Call getsecret_func if
need secret */
}, {
SASL_CB_LIST_END, NULL, NULL
}
};
static int getsecret_func(sasl_conn_t *conn,
void *context __attribute__((unused)),
int id,
sasl_secret_t **psecret)
{
*psecret = malloc(sizeof(sasl_secret_t)+7);
char * secret = "cyrus\0";
*psecret = malloc(sizeof(sasl_secret_t*)+strlen(secret)+1);
static sasl_secret_t *x;
x = (sasl_secret_t *) realloc(x, sizeof(sasl_secret_t) +
strlen(secret));
memcpy(x->data,secret,sizeof(sasl_secret_t)+strlen(secret)+1);
x->len=strlen(secret);
*psecret = x;
return SASL_OK;
}
static int getauthname_func(void *context,
int id,
const char **result,
unsigned *len)
{
if (id!=SASL_CB_AUTHNAME) return SASL_FAIL;
char *authname = "cyrus\0";
*result = malloc(sizeof(char)*strlen(authname)+1);
*result = authname;
unsigned length =strlen(authname); ;
len=
return SASL_OK;
}
char server[] = "localhost" ;
char port[] = "imap";
char mech[] ="CRAM-MD5";
char service[] = "imap";
int main() {
if(imclient_connect(, server, port,callbacks)) {
fprintf(stderr,
"error: Couldn't connect to %s %s\n",
server, port);
}
if(imclient_authenticate(imclient, mech,service,NULL, 0,256)) {
fprintf(stderr,
"error: Authentication failed");
}
return 0;
}
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
