RE: First steps in Dovecot; IMAP not working
>Two things to check: > > # What does dovecot think user "mailtest" has? > doveadm user mailtest I get this: field value uid 1002 gid 8 home/home/mailtest mailmbox:~/mail:INBOX=/var/mail/mailtest system_groups_user mailtest > # The sticky bit should be set on /var/mail (you didn't mention > # setting it. It probably doesn't have bearing on this problem, > # but it will make it a little more secure. > chmod 1777 /var/mail You're right, I didn't do it. >I usually don't use this command, I look at the log file which seems to have more details. Try > looking there for more diagnostics. Also, look at your MTA's logs as well. I'll try to locate them. Thank you.
Re: Dict proxy client returning empty string instead of multiline string
> > > > On 10/16/2016 11:16 PM, Pierre Jaury wrote:
> > > >> Hello,
> > > >>
> > > >> I am using a dict proxy for my sieve extdata plugin to access some
> > > >> fields from an SQLite database (autoreply text and other
> > > >> database-configured items).
> > > >>
> > > >> All tests are performed against version 2.2.25.
> > > >>
> > > >> $ dovecot --version
> > > >> 2.2.25 (7be1766)
> > > >>
> > > >> My configuration looks like:
> > > >>
> > > >> dict {
> > > >> sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict
> > > >> }
> > > >>
> > > >> [...]
> > > >> sieve_extdata_dict_uri = proxy::sieve
> > > >>
> > > >> I am able to read pretty much any attribute without any issue,
> > > >> except
> > > >> when the value contains a special character like "\r" or "\n". By
> > > >> using
> > > >> the doveadm dict client, I narrowed it down to the dictionary
> > > >> management
> > > >> part (either server, protocol or client).
> > > >>
> > > >> I was suspecting escaping functions from "lib/strescape.c" (mostly
> > > >> str_tabescape and its counterpart, used by "lib-dict/client.c"),
> > > >> so I
> > > >> monitored socket communications. It seems that escaping is done
> > > >> properly
> > > >> on the server and the socket is not an issue either.
> > > >>
> > > >> The following strace dump results from running "doveadm dict get"
> > > >> against the dict socket:
> > > >>
> > > >> connect(8, {sa_family=AF_UNIX, sun_path="..."}, 110) = 0
> > > >> fstat(8, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
> > > >> [...]
> > > >> write(8, "H2\t0\t0\tad...@domain.tld\tsieve\n", 30) = 30
> > > >> [...]
> > > >> read(8, "Otest\1r\1ntest\n", 8192) = 14
> > > >>
> > > >> Indeed "\1r" and "\1n" are the escape sequences used by
> > > >> "lib/strescape.c". I went deeped and debugged the call to
> > > >> "dict_lookup"
> > > >> performed by doveadm. Indeed the client gets the proper string
> > > >> from the
> > > >> socket and to my surprise, it is properly unescaped.
> > > >>
> > > >> Then, in "client_dict_lookup" ("lib-dict/dict-client.c"), the call
> > > >> to
> > > >> "p_strdup" returns an empty string (null byte set at the target
> > > >> address).
> > > >>
> > > >> Before the call to the dict "->lookup" attribute
> > > >> (client_dict_lookup):
> > > >>
> > > >>RAX: 0x773a37c0 (push r14)
> > > >>RBX: 0x6831b8 ("priv/reply_body")
> > > >>RCX: 0x7fffe240 --> 0x682a60 --> 0x6831b8
> > > >> ("priv/reply_body")
> > > >>RDX: 0x6831b8 ("priv/reply_body")
> > > >>RSI: 0x683288 --> 0x77653120 --> 0x773ea620 ([...])
> > > >>RDI: 0x690ad0 --> 0x77400713 --> 0x75250079786f7270
> > > >> ('proxy')
> > > >>
> > > >>0x773a1f10 :movrcx,r11 (value_r)
> > > >>0x773a1f13 :movrdx,r8 (key)
> > > >>0x773a1f16 :movrsi,r10 (pool)
> > > >>0x773a1f19 :movrdi,r9 (dict)
> > > >>0x773a1f1c :addrsp,0x8
> > > >> => 0x773a1f20 :jmprax
> > > >>
> > > >> Before the call to p_strdup in "client_dict_lookup":
> > > >>
> > > >>RSI: 0x6832d8 ("test\r\ntest") (lookup.result.value)
> > > >>RDI: 0x683288 --> 0x77653120 --> [...] (pool)
> > > >>RAX: 0x0 (result)
> > > >>
> > > >>0x773a384f: nop
> > > >>0x773a3850: movrsi,QWORD PTR [rsp+0x8]
> > > >>0x773a3855: movrdi,r14
> > > >> => 0x773a3858: call 0x7736d3c0
> > > >>0x773a385d: movQWORD PTR [r13+0x0],rax
> > > >>0x773a3861: movrsi,QWORD PTR [rsp+0x18]
> > > >>0x773a3866: xorrsi,QWORD PTR fs:0x28
> > > >>0x773a386f: moveax,ebx
> > > >>
> > > >> After the call:
> > > >>
> > > >>0x773a3850: movrsi,QWORD PTR [rsp+0x8]
> > > >>0x773a3855: movrdi,r14
> > > >>0x773a3858: call 0x7736d3c0
> > > >> => 0x773a385d: movQWORD PTR [r13+0x0],rax
> > > >>0x773a3861: movrsi,QWORD PTR [rsp+0x18]
> > > >>0x773a3866: xorrsi,QWORD PTR fs:0x28
> > > >>0x773a386f: moveax,ebx
> > > >>0x773a3871: jne0x773a38da
> > > >>
> > > >>RSI: 0x0
> > > >>RDI: 0x6832d8 --> 0x0
> > > >>RAX: 0x6832d8 --> 0x0 (result)
> > > >>
> > > >> It is worth noting that I can reproduce the exact same execution
> > > >> flow
> > > >> with a non-multiline result string (lookup.result.value) that is
> > > >> properly copied by "p_strdup" and returned in RAX, then displayed
> > > >> by
> > > >> doveadm.
> > > >>
Re: dovecot 2.2.25 BUG: local_name is not matching correctly
On 13.10.2016 16:09, Arkadiusz Miśkiewicz wrote:
Bug report:
When using dovecot 2.2.25 SNI capability it doesn't always match proper vhost
config. For example if we have such config:
local_name imap.example.com {
ssl_cert =
Hi!
Fixed in
https://github.com/dovecot/core/commit/c19c44f87ef3fe40cae4be9a86ee9327a7370e46
Aki Tuomi
Dovecot oy
Re: First steps in Dovecot; IMAP not working
Marnaud writes: "mailtest", the new user, is in group mail(8). In addition, I've added write permission for "others" to /var/mail. Now, I'm trying to send a message to "mailtest" from another, working, e-mail account and nothing happens. This time, "doveadm log errors" is empty. In short, I don't get any error but no mail either. Two things to check: # What does dovecot think user "mailtest" has? doveadm user mailtest # The sticky bit should be set on /var/mail (you didn't mention # setting it. It probably doesn't have bearing on this problem, # but it will make it a little more secure. chmod 1777 /var/mail Since "doveadm log errors" returns an empty result, where should I look for the problem? I usually don't use this command, I look at the log file which seems to have more details. Try looking there for more diagnostics. Also, look at your MTA's logs as well. Joseph Tam
Re: Massive LMTP Problems with dovecot
On 17 Oct 2016, at 17:31, Ralf Hildebrandt wrote: > > * Ralf Hildebrandt : > >>> It seems to loop in sha1_loop & hash_format_loop >> >> The problem occurs in both 2.3 and 2.2 (I just updated to 2.3 to check). > > I'm seeing the first occurence of that problem on the 10th of october! > > I was using (prior to the 10th) : 2.3.0~alpha0-1~auto+371 > On the 10th I upgraded (16:04) to: 2.3.0~alpha0-1~auto+376 > > I'd think the change must have been introduced between 371 and 376 :) > > I then went back to, issues went away: 2.2.25-1~auto+49 > and the issues reappeared with 2.2.25-1~auto+57 https://github.com/dovecot/core/commit/9b5fa7fdd9b9f1f61eaddda48036df200fc5e56e should fix this.
Re: Dict proxy client returning empty string instead of multiline string
Sorry, sent the wrong version, please see amended one attached.
Aki
> On October 17, 2016 at 10:18 PM Aki Tuomi wrote:
>
>
> Oh duh, it used datastack pool. Try again with the attached patch? Please
> remember to clear the previous one out before trying.
>
> Aki
>
> >
> > The trace is missing some symbols, I will debug tomorrow and see where
> > the call comes from exactly.
> >
> > Regards,
> >
> >
> > On 10/17/2016 06:23 PM, Aki Tuomi wrote:
> > > Hi!
> > >
> > > Looking at the code, I think the bug is that it just copies pointer to
> > > the value, it should, instead, duplicate the value to some memory region.
> > > Can you see if this following patch fixes it?
> > >
> > > Aki
> > >
> > >> On October 17, 2016 at 4:14 PM Pierre Jaury wrote:
> > >>
> > >>
> > >> Okay, it seems to me that the bug is due to "t_str_tabunescape" using
> > >> the unsafe datastack ("t_strdup_noconst") while the string is actually
> > >> returned in an async callback.
> > >>
> > >> Before it is handled by "client_dict_lookup", "client_dict_wait"
> > >> actually fires some IO loops that are more than likely to call "t_pop"
> > >> and free/flush the result string (I checked, it does call "t_pop" a
> > >> couple times indeed). Maybe it would be safer to use a standard
> > >> datastack pool when unescaping a string in that context.
> > >>
> > >> I could work on a patch that would:
> > >>
> > >> - add an optional "pool" attribute to the "client_dict_cmd" structure,
> > >> - pass the pool to the async lookup function,
> > >> - use the pool when escaping strings that should survive the callback
> > >>chain
> > >>
> > >> What do you think?
> > >>
> > >> Regards,
> > >> kaiyou
> > >>
> > >>
> > >> On 10/17/2016 09:52 AM, Pierre Jaury wrote:
> > >>> While trying to isolate properly and reproduce, I was able to trigger
> > >>> the same bug with the following code:
> > >>>
> > >>> struct dict *dict;
> > >>> char* dict_uri = "proxy::sieve";
> > >>> char* key = "priv/key";
> > >>> char* username = "ad...@domain.tld";
> > >>> char* value, error;
> > >>>
> > >>> dict_drivers_register_builtin();
> > >>> dict_init(dict_uri, DICT_DATA_TYPE_STRING, username,
> > >>> doveadm_settings->base_dir, &dict, &error);
> > >>> dict_lookup(dict, pool_datastack_create(), key, &value);
> > >>> printf(">%s\n", value); // outputs an empty string
> > >>> dict_deinit(&dict);
> > >>>
> > >>> I trimmed it to the bare minimal string manipulation functions involved
> > >>> but cannot reproduce in that case:
> > >>>
> > >>> pool_t pool = pool_datastack_create();
> > >>>
> > >>> char* s1 = "test\001n\001rtest";
> > >>> char* s2 = t_str_tabunescape(s1);
> > >>> char* s3 = p_strdup(pool, s2);
> > >>>
> > >>> printf("1>%s\n", s1);
> > >>> printf("2>%s\n", s2);
> > >>> printf("3>%s\n", s3); // all three output the string with NL and CR
> > >>>
> > >>> Maybe I am missing a function call in the process or maybe the issue is
> > >>> related to the way unescaping is performed in the async callback
> > >>> function in "dict-client.c", or maybe even some other edge case.
> > >>>
> > >>> Finally, I was able to run the first snippet without bug by removing the
> > >>> string duplication in "t_str_tabunescape" (which I realize is not a
> > >>> proper solution), or by explicitely using the following pool:
> > >>>
> > >>> return str_tabunescape(p_strdup(pool_datastack_create(), str));
> > >>>
> > >>>
> > >>> Hope this helps.
> > >>>
> > >>> kaiyou
> > >>>
> > >>>
> > >>> On 10/17/2016 07:51 AM, Aki Tuomi wrote:
> > Hi!
> >
> > This does sound like a bug, we'll have look.
> >
> > Aki
> >
> >
> > On 17.10.2016 01:26, Pierre Jaury wrote:
> > > I dived a little bit further into the rabbit hole, up to the point
> > > where
> > > debugging has become unpracticle but I still haven't found the root
> > > cause for sure.
> > >
> > > I read most of the code for "p_strdup" based on datastack memory pools
> > > (which are used for dictionary lookups both with doveadm and by
> > > extdata)
> > > and it seems ok. Still, after "t_malloc_real" is called in
> > > "t_malloc0",
> > > the allocated buffer has the same address as the source string.
> > >
> > > The only sensible explanation I can come up with is that during
> > > unescaping, strings are not allocated properly, leading to the memory
> > > pool reusing the string address and zeroing it in the process before
> > > the
> > > string copy operation.
> > >
> > > I will follow on this path tomorrow, any lead is more than welcome.
> > >
> > > kaiyou.
> > >
> > > On 10/16/2016 11:16 PM, Pierre Jaury wrote:
> > >> Hello,
> > >>
> > >> I am using a dict proxy for my sieve extdata plugin to access some
> > >> fields from an SQLite database (autoreply text and other
> > >> database-configured items).
> > >>
> > >> All tests are pe
Re: Dict proxy client returning empty string instead of multiline string
Oh duh, it used datastack pool. Try again with the attached patch? Please
remember to clear the previous one out before trying.
Aki
>
> The trace is missing some symbols, I will debug tomorrow and see where
> the call comes from exactly.
>
> Regards,
>
>
> On 10/17/2016 06:23 PM, Aki Tuomi wrote:
> > Hi!
> >
> > Looking at the code, I think the bug is that it just copies pointer to the
> > value, it should, instead, duplicate the value to some memory region. Can
> > you see if this following patch fixes it?
> >
> > Aki
> >
> >> On October 17, 2016 at 4:14 PM Pierre Jaury wrote:
> >>
> >>
> >> Okay, it seems to me that the bug is due to "t_str_tabunescape" using
> >> the unsafe datastack ("t_strdup_noconst") while the string is actually
> >> returned in an async callback.
> >>
> >> Before it is handled by "client_dict_lookup", "client_dict_wait"
> >> actually fires some IO loops that are more than likely to call "t_pop"
> >> and free/flush the result string (I checked, it does call "t_pop" a
> >> couple times indeed). Maybe it would be safer to use a standard
> >> datastack pool when unescaping a string in that context.
> >>
> >> I could work on a patch that would:
> >>
> >> - add an optional "pool" attribute to the "client_dict_cmd" structure,
> >> - pass the pool to the async lookup function,
> >> - use the pool when escaping strings that should survive the callback
> >>chain
> >>
> >> What do you think?
> >>
> >> Regards,
> >> kaiyou
> >>
> >>
> >> On 10/17/2016 09:52 AM, Pierre Jaury wrote:
> >>> While trying to isolate properly and reproduce, I was able to trigger
> >>> the same bug with the following code:
> >>>
> >>> struct dict *dict;
> >>> char* dict_uri = "proxy::sieve";
> >>> char* key = "priv/key";
> >>> char* username = "ad...@domain.tld";
> >>> char* value, error;
> >>>
> >>> dict_drivers_register_builtin();
> >>> dict_init(dict_uri, DICT_DATA_TYPE_STRING, username,
> >>> doveadm_settings->base_dir, &dict, &error);
> >>> dict_lookup(dict, pool_datastack_create(), key, &value);
> >>> printf(">%s\n", value); // outputs an empty string
> >>> dict_deinit(&dict);
> >>>
> >>> I trimmed it to the bare minimal string manipulation functions involved
> >>> but cannot reproduce in that case:
> >>>
> >>> pool_t pool = pool_datastack_create();
> >>>
> >>> char* s1 = "test\001n\001rtest";
> >>> char* s2 = t_str_tabunescape(s1);
> >>> char* s3 = p_strdup(pool, s2);
> >>>
> >>> printf("1>%s\n", s1);
> >>> printf("2>%s\n", s2);
> >>> printf("3>%s\n", s3); // all three output the string with NL and CR
> >>>
> >>> Maybe I am missing a function call in the process or maybe the issue is
> >>> related to the way unescaping is performed in the async callback
> >>> function in "dict-client.c", or maybe even some other edge case.
> >>>
> >>> Finally, I was able to run the first snippet without bug by removing the
> >>> string duplication in "t_str_tabunescape" (which I realize is not a
> >>> proper solution), or by explicitely using the following pool:
> >>>
> >>> return str_tabunescape(p_strdup(pool_datastack_create(), str));
> >>>
> >>>
> >>> Hope this helps.
> >>>
> >>> kaiyou
> >>>
> >>>
> >>> On 10/17/2016 07:51 AM, Aki Tuomi wrote:
> Hi!
>
> This does sound like a bug, we'll have look.
>
> Aki
>
>
> On 17.10.2016 01:26, Pierre Jaury wrote:
> > I dived a little bit further into the rabbit hole, up to the point where
> > debugging has become unpracticle but I still haven't found the root
> > cause for sure.
> >
> > I read most of the code for "p_strdup" based on datastack memory pools
> > (which are used for dictionary lookups both with doveadm and by extdata)
> > and it seems ok. Still, after "t_malloc_real" is called in "t_malloc0",
> > the allocated buffer has the same address as the source string.
> >
> > The only sensible explanation I can come up with is that during
> > unescaping, strings are not allocated properly, leading to the memory
> > pool reusing the string address and zeroing it in the process before the
> > string copy operation.
> >
> > I will follow on this path tomorrow, any lead is more than welcome.
> >
> > kaiyou.
> >
> > On 10/16/2016 11:16 PM, Pierre Jaury wrote:
> >> Hello,
> >>
> >> I am using a dict proxy for my sieve extdata plugin to access some
> >> fields from an SQLite database (autoreply text and other
> >> database-configured items).
> >>
> >> All tests are performed against version 2.2.25.
> >>
> >> $ dovecot --version
> >> 2.2.25 (7be1766)
> >>
> >> My configuration looks like:
> >>
> >> dict {
> >> sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict
> >> }
> >>
> >> [...]
> >> sieve_extdata_dict_uri = proxy::sieve
> >>
> >> I am able to read pretty much any attribute without any issue, except
> >> when t
Re: Dict proxy client returning empty string instead of multiline string
Thanks for your help, indeed duplicating the result sounds cleaner than
duplicating before escaping. However, you patch still fails, when
allocating in "pool_data_stack_malloc" this time. I get the following
stack trace:
Panic: pool_data_stack_malloc(): stack frame changed
Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x91662)
[0x7f4106ba1662] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x916e9)
[0x7f4106ba16e9] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0)
[0x7f4106b3aae1] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xac14e)
[0x7f4106bbc14e] ->
/usr/local/lib/dovecot/libdovecot.so.0(p_strdup+0x28) [0x7f4106bcbd88]
-> /usr/local/lib/dovecot/libdovecot.so.0(+0x5ef0c) [0x7f4106b6ef0c] ->
/usr/local/lib/dovecot/libdovecot.so.0(+0x5f30f) [0x7f4106b6f30f] ->
/usr/local/lib/dovecot/libdovecot.so.0(+0x5ebef) [0x7f4106b6ebef] ->
/usr/local/lib/dovecot/libdovecot.so.0(connection_input_default+0xb1)
[0x7f4106b9ee81] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x5f)
[0x7f4106bb5fdf] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109)
[0x7f4106bb7499] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25)
[0x7f4106bb6085] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7f4106bb6228] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x5ef6c)
[0x7f4106b6ef6c] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x5fafc)
[0x7f4106b6fafc] ->
/home/kaiyou/projects/dovecot/src/doveadm/.libs/doveadm(+0x1b4b0)
[0x55f11cd3e4b0] ->
/home/kaiyou/projects/dovecot/src/doveadm/.libs/doveadm(doveadm_cmd_ver2_to_cmd_wrapper+0x23a)
[0x55f11cd5ae6a] ->
/home/kaiyou/projects/dovecot/src/doveadm/.libs/doveadm(doveadm_cmd_run_ver2+0x555)
[0x55f11cd5bc15] ->
/home/kaiyou/projects/dovecot/src/doveadm/.libs/doveadm(doveadm_cmd_try_run_ver2+0x37)
[0x55f11cd5bc67] ->
/home/kaiyou/projects/dovecot/src/doveadm/.libs/doveadm(main+0x1da)
[0x55f11cd3c31a] -> /usr/lib/libc.so.6(__libc_start_main+0xf1)
[0x7f4106792291] ->
/home/kaiyou/projects/dovecot/src/doveadm/.libs/doveadm(_start+0x2a)
[0x55f11cd3c6fa]
The trace is missing some symbols, I will debug tomorrow and see where
the call comes from exactly.
Regards,
On 10/17/2016 06:23 PM, Aki Tuomi wrote:
> Hi!
>
> Looking at the code, I think the bug is that it just copies pointer to the
> value, it should, instead, duplicate the value to some memory region. Can you
> see if this following patch fixes it?
>
> Aki
>
>> On October 17, 2016 at 4:14 PM Pierre Jaury wrote:
>>
>>
>> Okay, it seems to me that the bug is due to "t_str_tabunescape" using
>> the unsafe datastack ("t_strdup_noconst") while the string is actually
>> returned in an async callback.
>>
>> Before it is handled by "client_dict_lookup", "client_dict_wait"
>> actually fires some IO loops that are more than likely to call "t_pop"
>> and free/flush the result string (I checked, it does call "t_pop" a
>> couple times indeed). Maybe it would be safer to use a standard
>> datastack pool when unescaping a string in that context.
>>
>> I could work on a patch that would:
>>
>> - add an optional "pool" attribute to the "client_dict_cmd" structure,
>> - pass the pool to the async lookup function,
>> - use the pool when escaping strings that should survive the callback
>>chain
>>
>> What do you think?
>>
>> Regards,
>> kaiyou
>>
>>
>> On 10/17/2016 09:52 AM, Pierre Jaury wrote:
>>> While trying to isolate properly and reproduce, I was able to trigger
>>> the same bug with the following code:
>>>
>>> struct dict *dict;
>>> char* dict_uri = "proxy::sieve";
>>> char* key = "priv/key";
>>> char* username = "ad...@domain.tld";
>>> char* value, error;
>>>
>>> dict_drivers_register_builtin();
>>> dict_init(dict_uri, DICT_DATA_TYPE_STRING, username,
>>> doveadm_settings->base_dir, &dict, &error);
>>> dict_lookup(dict, pool_datastack_create(), key, &value);
>>> printf(">%s\n", value); // outputs an empty string
>>> dict_deinit(&dict);
>>>
>>> I trimmed it to the bare minimal string manipulation functions involved
>>> but cannot reproduce in that case:
>>>
>>> pool_t pool = pool_datastack_create();
>>>
>>> char* s1 = "test\001n\001rtest";
>>> char* s2 = t_str_tabunescape(s1);
>>> char* s3 = p_strdup(pool, s2);
>>>
>>> printf("1>%s\n", s1);
>>> printf("2>%s\n", s2);
>>> printf("3>%s\n", s3); // all three output the string with NL and CR
>>>
>>> Maybe I am missing a function call in the process or maybe the issue is
>>> related to the way unescaping is performed in the async callback
>>> function in "dict-client.c", or maybe even some other edge case.
>>>
>>> Finally, I was able to run the first snippet without bug by removing the
>>> string duplication in "t_str_tabunescape" (which I realize is not a
>>> proper solution), or by explicitely using the following pool:
>>>
>>> return str_tabunescape(p_strdup(pool_datastack_create(), str));
>>>
>>>
>>> Hope this helps.
>>>
>>> kaiyou
>>>
>>>
>>> On 10/17/2016 07:51 AM, Aki Tuomi wrote
Re: Dict proxy client returning empty string instead of multiline string
Hi!
Looking at the code, I think the bug is that it just copies pointer to the
value, it should, instead, duplicate the value to some memory region. Can you
see if this following patch fixes it?
Aki
> On October 17, 2016 at 4:14 PM Pierre Jaury wrote:
>
>
> Okay, it seems to me that the bug is due to "t_str_tabunescape" using
> the unsafe datastack ("t_strdup_noconst") while the string is actually
> returned in an async callback.
>
> Before it is handled by "client_dict_lookup", "client_dict_wait"
> actually fires some IO loops that are more than likely to call "t_pop"
> and free/flush the result string (I checked, it does call "t_pop" a
> couple times indeed). Maybe it would be safer to use a standard
> datastack pool when unescaping a string in that context.
>
> I could work on a patch that would:
>
> - add an optional "pool" attribute to the "client_dict_cmd" structure,
> - pass the pool to the async lookup function,
> - use the pool when escaping strings that should survive the callback
>chain
>
> What do you think?
>
> Regards,
> kaiyou
>
>
> On 10/17/2016 09:52 AM, Pierre Jaury wrote:
> > While trying to isolate properly and reproduce, I was able to trigger
> > the same bug with the following code:
> >
> > struct dict *dict;
> > char* dict_uri = "proxy::sieve";
> > char* key = "priv/key";
> > char* username = "ad...@domain.tld";
> > char* value, error;
> >
> > dict_drivers_register_builtin();
> > dict_init(dict_uri, DICT_DATA_TYPE_STRING, username,
> > doveadm_settings->base_dir, &dict, &error);
> > dict_lookup(dict, pool_datastack_create(), key, &value);
> > printf(">%s\n", value); // outputs an empty string
> > dict_deinit(&dict);
> >
> > I trimmed it to the bare minimal string manipulation functions involved
> > but cannot reproduce in that case:
> >
> > pool_t pool = pool_datastack_create();
> >
> > char* s1 = "test\001n\001rtest";
> > char* s2 = t_str_tabunescape(s1);
> > char* s3 = p_strdup(pool, s2);
> >
> > printf("1>%s\n", s1);
> > printf("2>%s\n", s2);
> > printf("3>%s\n", s3); // all three output the string with NL and CR
> >
> > Maybe I am missing a function call in the process or maybe the issue is
> > related to the way unescaping is performed in the async callback
> > function in "dict-client.c", or maybe even some other edge case.
> >
> > Finally, I was able to run the first snippet without bug by removing the
> > string duplication in "t_str_tabunescape" (which I realize is not a
> > proper solution), or by explicitely using the following pool:
> >
> > return str_tabunescape(p_strdup(pool_datastack_create(), str));
> >
> >
> > Hope this helps.
> >
> > kaiyou
> >
> >
> > On 10/17/2016 07:51 AM, Aki Tuomi wrote:
> >> Hi!
> >>
> >> This does sound like a bug, we'll have look.
> >>
> >> Aki
> >>
> >>
> >> On 17.10.2016 01:26, Pierre Jaury wrote:
> >>> I dived a little bit further into the rabbit hole, up to the point where
> >>> debugging has become unpracticle but I still haven't found the root
> >>> cause for sure.
> >>>
> >>> I read most of the code for "p_strdup" based on datastack memory pools
> >>> (which are used for dictionary lookups both with doveadm and by extdata)
> >>> and it seems ok. Still, after "t_malloc_real" is called in "t_malloc0",
> >>> the allocated buffer has the same address as the source string.
> >>>
> >>> The only sensible explanation I can come up with is that during
> >>> unescaping, strings are not allocated properly, leading to the memory
> >>> pool reusing the string address and zeroing it in the process before the
> >>> string copy operation.
> >>>
> >>> I will follow on this path tomorrow, any lead is more than welcome.
> >>>
> >>> kaiyou.
> >>>
> >>> On 10/16/2016 11:16 PM, Pierre Jaury wrote:
> Hello,
>
> I am using a dict proxy for my sieve extdata plugin to access some
> fields from an SQLite database (autoreply text and other
> database-configured items).
>
> All tests are performed against version 2.2.25.
>
> $ dovecot --version
> 2.2.25 (7be1766)
>
> My configuration looks like:
>
> dict {
> sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict
> }
>
> [...]
> sieve_extdata_dict_uri = proxy::sieve
>
> I am able to read pretty much any attribute without any issue, except
> when the value contains a special character like "\r" or "\n". By using
> the doveadm dict client, I narrowed it down to the dictionary management
> part (either server, protocol or client).
>
> I was suspecting escaping functions from "lib/strescape.c" (mostly
> str_tabescape and its counterpart, used by "lib-dict/client.c"), so I
> monitored socket communications. It seems that escaping is done properly
> on the server and the socket is not an issue either.
>
> The following strace dump results from running "doveadm dict get"
> again
sieve duplicate locking
Hi! Does the duplicate sieve plugin do any locking to avoid duplicate parallel delivery of the same message? I sometimes experience duplicate mail delivery of messages with the same message-id, despite the use of a sieve duplicate filter. According to the log files, those messages are delivered in the same second by two parallel dovecot-lda processes. (Duplicate filtering works fine in all other cases.) RFC7352 states that the ID of a message may only be committed to the duplicate tracking list at the _end_ of a successful script execution, which may lead to race conditions. Maybe I am running into this? Is there an easy way to serialize mail delivery using some locking inside sieve? Or do I have to serialize per-user dovecot-lda delivery? Any experiences with that? I am using dovecot-2.2.25 and pidgeonhole-0.4.15. Mail is delivered using postfix-2.10 and dovecot-lda as mailbox_command. Mailbox format is maildir with LAYOUT=fs. Cheers, --leo -- e-mail ::: Leo.Bergolth (at) wu.ac.at fax ::: +43-1-31336-906050 location ::: IT-Services | Vienna University of Economics | Austria
Re: Dict proxy client returning empty string instead of multiline string
While trying to isolate properly and reproduce, I was able to trigger
the same bug with the following code:
struct dict *dict;
char* dict_uri = "proxy::sieve";
char* key = "priv/key";
char* username = "ad...@domain.tld";
char* value, error;
dict_drivers_register_builtin();
dict_init(dict_uri, DICT_DATA_TYPE_STRING, username,
doveadm_settings->base_dir, &dict, &error);
dict_lookup(dict, pool_datastack_create(), key, &value);
printf(">%s\n", value); // outputs an empty string
dict_deinit(&dict);
I trimmed it to the bare minimal string manipulation functions involved
but cannot reproduce in that case:
pool_t pool = pool_datastack_create();
char* s1 = "test\001n\001rtest";
char* s2 = t_str_tabunescape(s1);
char* s3 = p_strdup(pool, s2);
printf("1>%s\n", s1);
printf("2>%s\n", s2);
printf("3>%s\n", s3); // all three output the string with NL and CR
Maybe I am missing a function call in the process or maybe the issue is
related to the way unescaping is performed in the async callback
function in "dict-client.c", or maybe even some other edge case.
Finally, I was able to run the first snippet without bug by removing the
string duplication in "t_str_tabunescape" (which I realize is not a
proper solution), or by explicitely using the following pool:
return str_tabunescape(p_strdup(pool_datastack_create(), str));
Hope this helps.
kaiyou
On 10/17/2016 07:51 AM, Aki Tuomi wrote:
> Hi!
>
> This does sound like a bug, we'll have look.
>
> Aki
>
>
> On 17.10.2016 01:26, Pierre Jaury wrote:
>> I dived a little bit further into the rabbit hole, up to the point where
>> debugging has become unpracticle but I still haven't found the root
>> cause for sure.
>>
>> I read most of the code for "p_strdup" based on datastack memory pools
>> (which are used for dictionary lookups both with doveadm and by extdata)
>> and it seems ok. Still, after "t_malloc_real" is called in "t_malloc0",
>> the allocated buffer has the same address as the source string.
>>
>> The only sensible explanation I can come up with is that during
>> unescaping, strings are not allocated properly, leading to the memory
>> pool reusing the string address and zeroing it in the process before the
>> string copy operation.
>>
>> I will follow on this path tomorrow, any lead is more than welcome.
>>
>> kaiyou.
>>
>> On 10/16/2016 11:16 PM, Pierre Jaury wrote:
>>> Hello,
>>>
>>> I am using a dict proxy for my sieve extdata plugin to access some
>>> fields from an SQLite database (autoreply text and other
>>> database-configured items).
>>>
>>> All tests are performed against version 2.2.25.
>>>
>>> $ dovecot --version
>>> 2.2.25 (7be1766)
>>>
>>> My configuration looks like:
>>>
>>> dict {
>>> sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict
>>> }
>>>
>>> [...]
>>> sieve_extdata_dict_uri = proxy::sieve
>>>
>>> I am able to read pretty much any attribute without any issue, except
>>> when the value contains a special character like "\r" or "\n". By using
>>> the doveadm dict client, I narrowed it down to the dictionary management
>>> part (either server, protocol or client).
>>>
>>> I was suspecting escaping functions from "lib/strescape.c" (mostly
>>> str_tabescape and its counterpart, used by "lib-dict/client.c"), so I
>>> monitored socket communications. It seems that escaping is done properly
>>> on the server and the socket is not an issue either.
>>>
>>> The following strace dump results from running "doveadm dict get"
>>> against the dict socket:
>>>
>>> connect(8, {sa_family=AF_UNIX, sun_path="..."}, 110) = 0
>>> fstat(8, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
>>> [...]
>>> write(8, "H2\t0\t0\tad...@domain.tld\tsieve\n", 30) = 30
>>> [...]
>>> read(8, "Otest\1r\1ntest\n", 8192) = 14
>>>
>>> Indeed "\1r" and "\1n" are the escape sequences used by
>>> "lib/strescape.c". I went deeped and debugged the call to "dict_lookup"
>>> performed by doveadm. Indeed the client gets the proper string from the
>>> socket and to my surprise, it is properly unescaped.
>>>
>>> Then, in "client_dict_lookup" ("lib-dict/dict-client.c"), the call to
>>> "p_strdup" returns an empty string (null byte set at the target address).
>>>
>>> Before the call to the dict "->lookup" attribute (client_dict_lookup):
>>>
>>>RAX: 0x773a37c0 (push r14)
>>>RBX: 0x6831b8 ("priv/reply_body")
>>>RCX: 0x7fffe240 --> 0x682a60 --> 0x6831b8 ("priv/reply_body")
>>>RDX: 0x6831b8 ("priv/reply_body")
>>>RSI: 0x683288 --> 0x77653120 --> 0x773ea620 ([...])
>>>RDI: 0x690ad0 --> 0x77400713 --> 0x75250079786f7270 ('proxy')
>>>
>>>0x773a1f10 : movrcx,r11 (value_r)
>>>0x773a1f13 : movrdx,r8 (key)
>>>0x773a1f16 : movrsi,r10 (pool)
>>>0x773a1f19 : movrdi,r9 (dict)
>>>0x773a1f1c : addrsp,0x8
>>> => 0x773a1f20 : jmprax
>>>
>>> Before the call to p_strdup in "client_dict_lookup":
>>>
>>>RSI: 0x6832d8 ("test\
Re: Dict proxy client returning empty string instead of multiline string
Okay, it seems to me that the bug is due to "t_str_tabunescape" using
the unsafe datastack ("t_strdup_noconst") while the string is actually
returned in an async callback.
Before it is handled by "client_dict_lookup", "client_dict_wait"
actually fires some IO loops that are more than likely to call "t_pop"
and free/flush the result string (I checked, it does call "t_pop" a
couple times indeed). Maybe it would be safer to use a standard
datastack pool when unescaping a string in that context.
I could work on a patch that would:
- add an optional "pool" attribute to the "client_dict_cmd" structure,
- pass the pool to the async lookup function,
- use the pool when escaping strings that should survive the callback
chain
What do you think?
Regards,
kaiyou
On 10/17/2016 09:52 AM, Pierre Jaury wrote:
> While trying to isolate properly and reproduce, I was able to trigger
> the same bug with the following code:
>
> struct dict *dict;
> char* dict_uri = "proxy::sieve";
> char* key = "priv/key";
> char* username = "ad...@domain.tld";
> char* value, error;
>
> dict_drivers_register_builtin();
> dict_init(dict_uri, DICT_DATA_TYPE_STRING, username,
> doveadm_settings->base_dir, &dict, &error);
> dict_lookup(dict, pool_datastack_create(), key, &value);
> printf(">%s\n", value); // outputs an empty string
> dict_deinit(&dict);
>
> I trimmed it to the bare minimal string manipulation functions involved
> but cannot reproduce in that case:
>
> pool_t pool = pool_datastack_create();
>
> char* s1 = "test\001n\001rtest";
> char* s2 = t_str_tabunescape(s1);
> char* s3 = p_strdup(pool, s2);
>
> printf("1>%s\n", s1);
> printf("2>%s\n", s2);
> printf("3>%s\n", s3); // all three output the string with NL and CR
>
> Maybe I am missing a function call in the process or maybe the issue is
> related to the way unescaping is performed in the async callback
> function in "dict-client.c", or maybe even some other edge case.
>
> Finally, I was able to run the first snippet without bug by removing the
> string duplication in "t_str_tabunescape" (which I realize is not a
> proper solution), or by explicitely using the following pool:
>
> return str_tabunescape(p_strdup(pool_datastack_create(), str));
>
>
> Hope this helps.
>
> kaiyou
>
>
> On 10/17/2016 07:51 AM, Aki Tuomi wrote:
>> Hi!
>>
>> This does sound like a bug, we'll have look.
>>
>> Aki
>>
>>
>> On 17.10.2016 01:26, Pierre Jaury wrote:
>>> I dived a little bit further into the rabbit hole, up to the point where
>>> debugging has become unpracticle but I still haven't found the root
>>> cause for sure.
>>>
>>> I read most of the code for "p_strdup" based on datastack memory pools
>>> (which are used for dictionary lookups both with doveadm and by extdata)
>>> and it seems ok. Still, after "t_malloc_real" is called in "t_malloc0",
>>> the allocated buffer has the same address as the source string.
>>>
>>> The only sensible explanation I can come up with is that during
>>> unescaping, strings are not allocated properly, leading to the memory
>>> pool reusing the string address and zeroing it in the process before the
>>> string copy operation.
>>>
>>> I will follow on this path tomorrow, any lead is more than welcome.
>>>
>>> kaiyou.
>>>
>>> On 10/16/2016 11:16 PM, Pierre Jaury wrote:
Hello,
I am using a dict proxy for my sieve extdata plugin to access some
fields from an SQLite database (autoreply text and other
database-configured items).
All tests are performed against version 2.2.25.
$ dovecot --version
2.2.25 (7be1766)
My configuration looks like:
dict {
sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict
}
[...]
sieve_extdata_dict_uri = proxy::sieve
I am able to read pretty much any attribute without any issue, except
when the value contains a special character like "\r" or "\n". By using
the doveadm dict client, I narrowed it down to the dictionary management
part (either server, protocol or client).
I was suspecting escaping functions from "lib/strescape.c" (mostly
str_tabescape and its counterpart, used by "lib-dict/client.c"), so I
monitored socket communications. It seems that escaping is done properly
on the server and the socket is not an issue either.
The following strace dump results from running "doveadm dict get"
against the dict socket:
connect(8, {sa_family=AF_UNIX, sun_path="..."}, 110) = 0
fstat(8, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
[...]
write(8, "H2\t0\t0\tad...@domain.tld\tsieve\n", 30) = 30
[...]
read(8, "Otest\1r\1ntest\n", 8192) = 14
Indeed "\1r" and "\1n" are the escape sequences used by
"lib/strescape.c". I went deeped and debugged the call to "dict_lookup"
performed by doveadm. Indeed the client gets the proper string from the
socke
Re: logging TLS SNI hostname
> On Oct 17, 2016, at 2:41 AM, Arkadiusz Miśkiewicz wrote: > > On Monday 30 of May 2016, Arkadiusz Miśkiewicz wrote: >> Is there a way to log SNI hostname used in TLS session? Info is there in >> SSL_CTX_set_tlsext_servername_callback, dovecot copies it to >> ssl_io->host. >> >> Unfortunately I don't see it expanded to any variables ( >> http://wiki.dovecot.org/Variables ). Please consider this to be a feature >> request. >> >> The goal is to be able to see which hostname client used like: >> >> May 30 08:21:19 xxx dovecot: pop3-login: Login: user=, method=PLAIN, >> rip=1.1.1.1, lip=2.2.2.2, mpid=17135, TLS, SNI=pop3.somehost.org, >> session= > > Dear dovecot team, would be possible to add such variable ^ ? > > That would be neat feature because server operator would know what hostname > client uses to connect to server (which is really usefull in case of many > hostnames pointing to single IP). I’d love to be able to use this SNI domain name in the Dovecot IMAP proxy for use in the SQL password_query. This would allow the proxy to support multiple IMAP server domains each with their own set of users. And, it would save me money by using only the IP of the proxy for all the IMAP server domains instead of giving each domain a unique IP. Kevin
Re: Massive LMTP Problems with dovecot
* Ralf Hildebrandt : > > It seems to loop in sha1_loop & hash_format_loop > > The problem occurs in both 2.3 and 2.2 (I just updated to 2.3 to check). I'm seeing the first occurence of that problem on the 10th of october! I was using (prior to the 10th) : 2.3.0~alpha0-1~auto+371 On the 10th I upgraded (16:04) to: 2.3.0~alpha0-1~auto+376 I'd think the change must have been introduced between 371 and 376 :) I then went back to, issues went away: 2.2.25-1~auto+49 and the issues reappeared with 2.2.25-1~auto+57 Does that help? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Massive LMTP Problems with dovecot
* Ralf Hildebrandt : > * Ralf Hildebrandt : > > I attached gdb top a long running LMTP process: > > > > #0 sha1_loop (ctxt=0x7f3b1a4d7fa0, input=0x7f3b1a524860, len=0) at > > sha1.c:216 > >input_c = 0x7f3b1a524860 "\211PNG\r\n\032\n" > >gaplen = > >gapstart = > >off = 0 > >copysiz = > > > > #1 0x7f3b19195b29 in hash_format_loop (format=, > > data=0x7f3b1a524860, size=0) at hash-format.c:150 > >list = 0x7f3b1a4d7f80 > > It seems to loop in sha1_loop & hash_format_loop The problem occurs in both 2.3 and 2.2 (I just updated to 2.3 to check). -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Massive LMTP Problems with dovecot
* Ralf Hildebrandt : > I attached gdb top a long running LMTP process: > > #0 sha1_loop (ctxt=0x7f3b1a4d7fa0, input=0x7f3b1a524860, len=0) at sha1.c:216 >input_c = 0x7f3b1a524860 "\211PNG\r\n\032\n" >gaplen = >gapstart = >off = 0 >copysiz = > > #1 0x7f3b19195b29 in hash_format_loop (format=, > data=0x7f3b1a524860, size=0) at hash-format.c:150 >list = 0x7f3b1a4d7f80 It seems to loop in sha1_loop & hash_format_loop -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Massive LMTP Problems with dovecot
I attached gdb top a long running LMTP process:
#0 sha1_loop (ctxt=0x7f3b1a4d7fa0, input=0x7f3b1a524860, len=0) at sha1.c:216
input_c = 0x7f3b1a524860 "\211PNG\r\n\032\n"
gaplen =
gapstart =
off = 0
copysiz =
#1 0x7f3b19195b29 in hash_format_loop (format=,
data=0x7f3b1a524860, size=0) at hash-format.c:150
list = 0x7f3b1a4d7f80
#2 0x7f3b1916f5b8 in astream_decode_base64(astream=0x7f3b1a4cb030) at
istream-attachment-extractor.c:388
part = 0x7f3b1a4cb228
output = 0x7f3b1a5288c0
size = 0
buf = 0x7f3b1a528070
outfd = 24
extra_buf = 0x0
data = 0x7f3b1a52484e "iW"
ret =
input = 0x7f3b1a528530
base64_input = 0x7f3b1a5286f0
failed = false
#3 astream_part_finish (error_r=0x7ffc00bc2518, astream=) at
istream-attachment-extractor.c:485
info = {hash = 0x7f3b1a414c08
"ebd67eb141828144e22a6123b7c9e8ce3401a0db", start_offset = 41786, encoded_size
= 331456, base64_blocks_per_line = 0,
base64_have_crlf = false, part = 0x0}
digest_str = 0x7f3b1a414bd0
data = 0x0
ret = 0
input = 0x7f3b1a5288c0
output = 0x7f3b1a52484e
size = 139891821412464
#4 astream_end_of_part (astream=astream@entry=0x7f3b1a4cb030,
error_r=error_r@entry=0x7ffc00bc2518) at istream-attachment-extractor.c:571
part = 0x7f3b1a4cb228
old_size = 0
ret = 0
#5 0x7f3b1916fbdb in astream_read_next (retry_r=,
astream=0x7f3b1a4cb030) at istream-attachment-extractor.c:633
stream = 0x7f3b1a4cb030
block = {part = 0x7f3b1a4d8770, hdr = 0x0,
data = 0x7f3b19bf91e1
"\n--_008_VI1PR02MB139090A81DDBB9A3973922378AD00VI1PR02MB1390eurp_\nContent-Type:
image/png; name=\"image013.png\"\nContent-Description:
image013.png\nContent-Disposition: inline; filename=\"image013.png\"; si"...,
size = 64}
new_size =
ret =
old_size = 0
error = 0xcf803a94af74800
#6 i_stream_attachment_extractor_read (stream=0x7f3b1a4cb030) at
istream-attachment-extractor.c:668
astream = 0x7f3b1a4cb030
retry = false
ret =
#7 0x7f3b1919a1c3 in i_stream_read (stream=0x7f3b1a4cb0a0) at istream.c:174
_stream = 0x7f3b1a4cb030
old_size = 0
ret =
__FUNCTION__ = "i_stream_read"
#8 0x7f3b194c2c0b in index_attachment_save_continue (ctx=0x7f3b1a4c59a0)
at index-attachment.c:218
storage = 0x7f3b1a4907a0
attach = 0x7f3b1a4d8360
data =
size = 1
ret =
#9 0x7f3b1945dcd2 in mailbox_save_continue (ctx=ctx@entry=0x7f3b1a4c59a0)
at mail-storage.c:2113
_data_stack_cur_id = 4
ret =
#10 0x7f3b194540ee in mail_storage_try_copy (mail=0x7ffc00bc2658,
_ctx=0x7ffc00bc2658) at mail-copy.c:81
ctx = 0x7f3b1a4c59a0
pmail = 0x7ffc00bc2658
ret =
input = 0x7f3b1a4c4140
#11 mail_storage_copy (ctx=ctx@entry=0x7f3b1a4c59a0,
mail=mail@entry=0x7f3b1a48b770) at mail-copy.c:107
__FUNCTION__ = "mail_storage_copy"
#12 0x7f3b19474806 in mdbox_copy (_ctx=0x7f3b1a4c59a0, mail=0x7f3b1a48b770)
at mdbox-save.c:468
ctx = 0x7f3b1a4c59a0
save_mail = 0x7f3b1a48b770
src_mbox =
rec = {map_uid = 440968640, save_date = 32571}
guid_data = 0x7f3b1a4c59a0
wanted_guid = "p\245H\032;\177\000\000\267\225u\031;\177\000"
#13 0x7f3b180bd2f1 in fts_copy (ctx=0x7f3b1a4c59a0, mail=0x7f3b1a48b770) at
fts-storage.c:735
ft = 0x7f3b1a4c4c10
fbox =
#14 0x7f3b1945e16d in mailbox_copy_int (_ctx=_ctx@entry=0x7ffc00bc27a0,
mail=0x7f3b1a48b770) at mail-storage.c:2244
_data_stack_cur_id = 3
ctx = 0x7f3b1a4c59a0
t = 0x7f3b1a4c4c40
keywords = 0x0
pvt_flags = 0
backend_mail = 0x7f3b1a48b770
ret =
__FUNCTION__ = "mailbox_copy_int"
#15 0x7f3b1945e3e2 in mailbox_save_using_mail
(_ctx=_ctx@entry=0x7ffc00bc27a0, mail=) at mail-storage.c:2295
ctx =
__FUNCTION__ = "mailbox_save_using_mail"
#16 0x7f3b19759789 in mail_deliver_save (ctx=ctx@entry=0x7ffc00bc2930,
mailbox=, flags=flags@entry=0, keywords=keywords@entry=0x0,
storage_r=storage_r@entry=0x7ffc00bc2908) at mail-deliver.c:383
open_ctx = {user = 0x7f3b1a470e40, lda_mailbox_autocreate = false,
lda_mailbox_autosubscribe = false}
box = 0x7f3b1a4bb610
trans_flags =
t = 0x7f3b1a4c4c40
save_ctx = 0x0
headers_ctx = 0x0
kw = 0x0
error = MAIL_ERROR_NONE
mailbox_name = 0x7f3b19b8f258 "INBOX"
errstr = 0x0
guid = 0x0
changes = {pool = 0x7f3b1a4502b0, uid_validity = 440869512, saved_uids =
{arr = {buffer = 0x7f3b1995e400
, element_size = 1476711940}, v =
0x7f3b1995e400 , v_modifiable = 0x7f3b1995e400
},
ignored_modseq_changes = 440968560, changed = 59, no_read_perm =
Re: Massive LMTP Problems with dovecot
> We'll need the `dovecot -n` output first.
Here we go:
> # 2.2.devel (933d16f): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.devel (63f9b42)
> # OS: Linux 3.13.0-98-generic x86_64 Ubuntu 14.04.5 LTS
> default_vsz_limit = 2 G
> lmtp_user_concurrency_limit = 1
> mail_attachment_dir = /home/copymail/attachments
> mail_location = mdbox:~/mdbox
> mail_plugins = zlib fts fts_lucene
> mdbox_rotate_size = 128 M
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> special_use = \Drafts
> }
> mailbox Junk {
> special_use = \Junk
> }
> mailbox Sent {
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> special_use = \Trash
> }
> prefix =
> }
> passdb {
> args = username_format=%u /etc/dovecot/passwd
> driver = passwd-file
> }
> plugin {
> fts = lucene
> fts_autoindex = yes
> fts_languages = de,en
> fts_lucene = whitespace_chars=@.
> sieve = file:~/sieve;active=~/.dovecot.sieve
> zlib_save = gz
> zlib_save_level = 5
> }
> protocols = " imap lmtp"
> service imap-login {
> inet_listener imap {
> address = 127.0.0.1
> port = 143
> }
> inet_listener imaps {
> port = 993
> ssl = yes
> }
> }
> service lmtp {
> inet_listener lmtp {
> address = 141.42.1.208
> port = 1025
> }
> unix_listener /var/spool/postfix/private/dovecot-lmtp {
> group = postfix
> mode = 0660
> user = postfix
> }
> }
> ssl_ca = /etc/ssl/certs/ca-certificates.crt
> ssl_cert = ssl_cipher_list =
> EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
> ssl_key = # hidden, use -P to show it
> ssl_prefer_server_ciphers = yes
> ssl_protocols = !SSLv2 !SSLv3
> userdb {
> args = username_format=%u /etc/dovecot/passwd
> driver = passwd-file
> }
> protocol lmtp {
> mail_plugins = zlib fts fts_lucene
> }
Ralf Hildebrandt
Geschäftsbereich IT | Abteilung Netzwerk
Charité - Universitätsmedizin Berlin
Campus Benjamin Franklin
Hindenburgdamm 30 | D-12203 Berlin
Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
ralf.hildebra...@charite.de | http://www.charite.de
Re: Massive LMTP Problems with dovecot
Op 10/17/2016 om 3:24 PM schreef Ralf Hildebrandt: > Currently I'm having massive problems with LMTP delivery into dovcot. > dovecot/lmtp processes are piling up, eas using considerable amounts > of CPU: > > # ps auxwww|fgrep dove > > root 20537 0.0 0.0 18124 1196 ?Ss 15:18 0:00 > /usr/sbin/dovecot -c /etc/dovecot/dovecot.conf > dovecot 20541 0.0 0.0 9620 1084 ?S15:18 0:00 dovecot/anvil > root 20542 0.0 0.0 9752 1264 ?S15:18 0:00 dovecot/log > root 20544 0.0 0.0 21168 2276 ?S15:18 0:00 > dovecot/config > copymail 20580 72.8 0.0 39556 7036 ?R15:18 2:00 dovecot/lmtp > dovecot 20582 0.0 0.0 18568 1756 ?S15:18 0:00 dovecot/auth > copymail 20597 77.2 0.0 35688 5136 ?R15:18 2:06 dovecot/lmtp > copymail 20598 39.3 0.0 38060 5596 ?R15:18 1:04 dovecot/lmtp > copymail 20613 62.3 0.0 38036 5600 ?R15:18 1:41 dovecot/lmtp > copymail 20619 56.4 0.0 37732 7448 ?R15:18 1:31 dovecot/lmtp > copymail 20620 75.9 0.0 35872 5336 ?R15:18 2:03 dovecot/lmtp > copymail 20627 37.8 0.0 36480 5892 ?R15:18 1:01 dovecot/lmtp > copymail 20838 60.5 0.0 35640 5036 ?R15:19 0:59 dovecot/lmtp > copymail 20840 66.3 0.0 35920 5296 ?R15:19 1:04 dovecot/lmtp > copymail 20841 66.0 0.0 37456 6852 ?R15:19 1:04 dovecot/lmtp > copymail 20842 64.5 0.0 36424 5808 ?R15:19 1:02 dovecot/lmtp > copymail 20843 67.6 0.0 39612 7064 ?R15:19 1:05 dovecot/lmtp > > doveadm stop won't stop these, I have to use kill -9 on them. > I already tried disabling fts (entirely), still things won't speed up. > > I can't strace: > # strace -p 20841 > Process 20841 attached > > (and that's it) > > # dpkg -l|grep dovecot > ii dovecot-core2:2.2.25-1~auto+57amd64 secure > POP3/IMAP server - core files > ii dovecot-imapd 2:2.2.25-1~auto+57amd64 secure > POP3/IMAP server - IMAP daemon > ii dovecot-lmtpd 2:2.2.25-1~auto+57amd64 secure > POP3/IMAP server - LMTP server > ii dovecot-lucene 2:2.2.25-1~auto+57amd64 secure > POP3/IMAP server - Lucene support > ii dovecot-sieve 2:2.2.25-1~auto+57amd64 secure > POP3/IMAP server - Sieve filters support > > I also tried deleting the mdboxes, that also didn't change anything. > Ideas? We'll need the `dovecot -n` output first. Also, you could attach gdb to one of these processes and find out where it is stuck in what looks to be an infinite loop (produce a backtrace). Regards, Stephan.
Massive LMTP Problems with dovecot
Currently I'm having massive problems with LMTP delivery into dovcot. dovecot/lmtp processes are piling up, eas using considerable amounts of CPU: # ps auxwww|fgrep dove root 20537 0.0 0.0 18124 1196 ?Ss 15:18 0:00 /usr/sbin/dovecot -c /etc/dovecot/dovecot.conf dovecot 20541 0.0 0.0 9620 1084 ?S15:18 0:00 dovecot/anvil root 20542 0.0 0.0 9752 1264 ?S15:18 0:00 dovecot/log root 20544 0.0 0.0 21168 2276 ?S15:18 0:00 dovecot/config copymail 20580 72.8 0.0 39556 7036 ?R15:18 2:00 dovecot/lmtp dovecot 20582 0.0 0.0 18568 1756 ?S15:18 0:00 dovecot/auth copymail 20597 77.2 0.0 35688 5136 ?R15:18 2:06 dovecot/lmtp copymail 20598 39.3 0.0 38060 5596 ?R15:18 1:04 dovecot/lmtp copymail 20613 62.3 0.0 38036 5600 ?R15:18 1:41 dovecot/lmtp copymail 20619 56.4 0.0 37732 7448 ?R15:18 1:31 dovecot/lmtp copymail 20620 75.9 0.0 35872 5336 ?R15:18 2:03 dovecot/lmtp copymail 20627 37.8 0.0 36480 5892 ?R15:18 1:01 dovecot/lmtp copymail 20838 60.5 0.0 35640 5036 ?R15:19 0:59 dovecot/lmtp copymail 20840 66.3 0.0 35920 5296 ?R15:19 1:04 dovecot/lmtp copymail 20841 66.0 0.0 37456 6852 ?R15:19 1:04 dovecot/lmtp copymail 20842 64.5 0.0 36424 5808 ?R15:19 1:02 dovecot/lmtp copymail 20843 67.6 0.0 39612 7064 ?R15:19 1:05 dovecot/lmtp doveadm stop won't stop these, I have to use kill -9 on them. I already tried disabling fts (entirely), still things won't speed up. I can't strace: # strace -p 20841 Process 20841 attached (and that's it) # dpkg -l|grep dovecot ii dovecot-core2:2.2.25-1~auto+57amd64 secure POP3/IMAP server - core files ii dovecot-imapd 2:2.2.25-1~auto+57amd64 secure POP3/IMAP server - IMAP daemon ii dovecot-lmtpd 2:2.2.25-1~auto+57amd64 secure POP3/IMAP server - LMTP server ii dovecot-lucene 2:2.2.25-1~auto+57amd64 secure POP3/IMAP server - Lucene support ii dovecot-sieve 2:2.2.25-1~auto+57amd64 secure POP3/IMAP server - Sieve filters support I also tried deleting the mdboxes, that also didn't change anything. Ideas? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: First steps in Dovecot; IMAP not working
Sorry, my previous message got mangled. I'm re-writing it, quoting manually. I apologize for the traffic. (Sorry I read this list in digest form so frequently I'm half a step behind.) No problem. No, it's quite explicit. User "webuser" has uid/gid = 1001(webuser)/1000(ftpusers). Your mail spool has permission uid/gid = root(0)/mail(8), neither of which allows webuser to write to this mail spool to creates its own mail folder. You're right (I don't have enough Unix habits, it seems...). I couldn't change this user (it must be in the ftpusers group for other purposes), so I tried adding another user for testing mail. "mailtest", the new user, is in group mail(8). In addition, I've added write permission for "others" to /var/mail. Now, I'm trying to send a message to "mailtest" from another, working, e-mail account and nothing happens. This time, "doveadm log errors" is empty. In short, I don't get any error but no mail either. Aki Tuomi replies with several solutions: In your configuration, dovecot uses whatever user/group returned by PAM. Since the webuser has never logged in, it has no directory under /var/mail. If you want, you can a) override mail_uid and mail_gid in userdb/passdb b) pre-create /var/mail/webuser and chown it to webuser:ftpusers c) you can let ftpusers write to /var/mail. Ok, I thought I had to do all of them (and didn't understand step a)). So I've done step c) by allowing everyone write access. Or if you dynamically/frequently onboard mail accounts, and users cannot arbitrarily write into this directory, you can "chmod 1777 /var/mail/" and let dovecot auto-create it (might also want to set "lda_mailbox_autocreate = yes". I've done it right now; same problem. Since "doveadm log errors" returns an empty result, where should I look for the problem? Thank you.
Re: First steps in Dovecot; IMAP not working
Le 14 octobre 2016 à 14:28, Joseph Tam a écrit: (Sorry I read this list in digest form so frequently I'm half a step behind.) No problem. No, it's quite explicit. User "webuser" has uid/gid = 1001(webuser)/1000(ftpusers). Your mail spool has permission uid/gid = root(0)/mail(8), neither of which allows webuser to write to this mail spool to creates its own mail folder. You're right (I don't have enough Unix habits, it seems...). I couldn't change this user (it must be in the ftpusers group for other purposes), so I tried adding another user for testing mail. "mailtest", the new user, is in group mail(8). In addition, I've added write permission for "others" to /var/mail. Now, I'm trying to send a message to "mailtest" from another, working, e-mail account and nothing happens. This time, "doveadm log errors" is empty. In short, I don't get any error but no mail either. Aki Tuomi replies with several solutions: In your configuration, dovecot uses whatever user/group returned by PAM. Since the webuser has never logged in, it has no directory under /var/mail. If you want, you can a) override mail_uid and mail_gid in userdb/passdb b) pre-create /var/mail/webuser and chown it to webuser:ftpusers c) you can let ftpusers write to /var/mail. Ok, I thought I had to do all of them (and didn't understand step a)). So I've done step c) by allowing everyone write access. Or if you dynamically/frequently onboard mail accounts, and users cannot arbitrarily write into this directory, you can "chmod 1777 /var/mail/" and let dovecot auto-create it (might also want to set "lda_mailbox_autocreate = yes". I've done it right now; same problem. Since "doveadm log errors" returns an empty result, where should I look for the problem? Thank you.
Re: First steps in Dovecot; IMAP not working
Le 15 octobre 2016 à 07:35, mick crane a écrit: http://wiki.dovecot.org/FindMailLocation Ok, there are non-standard facts about my user. When I do: eval echo ~webuser I'm getting: /var/www/html/ This is because webuser is for an FTP account in that directory. I'm now trying with a "regular" user. Thank you for the link.
