Re: Search problem

[Aki Tuomi]

bt full would be more useful


---Aki TuomiDovecot oy
 Original message From: Federico Bartolucci  
Date: 28/05/2018  17:24  (GMT+02:00) To: dovecot@dovecot.org Subject: Re: 
Search problem 

Hello,

  

  thank you for your suggestion, I can obtain just this from the
  backtrace:

  


[Thread debugging using
  libthread_db enabled]

  Using host libthread_db library
  "/lib64/libthread_db.so.1".

  Core was generated by `dovecot/imap'.

  Program terminated with signal 6, Aborted.

  #0  0x7f44637bb1f7 in __GI_raise
  (sig=sig@entry=6) at
  ../nptl/sysdeps/unix/sysv/linux/raise.c:56

  56    return INLINE_SYSCALL (tgkill,
  3, pid, selftid, sig);




  What could we understand from this?

  

  -federico

  



Il 09/05/2018 18:52, Reio Remma ha
  scritto:



  
  On 09.05.2018 19:48, Federico
Bartolucci wrote:

  
  

Hello,

  

  when doing a simple search through the lucene indexes in some
  mailboxes (with actually many subfolders) the search
  terminates after a few seconds with no result and the dovecot
  log shows this error:

  

  Fatal: master: service(imap): child 15433 killed with signal 6
  (core not dumped)

  

  

  Any clue about the reasons? the lucene indexes have been
  already rebuilt and look OK.


  

  Probably worth gathering more info by getting
a full backtrace as per instructions in:



https://www.dovecot.org/bugreport.html

  

  Good luck,

  Reio




  

Re: Search problem

[Federico Bartolucci]

Hello,

thank you for your suggestion, I can obtain just this from the backtrace:

/[Thread debugging using libthread_db enabled]/
/Using host libthread_db library "/lib64/libthread_db.so.1"./
/Core was generated by `dovecot/imap'./
/Program terminated with signal 6, Aborted./
/#0  0x7f44637bb1f7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56/
/56    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);/


What could we understand from this?

-federico


Il 09/05/2018 18:52, Reio Remma ha scritto:
> On 09.05.2018 19:48, Federico Bartolucci wrote:
>> Hello,
>>
>> when doing a simple search through the lucene indexes in some
>> mailboxes (with actually many subfolders) the search terminates after
>> a few seconds with no result and the dovecot log shows this error:
>>
>> Fatal: master: service(imap): child 15433 killed with signal 6 (core
>> not dumped)
>>
>>
>> Any clue about the reasons? the lucene indexes have been already
>> rebuilt and look OK.
>
> Probably worth gathering more info by getting a full backtrace as per
> instructions in:
>
> https://www.dovecot.org/bugreport.html
>
> Good luck,
> Reio

Re: SSL error after upgrading to 2.31

[Hauke Fath]

On Mon, 28 May 2018 15:03:29 +0300, Aki Tuomi wrote:
>> Sounds good. How about (re)naming them ssl-{client,server}_ca?
> 
> There is already ssl_client_ca, for verifying clients. ssl_ca verifies
> certs when dovecot is connecting somewhere.

So there's three? I had no idea...

Cheerio,
hauke

-- 
 The ASCII Ribbon CampaignHauke Fath
() No HTML/RTF in emailInstitut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
 Respect for open standards  Ruf +49-6151-16-21344

Re: Problem in Pigeonhole sievec

[Thorsten Hater]

Thanks for the feedback, I simply expected that addresses and
simple strings were treated differently.

Thorsten

On Mon, May 28, 2018 at 1:55 PM, Steffen Kaiser 
wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On Mon, 28 May 2018, Thorsten Hater wrote:
>
> I stumbled upon the following behaviour of Pigeonhole, which I consider
>> to be problematic. A user deployed a Sieve script similar to the following
>> snippet
>>
>> if not anyof (address :is ["from","cc"] ["...", ..., "...@...
>> GARBAGE", ...] {
>>  fileinto "inbox.Trash";
>>  stop;
>> }
>>
>> Note the extra line break before GARBAGE. This script is obviously broken,
>> but gets accepted by sievec and only fails later, at runtime with
>>
>> line X: error: found stray carriage-return (CR) character in quoted
>>string started at line X.
>>
>> So, the question is whether line breaks in strings are allowed in general
>> and the runtime error is unavoidable, or should sievec return an error?
>>
>
> https://www.ietf.org/rfc/rfc3028.txt first hit of quoted-string
>
>  quoted-string = DQUOTE *CHAR DQUOTE
>;; in general, \ CHAR inside a string maps to CHAR
>;; so \" maps to " and \\ maps to \
>;; note that newlines and other characters are all allowed
>;; strings
>
> So, it's correct. But the address should reject the CR. I guess,
> Pigeonhole triggers the error for sanity purpose?
>
> - -- Steffen Kaiser
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1
>
> iQEVAwUBWwvuQsQnQQNheMxiAQILoAgAyRjSObVJkrAmxzyLau9gIvvMOM2R++HP
> pwsptIQ72xoYJOO/Lnd1TmfKTE9QYwtOGkSKr8tiJVD8JOpL5fUbB6mZNOTXkAv0
> TOW2gA7v06nXq6K0ETum8anoKTIF0o4j5aQJ5yQ5CrzlVQqUwTsf4mVVNqK0hn/L
> X5RAuCVQyx6sdvCB+lSOGmLv/fT8+xHS03U6jzCp/Yov5OKsT29oOOF6dXWR49Iw
> BL+DOd9T37hHF6ENp4A5wxX6iCMKLsWL0f5xTcxwRK5GOiCDoUH6ZpiywD0PtCuT
> VlusmbIByGON7foNlCPusTVcfq8GenMhOrgFcbp1PfRrShIQgsjWSg==
> =vgR1
> -END PGP SIGNATURE-
>

Re: Second rule isn't apply when first rule matches

[daniel_1983]

I Forgot to mention I am using dovecot version 2.2

Second rule isn't apply when first rule matches

[daniel_1983]

Dear list,
I want to define two concurrent rules : 
1. I want to flag an e-mail containing the word "cloud" in the body
2. I want to move mail sent explicitly to me (as opposed to mail sent to an 
alias I am part of) going to "INOBX.I must answere this"

If I put rule (1) first, everything works as expected. If put rule (2) first, 
only that rule applies.

Here's a small test case you should be able to reproduce on your setup. 

Let's work on this fake test e-mail :

$ cat testmail.eml

Date: Mon, 28 May 2018 12:41:53 +0100
From: fri...@otherdomain.tld
To: m...@mydomain.tld
Subject: dur dur la vie

mon frère, le cloud c'est le top du top


Here's the first, broken version of the script

$ cat test.sieve

require ["body","fileinto","imap4flags","vacation"];

# rule:[Mail about me]  

 
if anyof (header :contains "to" "m...@mydomain.tld", body :text :contains 
"ahmed")
{
  fileinto "INBOX.I must answere this";
}

# rule:[Mail about the Cloud]   

 
if body :text :contains "cloud"
{
addflag "\\Flagged";
}


Let's test it out, the two rules should be applied : 


$ sieve-test test.sieve testmail.eml 

Performed actions:

 * store message in folder: INBOX.I must answere

Implicit keep:

  (none)

sieve-test(root): Info: final result: success
$ 



Notice that the last rule isn't applied although it matches. Now we invert the 
order of the rules and apply the script on the same test e-mail : 


$ cat test.sieve

require ["body","fileinto","imap4flags","vacation"];
# rule:[Mail about the Cloud]   

 
if body :text :contains "cloud"
{
addflag "\\Flagged";
}

# rule:[Mail about me]  

 
if anyof (header :contains "to" "m...@mydomain.tld", body :text :contains 
"ahmed")
{
  fileinto "INBOX.I must answere this";
}



Running the test again : 

$ sieve-test test.sieve testmail.eml 

Performed actions:

 * store message in folder: INBOX.I must answere
+ add IMAP flags: \flagged

Implicit keep:

  (none)

sieve-test(root): Info: final result: success
$

The IMAP flag was set !


Any ideas ?

Re: SSL error after upgrading to 2.31

[Aki Tuomi]

On 28.05.2018 14:30, Hauke Fath wrote:
> On Mon, 28 May 2018 13:52:01 +0300, Aki Tuomi wrote:
>> I'm sure. But putting it as ssl_ca makes no sense, since it becomes
>> confused what it is for.
> I guess - I haven't had a need for client certs, and only ever used 
> ssl_ca for the server ca chain.
>
>> We can try restoring this as ssl_cert_chain setting in future release.
> Sounds good. How about (re)naming them ssl-{client,server}_ca?
>
> Cheerio,
> Hauke
>

There is already ssl_client_ca, for verifying clients. ssl_ca verifies
certs when dovecot is connecting somewhere.

Aki

Re: Problem in Pigeonhole sievec

[Steffen Kaiser]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, 28 May 2018, Thorsten Hater wrote:


I stumbled upon the following behaviour of Pigeonhole, which I consider
to be problematic. A user deployed a Sieve script similar to the following
snippet

if not anyof (address :is ["from","cc"] ["...", ..., "...@...
GARBAGE", ...] {
 fileinto "inbox.Trash";
 stop;
}

Note the extra line break before GARBAGE. This script is obviously broken,
but gets accepted by sievec and only fails later, at runtime with

line X: error: found stray carriage-return (CR) character in quoted
   string started at line X.

So, the question is whether line breaks in strings are allowed in general
and the runtime error is unavoidable, or should sievec return an error?


https://www.ietf.org/rfc/rfc3028.txt first hit of quoted-string

 quoted-string = DQUOTE *CHAR DQUOTE
   ;; in general, \ CHAR inside a string maps to CHAR
   ;; so \" maps to " and \\ maps to \
   ;; note that newlines and other characters are all allowed
   ;; strings

So, it's correct. But the address should reject the CR. I guess, 
Pigeonhole triggers the error for sanity purpose?


- -- 
Steffen Kaiser

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEVAwUBWwvuQsQnQQNheMxiAQILoAgAyRjSObVJkrAmxzyLau9gIvvMOM2R++HP
pwsptIQ72xoYJOO/Lnd1TmfKTE9QYwtOGkSKr8tiJVD8JOpL5fUbB6mZNOTXkAv0
TOW2gA7v06nXq6K0ETum8anoKTIF0o4j5aQJ5yQ5CrzlVQqUwTsf4mVVNqK0hn/L
X5RAuCVQyx6sdvCB+lSOGmLv/fT8+xHS03U6jzCp/Yov5OKsT29oOOF6dXWR49Iw
BL+DOd9T37hHF6ENp4A5wxX6iCMKLsWL0f5xTcxwRK5GOiCDoUH6ZpiywD0PtCuT
VlusmbIByGON7foNlCPusTVcfq8GenMhOrgFcbp1PfRrShIQgsjWSg==
=vgR1
-END PGP SIGNATURE-

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

[kfx]

On 28/05/2018 13:23, kfx wrote:

On 28/05/2018 13:04, Timo Sirainen wrote:
On 28 May 2018, at 13.28, kfx > wrote:



Especially what is in the "fts" header vs. "next uid" header? Does 
the UID in "fts" header keep changing every time you save a new 
mail? I suppose it will.


Diff between 2 emails:
next uid = 30104    |    next uid = 30105
last_indexed_uid = 30103    |    last_indexed_uid = 30104


So Dovecot thinks it has indexed everything.

You could also monitor (e.g. tcpdump/wireshark) the network traffic 
between Dovecot <-> Solr what happens when a new mail arrives. I 
suspect Dovecot sends it to Solr, which for whatever reason just 
ignores the update.


### TCPDUMP 
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml

37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernamename="body">Search Pattern: Kai8oovi

..




And Dovecot sends the mail.


# SOLR'S RESPONSE ###




 0
 0




And Solr receives it. Your tcpdump doesn't show softCommit="true" waitSearcher="true"/> being sent though. Do you see 
it being sent anywhere? 


Yes:

### TCPDUMP ###
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml


### /TCPDUMP ###



Does it make the mails visible if you run it yourself? Or if you run 
hard commit? :


curl http://:8983/solr/update?commit=true



# curl http://127.0.0.1:8983/solr/update?commit=true



     [0/0]


Error 404 Not Found

HTTP ERROR 404
Problem accessing /solr/update. Reason:
    Not Found



# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true




   0
   0



# doveadm search -u username mailbox INBOX body Kai8oovi
==> No result ('Kai8oovi' is the search pattern, it should returns 4 
results)


In the web interface of solr at http://127.0.0.1:8983/solr/#/~cores/dovecot

I can see:

lastModified: less than a minute ago
version:1428772
numDocs:6353615
maxDoc:6356213
deletedDocs:2598


So it IS indexing :(
Just below I see the "optimized:" parameter followed by an icon which 
seems saying that is NOT "optimized". Don't know if it's relevant.

This is driving me crazy :(

Re: SSL error after upgrading to 2.31

[Hauke Fath]

On Mon, 28 May 2018 13:52:01 +0300, Aki Tuomi wrote:
> I'm sure. But putting it as ssl_ca makes no sense, since it becomes
> confused what it is for.

I guess - I haven't had a need for client certs, and only ever used 
ssl_ca for the server ca chain.

> We can try restoring this as ssl_cert_chain setting in future release.

Sounds good. How about (re)naming them ssl-{client,server}_ca?

Cheerio,
Hauke

-- 
 The ASCII Ribbon CampaignHauke Fath
() No HTML/RTF in emailInstitut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
 Respect for open standards  Ruf +49-6151-16-21344

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

[kfx]

On 28/05/2018 13:04, Timo Sirainen wrote:
On 28 May 2018, at 13.28, kfx > wrote:



Especially what is in the "fts" header vs. "next uid" header? Does 
the UID in "fts" header keep changing every time you save a new mail? 
I suppose it will.


Diff between 2 emails:
next uid = 30104    |    next uid = 30105
last_indexed_uid = 30103    |    last_indexed_uid = 30104


So Dovecot thinks it has indexed everything.

You could also monitor (e.g. tcpdump/wireshark) the network traffic 
between Dovecot <-> Solr what happens when a new mail arrives. I 
suspect Dovecot sends it to Solr, which for whatever reason just 
ignores the update.


### TCPDUMP 
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml

37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernameSearch 
Pattern: Kai8oovi

..




And Dovecot sends the mail.


# SOLR'S RESPONSE ###




 0
 0




And Solr receives it. Your tcpdump doesn't show softCommit="true" waitSearcher="true"/> being sent though. Do you see it 
being sent anywhere? 


Yes:

### TCPDUMP ###
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml


### /TCPDUMP ###



Does it make the mails visible if you run it 
yourself? Or if you run hard commit? :


curl http://:8983/solr/update?commit=true



# curl http://127.0.0.1:8983/solr/update?commit=true
 




[0/0]


Error 404 Not Found

HTTP ERROR 404
Problem accessing /solr/update. Reason:
Not Found



# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true




  0
  0



# doveadm search -u username mailbox INBOX body Kai8oovi
==> No result ('Kai8oovi' is the search pattern, it should returns 4 
results)

Problem in Pigeonhole sievec

[Thorsten Hater]

Dear all,

I stumbled upon the following behaviour of Pigeonhole, which I consider
to be problematic. A user deployed a Sieve script similar to the following
snippet

if not anyof (address :is ["from","cc"] ["...", ..., "...@...
GARBAGE", ...] {
  fileinto "inbox.Trash";
  stop;
}

Note the extra line break before GARBAGE. This script is obviously broken,
but gets accepted by sievec and only fails later, at runtime with

line X: error: found stray carriage-return (CR) character in quoted
string started at line X.

So, the question is whether line breaks in strings are allowed in general
and the runtime error is unavoidable, or should sievec return an error?

Best regards,
 Thorsten

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

[Timo Sirainen]

On 28 May 2018, at 13.28, kfx  wrote:
> 
> 
>> Especially what is in the "fts" header vs. "next uid" header? Does the UID 
>> in "fts" header keep changing every time you save a new mail? I suppose it 
>> will. 
> 
> Diff between 2 emails:
> next uid = 30104|next uid = 30105
> last_indexed_uid = 30103|last_indexed_uid = 30104

So Dovecot thinks it has indexed everything.

>> You could also monitor (e.g. tcpdump/wireshark) the network traffic between 
>> Dovecot <-> Solr what happens when a new mail arrives. I suspect Dovecot 
>> sends it to Solr, which for whatever reason just ignores the update.
> 
> ### TCPDUMP 
> POST /solr/dovecot/update HTTP/1.1
> Host: localhost:8983
> Date: Mon, 28 May 2018 10:18:05 GMT
> Transfer-Encoding: chunked
> Connection: Keep-Alive
> Content-Type: text/xml
> 
> 37581 name="box">e0c58a309323515311083ea484a8 name="user">username name="id">37581/e0c58a309323515311083ea484a8/username name="body">Search Pattern: Kai8oovi
..
> 

And Dovecot sends the mail.

> # SOLR'S RESPONSE ###
> 
> 
> 
> 
>  0
>  0
> 
> 

And Solr receives it. Your tcpdump doesn't show  being sent though. Do you see it being sent anywhere? 
Does it make the mails visible if you run it yourself? Or if you run hard 
commit? :

curl http://:8983/solr/update?commit=true

Re: SSL error after upgrading to 2.31

[Aki Tuomi]

On 28.05.2018 13:05, Hauke Fath wrote:
> On 05/28/18 11:08, Aki Tuomi wrote:
>>
>>
>> On 28.05.2018 12:06, Hauke Fath wrote:
>>> On 05/21/18 17:55, Aki Tuomi wrote:
 ssl_ca is used only for validating client certificates.
>>>
>>> But it was used (though not documented, IIRC) for validating server
>>> certs, too. Since intermediate CA certs are usually valid a lot longer
>>> than the server certs, having to concat the certs is awkward, at best.
>>
>> As far as I know, it has never been working as replacement for adding
>> the chain to cert file.
>
> Well, you know your code better than I.  ;)
>
> But it has worked for us here pre-2.3 (see
> 
> ff., and confirmed by
> ).
>
> And from an admin POV, it makes a lot of sense to keep the
> intermediate cert chain separate from the server cert.
>
> Cheerio,
> hauke
>
I'm sure. But putting it as ssl_ca makes no sense, since it becomes
confused what it is for.

We can try restoring this as ssl_cert_chain setting in future release.

Aki

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

[kfx]

On 24/05/2018 15:33, Timo Sirainen wrote:

On 21 May 2018, at 14.11, kada...@gmail.com wrote:


Le 21/05/2018 à 12:38, Aki Tuomi a écrit :

can you try turning on pluign { fts_enforced = yes } and repeat your test?


Same (wrong) result:
1. Send an email with "too6Ouka" in the body

2. Search against "too6Ouka":
# doveadm search -u username mailbox INBOX body too6Ouka
--> No result

3. Force re-index:
# doveadm fts rescan -u username

4. Search again against "too6Ouka":
# doveadm search -u username mailbox INBOX body too6Ouka
--> e09cce0283e8695ab76002deed92 29055

Don't know if relevant, but on a side note, if I send a second message
with "too6Ouka" in the body, followed by:
# doveadm -v index -u username Inbox
--> doveadm(username): Info: INBOX: Cache is already up to date
And a search against the pattern immediately return only one result
instead of two.


So it looks like after "doveadm fts rescan" you can do initial indexing 
successfully. Afterwards the index isn't updated at all, unless you rebuild it entirely. 
fts_autoindex=yes appears to work as expected, because after fts rescan you did a search 
which automatically updated the index. So this doesn't have anything to do with 
fts_autoindex, just that your index updates don't seem to work at all.

What does it show in the header if you do:
doveadm dump /var/vmail/username/dovecot.index


Detected file type: index
-- INDEX: /var/vmail/username/dovecot.index
version .. = 7.3
base header size . = 120
header size .. = 640
record size .. = 16
compat flags . = 1
index id . = 1527164208 (2018-05-24 14:16:48)
flags  = 0
uid validity . = 1516890243 (2018-01-25 15:24:03)
next uid . = 30104
messages count ... = 17068
seen messages count .. = 17011
deleted messages count ... = 0
first recent uid . = 30104
first unseen uid lowwater  = 393
first deleted uid lowwater = 30087
log file seq . = 14
log file tail offset . = 28144
log file head offset . = 28144
log2 rotate time . = 1527487933 (2018-05-28 08:12:13)
last temp file scan .. = 0 (1970-01-01 01:00:00)
day stamp  = 1527458400 (2018-05-28 00:00:00)
day first uid[0] . = 30037
day first uid[1] . = 29921
day first uid[2] . = 29814
day first uid[3] . = 29669
day first uid[4] . = 2
day first uid[5] . = 0
day first uid[6] . = 0
day first uid[7] . = 0
-- Extension 0 --
name  = maildir
hdr_size  = 36
reset_id  = 0
record_offset = 0
record_size . = 0
record_align  = 0
header
 - new_check_time  = 2018-05-28 11:48:14
 - new_mtime . = 2018-05-28 11:47:59
 - new_mtime_nsecs ... = 129602175
 - cur_check_time  = 2018-05-28 11:48:14
 - cur_mtime . = 2018-05-28 11:48:01
 - cur_mtime_nsecs = 277650283
 - uidlist_mtime . = 2018-05-28 11:47:58
 - uidlist_mtime_nsecs = 521588557
 - uidlist_size .. = 1334026
-- Extension 1 --
name  = keywords
hdr_size  = 292
reset_id  = 0
record_offset = 5
record_size . = 3
record_align  = 1
-- Extension 2 --
name  = hdr-vsize
hdr_size  = 16
reset_id  = 0
record_offset = 0
record_size . = 0
record_align  = 8
header
 - highest uid . = 0
 - message count = 0
 - vsize ... = 0
-- Extension 3 --
name  = cache
hdr_size  = 0
reset_id  = 1527164233
record_offset = 8
record_size . = 4
record_align  = 4
-- Extension 4 --
name  = vsize
hdr_size  = 0
reset_id  = 0
record_offset = 12
record_size . = 4
record_align  = 4
-- Extension 5 --
name  = fts
hdr_size  = 12
reset_id  = 0
record_offset = 0
record_size . = 0
record_align  = 0
header
 - last_indexed_uid . = 30103
 - settings_checksum  = 0
-- Keywords --
  0 = unknown-1
  1 = unknown-2
  2 = unknown-5
  3 = unknown-8
  4 = unknown-6
  5 = unknown-3
  6 = unknown-7
  7 = unknown-0
  8 = NonJunk
  9 = $Forwarded
 10 = $label1
 11 = $label2
 12 = Junk
 13 = $label3
 14 = $label4




Especially what is in the "fts" header vs. "next uid" header? Does the UID in "fts" header keep changing every time you save a new mail? I suppose it will. 


Diff between 2 emails:
next uid = 30104|next uid = 30105
last_indexed_uid = 30103|last_indexed_uid = 30104



You could also monitor (e.g. tcpdump/wireshark) the network traffic between Dovecot 
<-> Solr what happens when a new mail arrives. I suspect Dovecot sends it to 
Solr, which for whatever reason just ignores the update.


### TCPDUMP 
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml

37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernamename="body">Search Pattern: 

Re: SSL error after upgrading to 2.31

[Hauke Fath]

On 05/28/18 11:08, Aki Tuomi wrote:



On 28.05.2018 12:06, Hauke Fath wrote:

On 05/21/18 17:55, Aki Tuomi wrote:

ssl_ca is used only for validating client certificates.


But it was used (though not documented, IIRC) for validating server
certs, too. Since intermediate CA certs are usually valid a lot longer
than the server certs, having to concat the certs is awkward, at best.


As far as I know, it has never been working as replacement for adding
the chain to cert file.


Well, you know your code better than I.  ;)

But it has worked for us here pre-2.3 (see 
 
ff., and confirmed by 
).


And from an admin POV, it makes a lot of sense to keep the intermediate 
cert chain separate from the server cert.


Cheerio,
hauke

--
 The ASCII Ribbon CampaignHauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
 Respect for open standards  Ruf +49-6151-16-21344

Re: SSL error after upgrading to 2.31

[Aki Tuomi]

On 28.05.2018 12:06, Hauke Fath wrote:
> On 05/21/18 17:55, Aki Tuomi wrote:
>> ssl_ca is used only for validating client certificates.
>
> But it was used (though not documented, IIRC) for validating server
> certs, too. Since intermediate CA certs are usually valid a lot longer
> than the server certs, having to concat the certs is awkward, at best.
>
> I would very much like to see the pre-2.3 behaviour of "ssl_ca" restored.
>
> Cheerio,
> hauke
>

As far as I know, it has never been working as replacement for adding
the chain to cert file.

Aki

Re: SSL error after upgrading to 2.31

[Hauke Fath]

On 05/21/18 17:55, Aki Tuomi wrote:

ssl_ca is used only for validating client certificates.


But it was used (though not documented, IIRC) for validating server 
certs, too. Since intermediate CA certs are usually valid a lot longer 
than the server certs, having to concat the certs is awkward, at best.


I would very much like to see the pre-2.3 behaviour of "ssl_ca" restored.

Cheerio,
hauke

--
 The ASCII Ribbon CampaignHauke Fath
() No HTML/RTF in email Institut für Nachrichtentechnik
/\ No Word docs in email TU Darmstadt
 Respect for open standards  Ruf +49-6151-16-21344

Re: Best mail encryption solution for per-user

[Aki Tuomi]

On 27.05.2018 21:16, m...@sjemm.net wrote:
> May 27, 2018 8:52 AM, "Aki Tuomi"  wrote:
>>> On 26 May 2018 at 10:36 m...@sjemm.net wrote:
>>>
>>> May 23, 2018 10:10 AM, m...@sjemm.net wrote:
>>> May 23, 2018 9:46 AM, "Aki Tuomi"  wrote:
>>>
>>> On 23.05.2018 10:15, m...@sjemm.net wrote:
>>>
>>> May 23, 2018 8:31 AM, "Aki Tuomi"  wrote:
>>>
>>> On 23.05.2018 09:13, m...@sjemm.net wrote:
>>> May 20, 2018 8:01 PM, m...@sjemm.net wrote:
>>> May 20, 2018 2:47 PM, "Aki Tuomi"  wrote:
>>> On 19 May 2018 at 16:40 m...@sjemm.net wrote:
>>>
>>> May 18, 2018 10:01 PM, "Aki Tuomi"  wrote:
>>> On 18 May 2018 at 21:44 m...@sjemm.net wrote:
>>>
>>> May 18, 2018 4:43 PM, "Aki Tuomi"  wrote:
>>> On 18 May 2018 at 17:38 m...@sjemm.net wrote:
>>>
>>> May 18, 2018 4:05 PM, "Aki Tuomi"  wrote:
>>> On 18 May 2018 at 16:43 m...@sjemm.net wrote:
>>>
>>> Hi Tai74 and Aki,
>>> I followed your conversation with interest on how to setup per user 
>>> encryption in dovecot.
>>> I have setup my dovecot with the following in a conf file:
>>>
>>> ==
>>>
>>> mail_attribute_dict = file:%h/Maildir/dovecot-attributes
>>> mail_plugins = $mail_plugins mail_crypt
>>> plugin {
>>>
>>> mail_crypt_curve = secp521r1
>>>
>>> mail_crypt_save_version = 2
>>>
>>> }
>>>
>>> ==
>>>
>>> This works nice, all emails are being encrypted and every user/folder has 
>>> keys.
>>> But as I understood from your conversation these keys are not protected. 
>>> And I want them to be
>>> protected by the users password used by imap.
>>>
>>> Those passwords are stored in a mysql DB file. ( I used a guide from 
>>> workaround [dot] org to set up
>>> the DB and postfix/dovecot)
>>>
>>> but how would i set it so, that the users password from the DB is used to 
>>> encrypt the keys?
>>>
>>> should i use mail_crypt_private_password = ?
>>> how do i point it to the mysql db then?
>>> im unsure about this
>>>
>>> Do you have any hints on this?
>>>
>>> Kind regards,
>>> Zjemm
>>>
>>> The passwords in your MySQL database are, hopefully, not in plaintext. If 
>>> you want to secure your
>>> user's keys using user's login password, you must have a TOOL that manages 
>>> this.
>>>
>>> You can use mail_crypt_private_password = %w in (mysql) passdb fields to 
>>> provide the user's login
>>> password as private password. You might want to run it thru some hash, so 
>>> %{sha1:password} might be
>>> a good option.
>>>
>>> You can change the key password using 'doveadm mailbox cryptokey', this 
>>> needs to be done every time
>>> user changes his password.
>>>
>>> Also note that if you go down this road, and the user forgets his password, 
>>> you will not be able to
>>> recover the emails without backup copy of the private key.
>>>
>>> Aki
>>>
>>> Hi Aki
>>>
>>> I used the following command:
>>> dovecot pw -s SHA256-CRYPT
>>>
>>> the output on the chosen password looks like: 
>>> {SHA256-CRYPT}$5$Rokc06a7In4SF3bO$OQpGQWqg
>>>
>>> This output is used to store in the password fields in the database. So no 
>>> plain text passwords no
>>> :)
>>>
>>> You can use mail_crypt_private_password = %w in (mysql) passdb fields to 
>>> provide the user's login
>>> password as private password.
>>>
>>> can you explain this a bit more for me?
>>>
>>> for now i have in the 10-auth.conf file the following:
>>> ==
>>> passdb {
>>> driver = sql
>>>
>>> # Path for SQL configuration file, see example-config/dovecot-sql.conf.ext
>>> args = /etc/dovecot/dovecot-sql.conf.ext
>>> }
>>>
>>> and:
>>>
>>> userdb {
>>> driver = static
>>> args = uid=vmail gid=vmail home=/var/vmail/%d/%n
>>> }
>>> ==
>>>
>>> then i have in dovecot-sql.conf.ext
>>> ==
>>> driver = mysql
>>> connect = host=x.x.x.x dbname=mailserver user=mailuser 
>>> password=mailpasswordexample
>>> default_pass_scheme = SHA256-CRYPT
>>> password_query = SELECT email as user, password FROM virtual_users WHERE 
>>> email='%u';
>>> ==
>>> Where do i need to set : mail_crypt_private_password = %w ?
>>>
>>> password as private password. You might want to run it thru some hash, so 
>>> %{sha1:password} might be
>>> a good option.
>>>
>>> the passwords are allready hashed in the DB using: dovecot pw -s 
>>> SHA256-CRYPT to genereate the has.
>>> so this step isnt nesesary anymore am i right?
>>>
>>> Thank you for your quick response, very helpfull
>>>
>>> Zjemm
>>>
>>> You misunderstood a bit. The idea is to use the *plaintext* password as the 
>>> password for the
>>> private key. Otherwise anyone could just decrypt it by looking at your 
>>> database where the hashed
>>> password is..
>>>
>>> So:
>>>
>>> password_query = SELECT email as user, password, '%w' AS 
>>> userdb_mail_crypt_private_password FROM
>>> virtual_users WHERE email='%u'
>>>
>>> Aki
>>>
>>> Hi Aki,
>>>
>>> Thank you very much for your help, i realy