Re: SOLR/Index?

[Larry Rosenman via dovecot]

Note the hits after the fts rescan/index.

Get Outlook for Android


From: Aki Tuomi 
Sent: Monday, April 15, 2019 12:55:07 AM
To: Larry Rosenman; John Fawcett
Cc: Dovecot Mailing List
Subject: Re: SOLR/Index?



On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
⌂72% [l...@thebighonker.lerctr.org:~] $ 
doveadm search mailbox lists/freebsd/ports-commiters  body 'sysutils'
[l...@thebighonker.lerctr.org:~] $ 
doveadm fts rescan
[l...@thebighonker.lerctr.org:~] $ 
doveadm index -q lists/freebsd/ports-commiters
⌂64% [l...@thebighonker.lerctr.org:~] $ 
tail -f /var/log/maillog
Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth 
failed, 1 attempts in 2 secs): user=, method=PLAIN, rip=180.180.217.124, 
lip=192.147.25.65, TLS: Connection closed, session=
Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=, 
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, 
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, 
session=
Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out 
in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 
23.100.68.192
Apr 14 19:30:55 thebighonker exim[14846]: 
H=(DaVinci-MWare.prophet21lab.com) 
[23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for 
mailto:i...@duke.org>>: host lookup did not complete
Apr 14 19:30:55 thebighonker exim[14846]: 
H=(DaVinci-MWare.prophet21lab.com) 
[23.100.68.192]:52130 I=[192.147.25.65]:25 
F=mailto:i...@duke.org>> temporarily rejected RCPT 
mailto:jpo...@why.net>>: Could not complete sender verify
Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=, 
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, 
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, 
session=
Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 
out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 
80.253.235.35
Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 
1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644)
^C
[l...@thebighonker.lerctr.org:~] 130 $ 
doveadm search mailbox lists/freebsd/ports-commiters  body 'sysutils/'


Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. 
FTS does not do substring searches by default.

Aki

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: 
larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: SOLR/Index?

[Aki Tuomi via dovecot]

On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
> ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
> lists/freebsd/ports-commiters  body 'sysutils'
> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
> [l...@thebighonker.lerctr.org:~] $ doveadm index -q
> lists/freebsd/ports-commiters
> ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected
> (auth failed, 1 attempts in 2 secs): user=, method=PLAIN,
> rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed,
> session=
> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
> session=
> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
> address 23.100.68.192
> Apr 14 19:30:55 thebighonker exim[14846]:
> H=(DaVinci-MWare.prophet21lab.com
> ) [23.100.68.192]:52130
> I=[192.147.25.65]:25 sender verify defer for  >: host lookup did not complete
> Apr 14 19:30:55 thebighonker exim[14846]:
> H=(DaVinci-MWare.prophet21lab.com
> ) [23.100.68.192]:52130
> I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
> temporarily rejected RCPT mailto:jpo...@why.net>>:
> Could not complete sender verify
> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
> session=
> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
> address 80.253.235.35
> Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919):
> Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644)
> ^C
> [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
> lists/freebsd/ports-commiters  body 'sysutils/'


Just minor nit, but you are searching for 'sysutils' first, then
'sysutils/'. FTS does not do substring searches by default.

Aki

>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> 
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: SOLR/Index?

[Larry Rosenman via dovecot]

⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
lists/freebsd/ports-commiters  body 'sysutils'
[l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
[l...@thebighonker.lerctr.org:~] $ doveadm index -q
lists/freebsd/ports-commiters
⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth
failed, 1 attempts in 2 secs): user=, method=PLAIN,
rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed,
session=
Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=,
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
session=
Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out
in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address
23.100.68.192
Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com)
[23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for <
i...@duke.org>: host lookup did not complete
Apr 14 19:30:55 thebighonker exim[14846]: H=(DaVinci-MWare.prophet21lab.com)
[23.100.68.192]:52130 I=[192.147.25.65]:25 F= temporarily
rejected RCPT : Could not complete sender verify
Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=,
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
session=
Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out
in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address
80.253.235.35
Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919):
Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644)
^C
[l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
lists/freebsd/ports-commiters  body 'sysutils/'
8097632f69627b5b5895bbe98eac 21077
8097632f69627b5b5895bbe98eac 21082
8097632f69627b5b5895bbe98eac 21083
8097632f69627b5b5895bbe98eac 21086
8097632f69627b5b5895bbe98eac 21118
8097632f69627b5b5895bbe98eac 21119
8097632f69627b5b5895bbe98eac 21121
8097632f69627b5b5895bbe98eac 21124
8097632f69627b5b5895bbe98eac 21125
8097632f69627b5b5895bbe98eac 21126
8097632f69627b5b5895bbe98eac 21127
8097632f69627b5b5895bbe98eac 21128
8097632f69627b5b5895bbe98eac 21141
8097632f69627b5b5895bbe98eac 21142
8097632f69627b5b5895bbe98eac 21168
8097632f69627b5b5895bbe98eac 21175
8097632f69627b5b5895bbe98eac 21180
8097632f69627b5b5895bbe98eac 21184
8097632f69627b5b5895bbe98eac 21186
8097632f69627b5b5895bbe98eac 21188
8097632f69627b5b5895bbe98eac 21195
8097632f69627b5b5895bbe98eac 21196
8097632f69627b5b5895bbe98eac 21198
8097632f69627b5b5895bbe98eac 21292
8097632f69627b5b5895bbe98eac 21312
8097632f69627b5b5895bbe98eac 21313
8097632f69627b5b5895bbe98eac 21323
8097632f69627b5b5895bbe98eac 21330
8097632f69627b5b5895bbe98eac 21344
8097632f69627b5b5895bbe98eac 21345
8097632f69627b5b5895bbe98eac 21348
8097632f69627b5b5895bbe98eac 21353
8097632f69627b5b5895bbe98eac 21354
8097632f69627b5b5895bbe98eac 21359
8097632f69627b5b5895bbe98eac 21367
8097632f69627b5b5895bbe98eac 21368
8097632f69627b5b5895bbe98eac 21369
8097632f69627b5b5895bbe98eac 21370
8097632f69627b5b5895bbe98eac 21371
8097632f69627b5b5895bbe98eac 21380
8097632f69627b5b5895bbe98eac 21390
8097632f69627b5b5895bbe98eac 21391
8097632f69627b5b5895bbe98eac 21392
8097632f69627b5b5895bbe98eac 21393
8097632f69627b5b5895bbe98eac 21394
8097632f69627b5b5895bbe98eac 21395
8097632f69627b5b5895bbe98eac 21439
8097632f69627b5b5895bbe98eac 21440
8097632f69627b5b5895bbe98eac 21480
8097632f69627b5b5895bbe98eac 21518
8097632f69627b5b5895bbe98eac 21538
8097632f69627b5b5895bbe98eac 21539
8097632f69627b5b5895bbe98eac 21593
8097632f69627b5b5895bbe98eac 21610
8097632f69627b5b5895bbe98eac 21612
8097632f69627b5b5895bbe98eac 21615
8097632f69627b5b5895bbe98eac 21682
8097632f69627b5b5895bbe98eac 21696
8097632f69627b5b5895bbe98eac 21697
8097632f69627b5b5895bbe98eac 21700
8097632f69627b5b5895bbe98eac 21701
8097632f69627b5b5895bbe98eac 21710
8097632f69627b5b5895bbe98eac 21743
8097632f69627b5b5895bbe98eac 21856
8097632f69627b5b5895bbe98eac 21858
8097632f69627b5b5895bbe98eac 21882
8097632f69627b5b5895bbe98eac 21883
8097632f69627b5b5895bbe98eac 21886
8097632f69627b5b5895bbe98eac 21887
8097632f69627b5b5895bbe98eac 21900
8097632f69627b5b5895bbe98eac 21910
8097632f69627b5b5895bbe98eac 21918
8097632f69627b5b5895bbe98eac 21930
8097632f69627b5b5895bbe98eac 21931
8097632f69627b5b5895bbe98eac 21955
8097632f69627b5b5895bbe98eac 21971
8097632f69627b5b5895bbe98eac 21986
8097632f69627b5b5895

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 01:39, Larry Rosenman via dovecot wrote:
>
> full solr.log at:
> https://www.lerctr.org/~ler/solr.log
>
> The search DOES make it to SOLR:
> ⌂77% [l...@thebighonker.lerctr.org:~] 130 $ grep sysutils
> /var/log/solr/solr.log
> 2019-04-14 18:31:34.749 INFO  (qtp349420578-7538) [   x:dovecot]
> o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select
> params={q={!lucene+q.op%3DAND}(hdr:sysutils\/+OR+body:sysutils\/)&fl=uid,score&sort=uid+asc&fq=%2Bbox:8097632f69627b5b5895bbe98eac+%2Buser:ler&rows=22644&wt=xml}
> hits=0 status=0 QTime=460
>
> Pick showing subjects:
> https://www.lerctr.org/~ler/sysutils_mail.png
>
> What else?
>
> I'm happy to provide access.
>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> 
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

so the search is returning no hits as you said. But can you show that
there is data in the index that should match?

doveadm search -u u...@example.com mailbox inbox body "sysutils/"

Can you do a controlled test and send yourself a test message with that
string and show the solr log where it is being inserted into the index
and then search for it with doveadm (just to rule out roundcube for the
moment) and show solr log for that search?

John

Re: SOLR/Index?

[Larry Rosenman via dovecot]

full solr.log at:
https://www.lerctr.org/~ler/solr.log

The search DOES make it to SOLR:
⌂77% [l...@thebighonker.lerctr.org:~] 130 $ grep sysutils
/var/log/solr/solr.log
2019-04-14 18:31:34.749 INFO  (qtp349420578-7538) [   x:dovecot]
o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select
params={q={!lucene+q.op%3DAND}(hdr:sysutils\/+OR+body:sysutils\/)&fl=uid,score&sort=uid+asc&fq=%2Bbox:8097632f69627b5b5895bbe98eac+%2Buser:ler&rows=22644&wt=xml}
hits=0 status=0 QTime=460

Pick showing subjects:
https://www.lerctr.org/~ler/sysutils_mail.png

What else?

I'm happy to provide access.

-- 
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 01:15, Larry Rosenman via dovecot wrote:
> Given all the discussion on FTS/Solr, etc, I have a question:
>
> I have autoindex set, and searching in roundcube most of the time does
> NOT find things,
> HOWEVER if I do:
> doveadm fts rescan
> doveadm index
>
> I can find things in the mailboxes.
>
> WHY?
>
> (doveconf -n attached).
>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> 
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

have you been able to check your solr logs to see if the queries that
find nothing actually arrive at solr? What is logged for those queries?

Some clients (don't know if roundcube is among those) don't send the
query in some circumstances.

Also when email arrives in dovecot is the automatic indexing working: do
you see solr logging for adding those messages.

John

SOLR/Index?

[Larry Rosenman via dovecot]

Given all the discussion on FTS/Solr, etc, I have a question:

I have autoindex set, and searching in roundcube most of the time does NOT
find things,
HOWEVER if I do:
doveadm fts rescan
doveadm index

I can find things in the mailboxes.

WHY?

(doveconf -n attached).

-- 
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
# 2.3.5.1 (7ec6d0ade): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.5 (2483b085)
# OS: FreeBSD 12.0-STABLE amd64  
# Hostname: thebighonker.lerctr.org
auth_mechanisms = plain login
auth_realms = lerctr.org thebighonker.lerctr.org tbh.lerctr.org 
thejonesonair.com thejonesonair.net why.net
default_vsz_limit = 1 G
deliver_log_format = msgid=%m: %$ (subject=%s from=%f size=%w)
doveadm_password = # hidden, use -P to show it
first_valid_gid = 0
first_valid_uid = 0
lda_mailbox_autocreate = yes
listen = 192.147.25.65, ::
login_access_sockets = tcpwrap
mail_attribute_dict = file:%h/mail/.imap/dovecot-mail-attributes
mail_location = mbox:~/mail:INBOX=~/mail/INBOX
mail_log_prefix = "%s(%u/%p): "
mail_plugins = " fts fts_solr notify virtual"
mail_privileged_group = mail
mail_server_admin = mailto:l...@lerctr.org
mail_server_comment = LERCTR Mail Server
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext vacation-seconds editheader mboxmetadata 
servermetadata vnd.dovecot.debug imapsieve vnd.dovecot.imapsieve
namespace archive {
  hidden = no
  list = no
  location = mbox:~/MAIL-ARCHIVE
  prefix = ARCHIVE/
  separator = /
}
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox INBOX {
auto = create
  }
  mailbox SENT {
special_use = \Sent
  }
  mailbox SPAM {
special_use = \Junk
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  mailbox virtual/Flagged {
special_use = \Flagged
  }
  mailbox virtual/all {
special_use = \All
  }
  prefix = 
  separator = /
}
namespace virtual {
  hidden = no
  list = yes
  location = virtual:~/MAIL-VIRTUAL
  prefix = Virtual/
  separator = /
}
passdb {
  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
passdb {
  args = user=%Ln noauthenticate
  driver = static
  skip = authenticated
}
passdb {
  args = failure_show_msg=yes session=yes max_requests=20
  driver = pam
  skip = authenticated
}
plugin {
  fts = solr
  fts_autoindex = yes
  fts_solr = url=http://thebighonker.lerctr.org:8983/solr/dovecot/
  fts_tika = http://localhost:9998/tika/
  imapsieve_mailbox1_before = 
file:/usr/local/share/dovecot-pigeonhole/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = SPAM
  imapsieve_mailbox2_before = 
file:/usr/local/share/dovecot-pigeonhole/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = SPAM
  imapsieve_mailbox2_name = *
  imapsieve_url = sieve://thebighonker.lerctr.org
  mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename 
flag_change append
  mail_log_fields = uid box msgid size from subject vsize flags
  recipient_delimiter = +-_
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
  sieve_execute_bin_dir = /usr/local/share/dovecot-pigeonhole/sieve
  sieve_extensions = +editheader +vacation-seconds +mboxmetadata 
+servermetadata +vnd.dovecot.debug
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute
  sieve_pipe_bin_dir = /usr/local/share/dovecot-pigeonhole/sieve
  sieve_plugins = sieve_imapsieve sieve_extprograms
}
postmaster_address = postmas...@lerctr.org
protocols = imap pop3 lmtp sieve
recipient_delimiter = +-_
service anvil {
  unix_listener anvil {
group = mail
mode = 0666
  }
}
service auth {
  unix_listener auth-client {
mode = 0666
  }
  unix_listener auth-master {
mode = 0666
  }
}
service doveadm {
  inet_listener http {
port = 8080
ssl = yes
  }
}
service indexer-worker {
  drop_priv_before_exec = yes
}
service lmtp {
  inet_listener lmtp {
address = 127.0.0.1
port = 24
  }
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
  inet_listener sieve_deprecated {
port = 2000
  }
}
service stats {
  unix_listener stats-reader {
group = mail
mode = 0666
user = 
  }
  unix_listener stats-writer {
group = mail
mode = 0666
user = 
  }
}
service tcpwrap {
  unix_listener login/tcpwrap {
group = $default_login_user
mode = 0600
user = $default_login_user
  }
}
ssl_cert = 

Re: dovecot Digest, Vol 192, Issue 52

[Shawn Heisey via dovecot]

On 4/14/2019 8:59 AM, Peter Mogensen via dovecot wrote:

I run with

SOLR_JAVA_MEM="-Xmx8g -Xms2g"


Without other details, I cannot even offer a guess as to whether an 8GB 
heap is enough.  How many documents are in all the indexes that Solr 
instance is handling?  Can you share your solrconfig.xml file?



Being able to configure it is great.
But I don't think it solves much. I recompiled with 100 as batch size
and it still ended in timeouts.
Then I recompiled with 10min timeout and now I see all the batches
completing and their processesing time is mostly between 1 and 2 minutes
(so all would have failed).


1 to 2 minutes for 100 documents to index would be an indication that 
something is very wrong with that Solr instance.  It seems to me that 
the batch should take less than a second if there is no commit.


Thanks,
Shawn

Re: [PATCH] Re: Solr connection timeout hardwired to 60s

[Shawn Heisey via dovecot]

On 4/14/2019 7:59 AM, John Fawcett via dovecot wrote:

From dovecot point of view I can see the following as potentially useful
features:

1) a configurable batch size would enable to tune the number of emails
per request and help stay under the 60 seconds hard coded http request
timeout. A configurable http timeout would be less useful, since this
will potentially run into other timeouts on solr side.


Even if several thousand emails are sent per batch, unless they're 
incredibly large, I can't imagine indexing them taking more than a few 
seconds.  Does dovecot send attachments to Solr as well as the email 
itself?  Hopefully it doesn't.  If it does, then you would want a 
smaller batch size.


But if the heap size for Solr is not big enough, that can cause major 
delays no matter what requests are being sent, because Java will be 
spending most of its time doing garbage collection.


I'm also assuming that the Solr server is on the same LAN as dovecot and 
that transferring the update data does not take a long time.


Thanks,
Shawn

Re: Fwd: Problem with solr working, but not indexing

[TG Servers via dovecot]

  
  
Thanks for providing help this quick!
  
  It's always the small things
  I just added now 
    mail_plugins = $mail_plugins fts
  fts_solr
  globally, right in front of the 2 protocol sections and it is
  indexing like hell now :)
  
  It seems it had to be done globally as I saw no reaction when
  putting it into doveadm protocol section
  
  Thanks!
  

On 14/04/2019 22:39, Aki Tuomi wrote:


  
  
   You need to load it either globally or for doveadm protocol
as well. 
   
  
   Loading it globally should be safe. 
   
  
   Aki 
  
 On 14 April 2019 23:36 TG Servers via dovecot
   wrote: 
 

 


 I have this in my dovecot.conf already, yes :

protocol imap {
    mail_plugins = $mail_plugins quota imap_quota imap_sieve
fts fts_solr
    mail_max_userip_connections = 20
    imap_idle_notify_interval = 24 mins
}

protocol lmtp {
    postmaster_address = postmas...@xxx.com
    mail_plugins = $mail_plugins sieve fts fts_solr
}
---


   
   On 14/04/2019 22:30, Aki Tuomi
via dovecot wrote: 
  
  
 


   On 14 April 2019 23:22 TG Servers via dovecot 
wrote: 
   
  
   
  
  Hi,

I have setup dovecot 2.3.5.1 with solr 7.7.1
Everything seems to be working so far except that solr
doesn't index a single message.
Solr is running, the web api can be accessed, I see the
dovecot core there, but with zero docs.

If I trigger a "body" search from Thunderbird solr is
responding and searching, but hitting 0 of course. 

Looks like that : 2019-04-14 19:57:42.789 INFO 
(qtp898557489-40) [   x:dovecot] o.a.s.c.S.Request
[dovecot]  webapp=/solr path=/select
params={q={!lucene+q.op%3DAND}body:foobar&fl=uid,score&sort=uid+asc&fq=%2Bbox:925efe067ac8a35ce143828c97da+%2Buser:x...@xxx.net&rows=2&wt=xml}
hits=0 status=0 QTime=2

Everytime a new message comes in indexer-worker is
working but indexing 0.

Looks like that : 2019-04-14T19:39:36.887817+02:00 riot
dovecot: indexer-worker(x...@xxx.net)<31697><3pDsLlhws1zMewAAgoyX2g:vb2mNFhws1zRewAAgoyX2g>:
Indexed 0 messages in INBOX

A "doveadm fts rescan" just brings up this message :
"Fatal: Unknown command 'fts', but plugin fts exists.
Try to set mail_plugins=fts"
   
 

 Have you tried to add 
 

 mail_plugins = $mail_plugins fts fts_solr to config 
 My
dovecot settings are :
plugin {
    sieve_plugins = sieve_imapsieve sieve_extprograms
    sieve_before =
/var/vmail/sieve/global/spam-global.sieve
    sieve = file:/var/vmail/sieve/%d/%n/scripts;active=/var/vmail/sieve/%d/%n/active-script.sieve

    ###
    ### Spam learning
    ...
    
    sieve_pipe_bin_dir = /usr/bin
    sieve_global_extensions = +vnd.dovecot.pipe

    fts = solr
    fts_autoindex = yes
    fts_solr = url="" class="moz-txt-link-freetext"
  href="http://localhost:8988/solr/dovecot/"
  moz-do-not-send="true">http://localhost:8988/solr/dovecot/
debug

    ...
}

I don't know what to do or what I am doing wrong because
solr is responding as said. Also the data directory of
the core is empty.
If I should provide more information please tell me
which, it's my own server so I have access to everything

Can anyone please help with that??

Thanks.



   
 If the suggested change does not help, send doveconf
  -n 

  ---
Aki Tuomi

  
  
  

Re: Fwd: Problem with solr working, but not indexing

[Aki Tuomi via dovecot]

 
 
  
   You need to load it either globally or for doveadm protocol as well.
  
  
   
  
  
   Loading it globally should be safe.
  
  
   
  
  
   Aki
  
  
   
On 14 April 2019 23:36 TG Servers via dovecot  wrote:
   
   

   
   

   
   
   
I have this in my dovecot.conf already, yes :protocol imap {    mail_plugins = $mail_plugins quota imap_quota imap_sieve fts fts_solr    mail_max_userip_connections = 20    imap_idle_notify_interval = 24 mins}protocol lmtp {    postmaster_address = postmas...@xxx.com    mail_plugins = $mail_plugins sieve fts fts_solr}---


 On 14/04/2019 22:30, Aki Tuomi via dovecot wrote:
 


 
  
 
 
  
   On 14 April 2019 23:22 TG Servers via dovecot 
wrote:
  
  
   
  
  
   
  
  Hi,I have setup dovecot 2.3.5.1 with solr 7.7.1Everything seems to be working so far except that solr doesn't index a single message.Solr is running, the web api can be accessed, I see the dovecot core there, but with zero docs.If I trigger a "body" search from Thunderbird solr is responding and searching, but hitting 0 of course. Looks like that : 2019-04-14 19:57:42.789 INFO  (qtp898557489-40) [   x:dovecot] o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select params={q={!lucene+q.op%3DAND}body:foobar&fl=uid,score&sort=uid+asc&fq=%2Bbox:925efe067ac8a35ce143828c97da+%2Buser:x...@xxx.net&rows=2&wt=xml} hits=0 status=0 QTime=2Everytime a new message comes in indexer-worker is working but indexing 0.Looks like that : 2019-04-14T19:39:36.887817+02:00 riot dovecot: indexer-worker(x...@xxx.net)<31697><3pDsLlhws1zMewAAgoyX2g:vb2mNFhws1zRewAAgoyX2g>: Indexed 0 messages in INBOXA "doveadm fts rescan" just brings up this message : "Fatal: Unknown command 'fts', but plugin fts exists. Try to set mail_plugins=fts"
 
 
  
 
 
  Have you tried to add
 
 
  
 
 
  mail_plugins = $mail_plugins fts fts_solr to config
 
 
  My dovecot settings are :plugin {    sieve_plugins = sieve_imapsieve sieve_extprograms    sieve_before = /var/vmail/sieve/global/spam-global.sieve    sieve = file:/var/vmail/sieve/%d/%n/scripts;active=/var/vmail/sieve/%d/%n/active-script.sieve    ###    ### Spam learning    ...        sieve_pipe_bin_dir = /usr/bin    sieve_global_extensions = +vnd.dovecot.pipe    fts = solr    fts_autoindex = yes    fts_solr = url="" class="moz-txt-link-freetext" href="http://localhost:8988/solr/dovecot/">http://localhost:8988/solr/dovecot/ debug    ...}I don't know what to do or what I am doing wrong because solr is responding as said. Also the data directory of the core is empty.If I should provide more information please tell me which, it's my own server so I have access to everythingCan anyone please help with that??Thanks.
 
 
  If the suggested change does not help, send doveconf -n
 
 
  ---
Aki Tuomi
 


   
  
  
   
  
  
   ---
Aki Tuomi
   
 

Fwd: Problem with solr working, but not indexing

[TG Servers via dovecot]

  
  

I have this in
my dovecot.conf already, yes :

protocol imap {
    mail_plugins = $mail_plugins quota imap_quota imap_sieve fts
fts_solr
    mail_max_userip_connections = 20
    imap_idle_notify_interval = 24 mins
}

protocol lmtp {
    postmaster_address = postmas...@xxx.com
    mail_plugins = $mail_plugins sieve fts fts_solr
}
---
This is doveconf -n :
# 2.3.5.1 (7ec6d0ade): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.5 (2483b085)
# OS: Linux 3.10.0-957.10.1.el7.x86_64 x86_64 CentOS Linux
release 7.6.1810 (Core)
# Hostname: xxx.xxx.com
auth_mechanisms = plain login
auth_verbose = yes
listen = *
mail_gid = vmail
mail_home = /var/vmail/mailboxes/%d/%n
mail_location = maildir:~/mail:LAYOUT=fs
mail_privileged_group = vmail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext imapsieve vnd.dovecot.imapsieve
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox Spam {
    auto = subscribe
    special_use = \Junk
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix =
  separator = .
  type = private
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}
plugin {
  fts = solr
  fts_autoindex = yes
  fts_solr = url="" class="moz-txt-link-freetext"
  href="http://localhost:8988/solr/dovecot/"
  moz-do-not-send="true">http://localhost:8988/solr/dovecot/
debug
  imapsieve_mailbox1_before = file:/var/vmail/sieve/global/learn-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Spam
  imapsieve_mailbox2_before = file:/var/vmail/sieve/global/learn-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Spam
  imapsieve_mailbox2_name = *
  quota = maildir:User quota
  quota_exceeded_message = User %u has exhausted allowed storage
space.
  sieve =
file:/var/vmail/sieve/%d/%n/scripts;active=/var/vmail/sieve/%d/%n/active-script.sieve
  sieve_before = /var/vmail/sieve/global/spam-global.sieve
  sieve_global_extensions = +vnd.dovecot.pipe
  sieve_pipe_bin_dir = /usr/bin
  sieve_plugins = sieve_imapsieve sieve_extprograms
}
protocols = imap lmtp sieve
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0660
    user = postfix
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0660
    user = vmail
  }
}
service imap-login {
  inet_listener imap {
    port = 0
  }
  inet_listener imaps {
    port = 993
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0660
    user = postfix
  }
  user = vmail
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}
ssl = required
ssl_ca = 
ssl_cert = 
ssl_cipher_list =
ALL:!LOW:!SSLv2:!SSLv3:!TLSv1.0:!EXP:!aNULL:!MEDIUM:!CAMELLIA:@STRENGTH
ssl_client_ca_dir = /etc/ssl/certs
ssl_client_ca_file = /etc/ssl/certs/ca-bundle.crt
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_min_protocol = TLSv1.2
ssl_prefer_server_ciphers = yes
userdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}
protocol imap {
  imap_idle_notify_interval = 24 mins
  mail_max_userip_connections = 20
  mail_plugins = " quota imap_quota imap_sieve fts fts_solr"
}
protocol lmtp {
  mail_plugins = " sieve fts fts_solr"
  postmaster_address = postmas...@xxx.com
}



  
  On 14/04/2019 22:30, Aki Tuomi via
 

Re: Problem with solr working, but not indexing

[Aki Tuomi via dovecot]

 
 
  
   
  
  
   
On 14 April 2019 23:22 TG Servers via dovecot  wrote:
   
   

   
   

   
   Hi,I have setup dovecot 2.3.5.1 with solr 7.7.1Everything seems to be working so far except that solr doesn't index a single message.Solr is running, the web api can be accessed, I see the dovecot core there, but with zero docs.If I trigger a "body" search from Thunderbird solr is responding and searching, but hitting 0 of course. Looks like that : 2019-04-14 19:57:42.789 INFO  (qtp898557489-40) [   x:dovecot] o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select params={q={!lucene+q.op%3DAND}body:foobar&fl=uid,score&sort=uid+asc&fq=%2Bbox:925efe067ac8a35ce143828c97da+%2Buser:x...@xxx.net&rows=2&wt=xml} hits=0 status=0 QTime=2Everytime a new message comes in indexer-worker is working but indexing 0.Looks like that : 2019-04-14T19:39:36.887817+02:00 riot dovecot: indexer-worker(x...@xxx.net)<31697><3pDsLlhws1zMewAAgoyX2g:vb2mNFhws1zRewAAgoyX2g>: Indexed 0 messages in INBOXA "doveadm fts rescan" just brings up this message : "Fatal: Unknown command 'fts', but plugin fts exists. Try to set mail_plugins=fts"
  
  
   
  
  
   Have you tried to add
  
  
   
  
  
   mail_plugins = $mail_plugins fts fts_solr to config
  
  
   My dovecot settings are :plugin {    sieve_plugins = sieve_imapsieve sieve_extprograms    sieve_before = /var/vmail/sieve/global/spam-global.sieve    sieve = file:/var/vmail/sieve/%d/%n/scripts;active=/var/vmail/sieve/%d/%n/active-script.sieve    ###    ### Spam learning    ...        sieve_pipe_bin_dir = /usr/bin    sieve_global_extensions = +vnd.dovecot.pipe    fts = solr    fts_autoindex = yes    fts_solr = url="" href="http://localhost:8988/solr/dovecot/" class="moz-txt-link-freetext">http://localhost:8988/solr/dovecot/ debug    ...}I don't know what to do or what I am doing wrong because solr is responding as said. Also the data directory of the core is empty.If I should provide more information please tell me which, it's my own server so I have access to everythingCan anyone please help with that??Thanks.
  
  
   If the suggested change does not help, send doveconf -n
  
  
   ---
Aki Tuomi
   
 

Problem with solr working, but not indexing

[TG Servers via dovecot]

  
  
Hi,
  
  I have setup dovecot 2.3.5.1 with solr 7.7.1
  Everything seems to be working so far except that solr doesn't
  index a single message.
  Solr is running, the web api can be accessed, I see the dovecot
  core there, but with zero docs.
  
  If I trigger a "body" search from Thunderbird solr is responding
  and searching, but hitting 0 of course. 
  
  Looks like that : 2019-04-14 19:57:42.789 INFO  (qtp898557489-40)
  [   x:dovecot] o.a.s.c.S.Request [dovecot]  webapp=/solr
  path=/select
params={q={!lucene+q.op%3DAND}body:foobar&fl=uid,score&sort=uid+asc&fq=%2Bbox:925efe067ac8a35ce143828c97da+%2Buser:x...@xxx.net&rows=2&wt=xml}
  hits=0 status=0 QTime=2
  
  Everytime a new message comes in indexer-worker is working but
  indexing 0.
  
  Looks like that : 2019-04-14T19:39:36.887817+02:00 riot dovecot:
indexer-worker(x...@xxx.net)<31697><3pDsLlhws1zMewAAgoyX2g:vb2mNFhws1zRewAAgoyX2g>:
  Indexed 0 messages in INBOX
  
  A "doveadm fts rescan" just brings up this message : "Fatal:
  Unknown command 'fts', but plugin fts exists. Try to set
  mail_plugins=fts"
  
  My dovecot settings are :
  plugin {
      sieve_plugins = sieve_imapsieve sieve_extprograms
      sieve_before = /var/vmail/sieve/global/spam-global.sieve
      sieve =
file:/var/vmail/sieve/%d/%n/scripts;active=/var/vmail/sieve/%d/%n/active-script.sieve
  
      ###
      ### Spam learning
      ...
      
      sieve_pipe_bin_dir = /usr/bin
      sieve_global_extensions = +vnd.dovecot.pipe
  
      fts = solr
      fts_autoindex = yes
      fts_solr = url="" class="moz-txt-link-freetext" href="http://localhost:8988/solr/dovecot/">http://localhost:8988/solr/dovecot/ debug
  
      ...
  }
  
  I don't know what to do or what I am doing wrong because solr is
  responding as said. Also the data directory of the core is empty.
  If I should provide more information please tell me which, it's my
  own server so I have access to everything
  
  Can anyone please help with that??
  
  Thanks.
  
  
  

  

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 17:55, John Fawcett via dovecot wrote:
> The solr server is a small test virtual machine with 0.2 (shared) vCPU
> and 0.6MB of memory and non SSD storage. It can index around 2000 emails
> per minute when there is no other activity. Average email size is about
> 45Kb. I'm not indexing attachments.
>
> John
>
I more than double the performance by using a 0.5 shared vCPU and SSD
storage. Approx 4500 emails per minute.

John

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 17:16, Peter Mogensen via dovecot wrote:
> sorry... I got distracted half way and forgot to put a meaningfull
> subject so the archive could figure out the thread. - resending.
>
> On 4/14/19 4:04 PM, dovecot-requ...@dovecot.org wrote:
>
>>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>>> false on the autoCommit.? The autoSoftCommit setting is not enabled by
>>> default, but depending on how the index was created, Solr might try to
>>> set autoSoftCommit to 3 seconds ... which is WAY too short.
> I just run with the default. 15s autoCommit and no autoSoftCommit
>
>>> This thread says that dovecot is sending explicit commits.? 
> I see explicit /update req. with softCommit and waitSearcer=true in a
> tcpdump.
>
>>> One thing
>>> that might be happening to exceed 60 seconds is an extremely long
>>> commit, which is usually caused by excessive cache autowarming, but
>>> might be related to insufficient memory.? The max heap setting on an
>>> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY
>>> small, and it doesn't take much index data before a much larger heap
>>> is required.
> I run with
>
> SOLR_JAVA_MEM="-Xmx8g -Xms2g"
>
>> I looked into the code (version 2.3.5.1):
> This is 2.2.35. I haven't checked the source difference to 2.3.x I must
> admit.
>
>> I immagine that one of the reasons dovecot sends softCommits is because
>> without autoindex active and even if mailboxes are periodically indexed
>> from cron, the last emails received with be indexed at the moment of the
>> search.? 
> I expect that dovecot has to because of it's default behavior by only
> bringing the index up-to-date just before search. So it has towait for
> the index result to be available if there's been any new mails indexed.
>
>> 1) a configurable batch size would enable to tune the number of emails
>> per request and help stay under the 60 seconds hard coded http request
>> timeout. A configurable http timeout would be less useful, since this
>> will potentially run into other timeouts on solr side.
> Being able to configure it is great.
> But I don't think it solves much. I recompiled with 100 as batch size
> and it still ended in timeouts.
> Then I recompiled with 10min timeout and now I see all the batches
> completing and their processesing time is mostly between 1 and 2 minutes
> (so all would have failed).
>
> To me it looks like Solr just takes too long time to index. This is no
> small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
> and for this test it's not doing anything else, so I'm a bit surprised
> that even with only a few users this takes so long time.
>
> /Peter
>
>
Peter

I suppose you could go with a batch size of 50. If it's linear, you
could still keep under the default 60 seconds http request time :-)

I'm now testing with solr settings autoCommit 15 seconds, autoSoftCommit
60 seconds and sending no softCommits from dovecot and 500 batch size.

I've set up

    /usr/local/bin/doveadm index -A "*"

in crontab every 5 minutes so indexes will stay mostly up to date to
minimize amount of mail not already visible in the index when searches
are done.

The solr server is a small test virtual machine with 0.2 (shared) vCPU
and 0.6MB of memory and non SSD storage. It can index around 2000 emails
per minute when there is no other activity. Average email size is about
45Kb. I'm not indexing attachments.

John

Re: [PATCH] Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 16:04, Aki Tuomi via dovecot wrote:
>
>> On 14 April 2019 16:59 John Fawcett via dovecot < dovecot@dovecot.org
>> > wrote:
>>
>>
>> On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
>>> On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:
 If this value was made configurable people could set it to what they
 want. However the underlying problem is likely on solr configuration.
>>> The Jetty that is included in Solr has its idle timeout set to 50
>>> seconds.  But in practice, I have not seen this timeout trigger ...
>>> and if the OP is seeing a 60 second timeout, then the 50 second idle
>>> timeout in Jetty must not be occurring.
>>> There may be a socket timeout configured on inter-server requests --
>>> distributed queries or the load balancing that SolrCloud does.  I can
>>> never remember whether this is the case by default.  I think it is.
>> >> If there is an issue on initial indexing, where you are not really
>> >> concerned about qucik visibility but just getting things into the
>> index
>> >> efficiently, a better approach would be for dovecot plugin not to
>> send
>> >> any commit or softCommit (or waitSearcher either) and that should
>> speed
>> >> things up. You'd need to configure solr with a long autoSoftCommit
>> >> maxTime and a reasonable autoCommit maxTime, which you could then
>> >> reconfigure when the load was done.
>> >
>>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>>> false on the autoCommit.  The autoSoftCommit setting is not enabled by
>>> default, but depending on how the index was created, Solr might try to
>>> set autoSoftCommit to 3 seconds ... which is WAY too short.
>>> I will usually increase the autoCommit time to 60 seconds, just to
>>> reduce the amount of work that Solr is doing.  The autoSoftCommit
>>> time, if it is used, should be set to a reasonably long value ...
>>> values between two and five minutes would be good.  Attempting to use
>>> a very short autoSoftCommit time will usually lead to problems.
>>> This thread says that dovecot is sending explicit commits.  One thing
>>> that might be happening to exceed 60 seconds is an extremely long
>>> commit, which is usually caused by excessive cache autowarming, but
>>> might be related to insufficient memory.  The max heap setting on an
>>> out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY
>>> small, and it doesn't take much index data before a much larger heap
>>> is required.
>>> Thanks,
>>> Shawn
>> I looked into the code (version 2.3.5.1): the fts-solr plugin is not
>> sending softCommit every 1000 emails. Emails from a single folder are
>> batched in up to a maximum of 1000 emails per request, but the
>> softCommit gets sent once per mailbox folder at the end of all the
>> requests for that folder.
>>
>> I immagine that one of the reasons dovecot sends softCommits is because
>> without autoindex active and even if mailboxes are periodically indexed
>> from cron, the last emails received with be indexed at the moment of the
>> search.  So while sending softCommit has the advantage of including
>> recent mails in searches, it means that softCommits are being done upon
>> user request. Frequency depends on user activity.
>>
>> Going back to the original problem: seems the first advice to Peter is
>> to look into solr configuration as others have said.
>>
>> From dovecot point of view I can see the following as potentially useful
>> features:
>>
>> 1) a configurable batch size would enable to tune the number of emails
>> per request and help stay under the 60 seconds hard coded http request
>> timeout. A configurable http timeout would be less useful, since this
>> will potentially run into other timeouts on solr side.
>>
>> 2) abilty to turn off softCommits so as to have a more predictable
>> softCommit workload. In that case autoSoftCommit should be configured in
>> solr. In order to minimize risk of recent emails not appearing in search
>> results, periodic indexing could be set up by cron.
>>
>> I've attached a patch, any comments are welcome (especially about
>> getting settings from the backend context).
>>
>> Example config
>>
>> plugin {
>>   fts = solr
>>   fts_solr =
>> url= https://user:passw...@solr.example.com:443/solr/dovecot/
>> batch_size=500 no_soft_commit
>> }
>>
>> John
>
> Can you please open a pull request to https://github.com/dovecot/core ?
> ---
> Aki Tuomi

Done, thanks for considering it.

John

Re: Solr connection timeout hardwired to 60s

[Peter Mogensen via dovecot]

sorry... I got distracted half way and forgot to put a meaningfull
subject so the archive could figure out the thread. - resending.

On 4/14/19 4:04 PM, dovecot-requ...@dovecot.org wrote:

>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>> false on the autoCommit.? The autoSoftCommit setting is not enabled by
>> default, but depending on how the index was created, Solr might try to
>> set autoSoftCommit to 3 seconds ... which is WAY too short.

I just run with the default. 15s autoCommit and no autoSoftCommit

>> This thread says that dovecot is sending explicit commits.? 

I see explicit /update req. with softCommit and waitSearcer=true in a
tcpdump.

>> One thing
>> that might be happening to exceed 60 seconds is an extremely long
>> commit, which is usually caused by excessive cache autowarming, but
>> might be related to insufficient memory.? The max heap setting on an
>> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY
>> small, and it doesn't take much index data before a much larger heap
>> is required.

I run with

SOLR_JAVA_MEM="-Xmx8g -Xms2g"

> I looked into the code (version 2.3.5.1):

This is 2.2.35. I haven't checked the source difference to 2.3.x I must
admit.

> I immagine that one of the reasons dovecot sends softCommits is because
> without autoindex active and even if mailboxes are periodically indexed
> from cron, the last emails received with be indexed at the moment of the
> search.? 

I expect that dovecot has to because of it's default behavior by only
bringing the index up-to-date just before search. So it has towait for
the index result to be available if there's been any new mails indexed.

> 1) a configurable batch size would enable to tune the number of emails
> per request and help stay under the 60 seconds hard coded http request
> timeout. A configurable http timeout would be less useful, since this
> will potentially run into other timeouts on solr side.

Being able to configure it is great.
But I don't think it solves much. I recompiled with 100 as batch size
and it still ended in timeouts.
Then I recompiled with 10min timeout and now I see all the batches
completing and their processesing time is mostly between 1 and 2 minutes
(so all would have failed).

To me it looks like Solr just takes too long time to index. This is no
small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
and for this test it's not doing anything else, so I'm a bit surprised
that even with only a few users this takes so long time.

/Peter

Re: dovecot Digest, Vol 192, Issue 52

[Peter Mogensen via dovecot]

On 4/14/19 4:04 PM, dovecot-requ...@dovecot.org wrote:

>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>> false on the autoCommit.? The autoSoftCommit setting is not enabled by
>> default, but depending on how the index was created, Solr might try to
>> set autoSoftCommit to 3 seconds ... which is WAY too short.

I just run with the default. 15s autoCommit and no autoSoftCommit

>> This thread says that dovecot is sending explicit commits.? 

I see explicit /update req. with softCommit and waitSearcer=true in a
tcpdump.

>> One thing
>> that might be happening to exceed 60 seconds is an extremely long
>> commit, which is usually caused by excessive cache autowarming, but
>> might be related to insufficient memory.? The max heap setting on an
>> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY
>> small, and it doesn't take much index data before a much larger heap
>> is required.

I run with

SOLR_JAVA_MEM="-Xmx8g -Xms2g"

> I looked into the code (version 2.3.5.1):

This is 2.2.35. I haven't checked the source difference to 2.3.x I must
admit.

> I immagine that one of the reasons dovecot sends softCommits is because
> without autoindex active and even if mailboxes are periodically indexed
> from cron, the last emails received with be indexed at the moment of the
> search.? 

I expect that dovecot has to because of it's default behavior by only
bringing the index up-to-date just before search. So it has towait for
the index result to be available if there's been any new mails indexed.

> 1) a configurable batch size would enable to tune the number of emails
> per request and help stay under the 60 seconds hard coded http request
> timeout. A configurable http timeout would be less useful, since this
> will potentially run into other timeouts on solr side.

Being able to configure it is great.
But I don't think it solves much. I recompiled with 100 as batch size
and it still ended in timeouts.
Then I recompiled with 10min timeout and now I see all the batches
completing and their processesing time is mostly between 1 and 2 minutes
(so all would have failed).

To me it looks like Solr just takes too long time to index. This is no
small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
and for this test it's not doing anything else, so I'm a bit surprised
that even with only a few users this takes so long time.

/Peter

Re: Mail account brute force / harassment

[mj via dovecot]

Hi,

On 4/12/19 11:05 PM, Joseph Tam via dovecot wrote:

"www.blocklist.de" is a nifty source.  Could you suggest other publically
available blacklists?



The ones we are using are:


"file:///etc/ipset-blacklist/ip-blacklist-custom.list" # optional, for your 
personal nemeses (no typo, plural)


In this file we have our own manual additions


"https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1"; # Project Honey 
Pot Directory of Dictionary Attacker IPs
"https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1";  # TOR 
Exit Nodes
"https://www.maxmind.com/en/high-risk-ip-sample-list"; # MaxMind GeoIP 
Anonymous Proxies
"http://danger.rulez.sk/projects/bruteforceblocker/blist.php"; # 
BruteForceBlocker IP List
"https://www.spamhaus.org/drop/drop.lasso"; # Spamhaus Don't Route Or Peer 
List (DROP)
"http://cinsscore.com/list/ci-badguys.txt"; # C.I. Army Malicious IP List
"https://lists.blocklist.de/lists/all.txt"; # blocklist.de attackers
"http://blocklist.greensnow.co/greensnow.txt"; # GreenSnow

"https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset";
 # Firehol Level 1

"https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/stopforumspam_7d.ipset";
 # Stopforumspam via Firehol


MJ

Re: [PATCH] Re: Solr connection timeout hardwired to 60s

[Aki Tuomi via dovecot]

 
 
  
   
  
  
   
On 14 April 2019 16:59 John Fawcett via dovecot <
dovecot@dovecot.org> wrote:
   
   

   
   

   
   
On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
   
   

 On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:


 
  If this value was made configurable people could set it to what they
 
 
  want. However the underlying problem is likely on solr configuration.
 

   
   

 The Jetty that is included in Solr has its idle timeout set to 50


 seconds.  But in practice, I have not seen this timeout trigger ...


 and if the OP is seeing a 60 second timeout, then the 50 second idle


 timeout in Jetty must not be occurring.

   
   

 There may be a socket timeout configured on inter-server requests --


 distributed queries or the load balancing that SolrCloud does.  I can


 never remember whether this is the case by default.  I think it is.

   
   
>> If there is an issue on initial indexing, where you are not really
   
   
>> concerned about qucik visibility but just getting things into the index
   
   
>> efficiently, a better approach would be for dovecot plugin not to send
   
   
>> any commit or softCommit (or waitSearcher either) and that should speed
   
   
>> things up. You'd need to configure solr with a long autoSoftCommit
   
   
>> maxTime and a reasonable autoCommit maxTime, which you could then
   
   
>> reconfigure when the load was done.
   
   
>
   
   

 Solr ships with autoCommit set to 15 seconds and openSearcher set to


 false on the autoCommit.  The autoSoftCommit setting is not enabled by


 default, but depending on how the index was created, Solr might try to


 set autoSoftCommit to 3 seconds ... which is WAY too short.

   
   

 I will usually increase the autoCommit time to 60 seconds, just to


 reduce the amount of work that Solr is doing.  The autoSoftCommit


 time, if it is used, should be set to a reasonably long value ...


 values between two and five minutes would be good.  Attempting to use


 a very short autoSoftCommit time will usually lead to problems.

   
   

 This thread says that dovecot is sending explicit commits.  One thing


 that might be happening to exceed 60 seconds is an extremely long


 commit, which is usually caused by excessive cache autowarming, but


 might be related to insufficient memory.  The max heap setting on an


 out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY


 small, and it doesn't take much index data before a much larger heap


 is required.

   
   

 Thanks,


 Shawn

   
   
I looked into the code (version 2.3.5.1): the fts-solr plugin is not
   
   
sending softCommit every 1000 emails. Emails from a single folder are
   
   
batched in up to a maximum of 1000 emails per request, but the
   
   
softCommit gets sent once per mailbox folder at the end of all the
   
   
requests for that folder.
   
   

   
   
I immagine that one of the reasons dovecot sends softCommits is because
   
   
without autoindex active and even if mailboxes are periodically indexed
   
   
from cron, the last emails received with be indexed at the moment of the
   
   
search.  So while sending softCommit has the advantage of including
   
   
recent mails in searches, it means that softCommits are being done upon
   
   
user request. Frequency depends on user activity.
   
   

   
   
Going back to the original problem: seems the first advice to Peter is
   
   
to look into solr configuration as others have said.
   
   

   
   
From dovecot point of view I can see the following as potentially useful
   
   
features:
   
   

   
   
1) a configurable batch size would enable to tune the number of emails
   
   
per request and help stay under the 60 seconds hard coded http request
   
   
timeout. A configurable http timeout would be less useful, since this
   
   
will potentially run into other timeouts on solr side.
   
   

   
   
2) abilty to turn off softCommits so as to have a more predictable
   
   
softCommit workload. In that case autoSoftCommit should be configured in
   
   
solr. In order to minimize risk of recent emails not appearing in search
   
   
results, periodic indexing could be set up by cron.
   
   

   
   
I've attached a patch, any comments are welcome (especially about
   
   
getting settings from the backend context).
   
   

   
   
Example config
   
   

   
   
plugin {
   
   
  fts = solr
   
   
  fts_solr =
 

[PATCH] Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
> On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:
>> If this value was made configurable people could set it to what they
>> want. However the underlying problem is likely on solr configuration.
>
> The Jetty that is included in Solr has its idle timeout set to 50
> seconds.  But in practice, I have not seen this timeout trigger ...
> and if the OP is seeing a 60 second timeout, then the 50 second idle
> timeout in Jetty must not be occurring.
>
> There may be a socket timeout configured on inter-server requests --
> distributed queries or the load balancing that SolrCloud does.  I can
> never remember whether this is the case by default.  I think it is.
>
>> If there is an issue on initial indexing, where you are not really
>> concerned about qucik visibility but just getting things into the index
>> efficiently, a better approach would be for dovecot plugin not to send
>> any commit or softCommit (or waitSearcher either) and that should speed
>> things up. You'd need to configure solr with a long autoSoftCommit
>> maxTime and a reasonable autoCommit maxTime, which you could then
>> reconfigure when the load was done.
>
> Solr ships with autoCommit set to 15 seconds and openSearcher set to
> false on the autoCommit.  The autoSoftCommit setting is not enabled by
> default, but depending on how the index was created, Solr might try to
> set autoSoftCommit to 3 seconds ... which is WAY too short.
>
> I will usually increase the autoCommit time to 60 seconds, just to
> reduce the amount of work that Solr is doing.  The autoSoftCommit
> time, if it is used, should be set to a reasonably long value ...
> values between two and five minutes would be good.  Attempting to use
> a very short autoSoftCommit time will usually lead to problems.
>
> This thread says that dovecot is sending explicit commits.  One thing
> that might be happening to exceed 60 seconds is an extremely long
> commit, which is usually caused by excessive cache autowarming, but
> might be related to insufficient memory.  The max heap setting on an
> out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY
> small, and it doesn't take much index data before a much larger heap
> is required.
>
> Thanks,
> Shawn

I looked into the code (version 2.3.5.1): the fts-solr plugin is not
sending softCommit every 1000 emails. Emails from a single folder are
batched in up to a maximum of 1000 emails per request, but the
softCommit gets sent once per mailbox folder at the end of all the
requests for that folder.

I immagine that one of the reasons dovecot sends softCommits is because
without autoindex active and even if mailboxes are periodically indexed
from cron, the last emails received with be indexed at the moment of the
search.  So while sending softCommit has the advantage of including
recent mails in searches, it means that softCommits are being done upon
user request. Frequency depends on user activity.

Going back to the original problem: seems the first advice to Peter is
to look into solr configuration as others have said.

>From dovecot point of view I can see the following as potentially useful
features:

1) a configurable batch size would enable to tune the number of emails
per request and help stay under the 60 seconds hard coded http request
timeout. A configurable http timeout would be less useful, since this
will potentially run into other timeouts on solr side.

2) abilty to turn off softCommits so as to have a more predictable
softCommit workload. In that case autoSoftCommit should be configured in
solr. In order to minimize risk of recent emails not appearing in search
results, periodic indexing could be set up by cron.

I've attached a patch, any comments are welcome (especially about
getting settings from the backend context).

Example config

plugin {
  fts = solr
  fts_solr =
url=https://user:passw...@solr.example.com:443/solr/dovecot/
batch_size=500 no_soft_commit
}

John

--- src/plugins/fts-solr/fts-solr-plugin.h.orig 2019-04-14 15:12:07.694289402 
+0200
+++ src/plugins/fts-solr/fts-solr-plugin.h  2019-04-14 14:04:17.213939414 
+0200
@@ -12,8 +12,10 @@
 
 struct fts_solr_settings {
const char *url, *default_ns_prefix;
+   unsigned int batch_size;
bool use_libfts;
bool debug;
+   bool no_soft_commit;
 };
 
 struct fts_solr_user {
--- src/plugins/fts-solr/fts-solr-plugin.c.orig 2019-04-14 11:41:03.591782439 
+0200
+++ src/plugins/fts-solr/fts-solr-plugin.c  2019-04-14 14:37:46.059433864 
+0200
@@ -10,6 +10,8 @@
 #include "fts-solr-plugin.h"
 
 
+#define DEFAULT_SOLR_BATCH_SIZE 1000
+
 const char *fts_solr_plugin_version = DOVECOT_ABI_VERSION;
 struct http_client *solr_http_client = NULL;
 
@@ -37,6 +39,10 @@
} else if (str_begins(*tmp, "default_ns=")) {
set->default_ns_prefix =
p_strdup(user->pool, *tmp + 11);
+   } else if (str_begins(*tmp

Re: permissions for quota-status

[Aki Tuomi via dovecot]

 
 
  
   
  
  
   
On 14 April 2019 16:17 Andreas Meyer via dovecot <
dovecot@dovecot.org> wrote:
   
   

   
   

   
   
Hello!
   
   

   
   
doveconf --version 2.2.36
   
   

   
   
I want to move to a new server with dovecot but get an error
   
   
Error: service(quota-status): listen(*, 12340) failed: Permission denied
   
   

   
   
when I define
   
   

   
   
service quota-status {
   
   
executable = quota-status -p postfix
   
   
inet_listener {
   
   
port = 12340
   
   
}
   
   
client_limit = 1
   
   
}
   
   

   
   
private]# ll
   
   
insgesamt 0
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 anvil
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 bounce
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 defer
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 discard
   
   
srw---. 1 postfix postfix 0 14. Apr 15:08 dovecot-lmtp
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 error
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 lmtp
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 local
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxymap
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxywrite
   
   
srw-rw. 1 postfix postfix 0 14. Apr 14:58 quota-status
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 relay
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 retry
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 rewrite
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 scache
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 smtp
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 tlsmgr
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 trace
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 verify
   
   
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 virtual
   
   

   
   
Don't know what's wrong. Can someone help?
   
   

   
   
Kind regards
   
   

   
   
Andreas
   
   
--
   
   
PGP-Fingerprint: D392 5D21 0299 63D7 5BAE 4562 1E56 B2EA 81A2 59F1
   
  
  
   
  
  
   Check /var/log/audit/audit.log - selinux is probably the culprit.
  
  
   ---
Aki Tuomi
   
 

permissions for quota-status

[Andreas Meyer via dovecot]

Hello!

doveconf --version 2.2.36

I want to move to a new server with dovecot but get an error
Error: service(quota-status): listen(*, 12340) failed: Permission denied

when I define

service quota-status {
executable = quota-status -p postfix
inet_listener {
port = 12340
}
client_limit = 1
}

private]# ll
insgesamt 0
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 anvil
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 bounce
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 defer
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 discard
srw---. 1 postfix postfix 0 14. Apr 15:08 dovecot-lmtp
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 error
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 lmtp
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 local
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxymap
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxywrite
srw-rw. 1 postfix postfix 0 14. Apr 14:58 quota-status
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 relay
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 retry
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 rewrite
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 scache
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 smtp
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 tlsmgr
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 trace
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 verify
srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 virtual

Don't know what's wrong. Can someone help?

Kind regards

  Andreas
-- 
PGP-Fingerprint: D392 5D21 0299 63D7 5BAE 4562 1E56 B2EA 81A2 59F1


pgpKO0kF7L5_K.pgp
Description: Digitale Signatur von OpenPGP

Re: FTS delays

[Joan Moreau via dovecot]

I have tried to spend some time of understanding the logic (if any !) of
the fts part 


Honestly, the one who created this mess shall be the one to fix it, or
one shall refactor it totally. 

Basically, the fts "core" should be able to do 

- select the backend according to conf file 

- send new emails/maiblox to backend 

- send teh ID of the emails to be removed 

- resend an entire mailbox ('rescan') 


- send the search parameters (from client) to backend and return the
email to front end based on backend results (and NOTHING more) 

Today, the fts part is plain wong and must be totally reviewed. 


I do not have the time but I can participate in testing if someone is
ready to roll up its sleeves on teh mater 


THe "loop" part seems the most urgent : It breaks everything (search
timeout 100% of the time) 


On 2019-04-06 09:56, Joan Moreau via dovecot wrote:


For the point 1, this is not "suboptimal", it is plain wrong (results are damn 
wrong ! and this is not related to the backend, but the FTS logic in Dovecot core)

For the point 2 , this has been discussed already numerous times but without action. The dovecot core shall be the one re-submitting the emails to scan, not the backend to try to figure out where and which are the emails to be re-scaned 

For the point 3, I will do a bit of research in the existing code and will get back to you 

For the point 4, this is random. FTS backend (xapian, lucene, solr, whatever..) returns X, then dovecot core choose to select only Y emails. THis is a clear bug. 

On 2019-04-05 20:08, Josef 'Jeff' Sipek via dovecot wrote: 
On Fri, Apr 05, 2019 at 19:33:57 +0800, Joan Moreau via dovecot wrote: Hi 

If you plan to fix the FTS part of Dovecot, I will be very gratefull. 
I'm trying to figure out what is causing the 3rd issue you listed, so we can

decide how severe it is and therefore how quickly it needs to be fixed.  At
the moment we are unable to reproduce it, and therefore we cannot fix it.

Not sure this is related to any specific commit but rahter the overall
design 
Ok.


The list of bugs so far 


1 - Double call to fts plugins with inconsistent parameter (first call
diferent from second call for the same request) 
Understood.  It is my understanding that this is simply suboptimal rather

than causing crashes/etc.

2 - "Rescan" features for now consists of deleting indexes. SHall be
resending emails to rescan to the fts plugin instead 
I'm not sure I follow.  The rescan operation is invoked on the fts backend

and it is up to the implementation to somehow ensure that after it is done
the fts index is up to date.  The easiest way to implement it is to simply
delete the fts index and re-index all the mails.  That is what currently
happens in the solr backend.

The lucene fts backend does a more complicated matching of the fts index
with the emails.  Finally, the deprecated squat backend seem to ignore the
rescan requests (its rescan vfunc is NULL).

3 - the loop when body search (just do a "doveadm search -u user@domain
mailbox inbox text whatevertexte") 


Refer to my email to Timo on 2019-04-03 18:30 on the same thread for bug
details 

(especially the loop) 
This seems to be the most important of the 4 issues you listed, so I'd like

to focus on this one for now.

As I mentioned, we cannot reproduce this ourselves.  So, we need your help
to narrow things down.  Therefore, can you give us the commit hashes of
revisions that you know are good and which are bad?  You can use git-bisect
to narrow the range down.

4 - Most notably, I notice that header search usually does not care
about fts plugin (even with fts_enforced) and rely on some internal
search , which si total non-sense 
You're right, that doesn't seem to make sense.  Can you provide a test case?


Jeff.

Let me know how can I help on thos 4 points 


On 2019-04-05 18:37, Josef 'Jeff' Sipek wrote:

On Fri, Apr 05, 2019 at 17:45:36 +0800, Joan Moreau wrote: 

I am on master (very latest) 

No clue exactly when this problem appears, but 


1 - the "request twice the fts plugin instead of once" issue has always
been there (since my first RC release of fts-xapian) 
Ok, good to know.


2 - the body/text loop has appeared recently (maybe during the month of
March) 
Our testing doesn't seem to be able to reproduce this.  Can you try to

git-bisect this to find which commit broke it?

Thanks,

Jeff.

On 2019-04-05 16:36, Josef 'Jeff' Sipek via dovecot wrote:

On Wed, Apr 03, 2019 at 19:02:52 +0800, Joan Moreau via dovecot wrote: 

issue seems in the Git version : 
Which git revision?


Before you updated to the broken revision, which revision/version were you
running?

Can you try it with 5f6e39c50ec79ba8847b2fdb571a9152c71cd1b6 (the commit
just before the fts_enforced=body introduction)?  That's the only recent fts
change.

Thanks,

Jeff.

On 2019-04-03 18:58, @lbutlr via dovecot wrote:

On 3 Apr 2019, at 04:30, Joan Moreau via dovecot  wrote: 

doveadm search -u j...@grosjo.net mailbox inbox text mil

nginx configuration to pass x-originating-ip

[André Rodier via dovecot]

Hello,

There is a bug in SOGo, as it sends the original IP after successful
login, and not before the login process. I traced the bug to the source
code. https://sogo.nu/bugs/view.php?id=2979.

Then, in my research, I found this old thread:

https://forum.nginx.org/read.php?29,237299,237367#msg-237367

Can I use Nginx as an IMAP proxy to add the missing ID. I suspect this
is something that can be achieved?

I already have an Nginx for the SOGo front-end, so I would add another
one for the IMAP proxy. With a little bit of luck, I should be able to
pass the original IP between the two servers

Does anyone has a working example?

Thanks,
André

-- 
André Rodier
HomeBox: https://github.com/progmaticltd/homebox

Re: Extended logging / moved mails jumping back

[Reto Brunner via dovecot]

On Sun, Apr 14, 2019 at 12:04:36PM +0200, Martin Müller via dovecot wrote:
> relay=dovecot, delay=0.13, delays=0.07/0/0/0.06, dsn=4.3.0, status=deferred
> (temporary failure. Command output: Can't open log file
> /var/log/mail.dovecot-error: Permission denied )
>[...]
> Here the output of ls -la /var/log/mail.dovecot-error
> -rw-r--r-- 1 root root   21259 Apr 14 11:24 /var/log/mail.dovecot-error
>[...]
> Any hints for me?

Well, fix the permission errors?
Give write access to the docecot user (or whatever you use) for the log file.

Also take care if you use the systemd service, there may be other restrictions 
in place (ProtectSystem etc)

Re: Extended logging / moved mails jumping back

[Martin Müller via dovecot]

Hi!


Now I have to check, if this a Thunderbird-Issue or is this a 
dovecot-issue. For that reason, I will activate the extended logging 
of dovecot.


I cant see such events in the logfiles. Which switch is to turn on to 
log all events?
Or do anyone know the reason for the annoying 
"copy/move-the-mail-back"-issue?


Thank you in advance for all inputs.

regards, martin


You are missing

mail_plugins = $mail_plugins notify mail_log


Thank you - it didnt work for me (yet). But I think there is another 
problem which I have to solve first. I recognised that the maildelivery 
stops after turning on the logging to the three files.


The incomming mails are held in the mailq, in syslog appears

relay=dovecot, delay=0.13, delays=0.07/0/0/0.06, dsn=4.3.0, 
status=deferred (temporary failure. Command output: Can't open log file 
/var/log/mail.dovecot-error: Permission denied )


In /var/log/mail.dovecot-error are lines like
2019-04-14 10:16:18 auth: Warning: auth client 0 disconnected with 1 
pending requests: Connection reset by peer


Here the output of ls -la /var/log/mail.dovecot-error
-rw-r--r-- 1 root root   21259 Apr 14 11:24 /var/log/mail.dovecot-error


If I turn the extended logging off, all works fine.
`postqueue -f` releases all held mails to their boxes.

Any hints for me?


Thank you!

martin

Re: Post login scripts environment

[André Rodier via dovecot]

On Sun, 2019-04-07 at 19:03 +0300, Aki Tuomi wrote:
> > On 7 April 2019 18:55 Aki Tuomi via dovecot  wrote:
> > 
> >  
> > > On 7 April 2019 18:45 André Rodier via dovecot  
> > > wrote:
> > > 
> > >  
> > > On Sun, 2019-04-07 at 17:49 +0300, Aki Tuomi via dovecot wrote:
> > > > > On 7 April 2019 17:26 André Rodier via dovecot < dovecot@dovecot.org> 
> > > > > wrote:
> > > > > 
> > > > > 
> > > > > Dear Dovecot users,
> > > > > 
> > > > > I am running Dovecot 2.2.27 on Debian Stretch, no issue so far.
> > > > > 
> > > > > I wonder if there is a way to pass the remote IP address, in an
> > > > > environment variable, in the post login script.
> > > > > 
> > > > > My Post login scripts are working well, except that when the server is
> > > > > accessed through a webmail (Roundcube or SOGo), the remote IP address
> > > > > is systematically 127.0.0.1.
> > > > > 
> > > > > The other question I have is, is it possible to pass the user agent of
> > > > > the email client used to access the server? I know this can be easily
> > > > > forged, but I would like to log it.
> > > > > 
> > > > > Thanks for your insight.
> > > > > 
> > > > > --
> > > > > André Rodier
> > > > 
> > > > You can use IMAP ID command to pass e.g. x-originating-ip. See 
> > > > https://github.com/dovecot/core/blob/master/src/imap-login/imap-login-cmd-id.c
> > > > ---
> > > > Aki Tuomi
> > > 
> > > Thanks, Aki,
> > > 
> > > I had a look on the version, I don't think this is implemented in
> > > 2.2.27, it it seems this file has been added in 2.3.
> > > 
> > > I may have to use a more recent version of Dovecot, but I think this is
> > > exactly what I was looking for, for the IP address.
> > > 
> > > Regarding the original user agent (e.g. Evolution, Thunderbird, etc.),
> > > I suppose I can use the same approach?
> > > 
> > > Thanks again for your help.
> > > 
> > > André
> > > 
> > 
> > This feature is supported since 1.2 alpha.
> > 
> > Aki
> 
> See https://wiki2.dovecot.org/Design/ParameterForwarding for more details on 
> this feature. I forgot to link this in the original reply.
> 
> Aki

Dear Aki et al,

Thank you, this is working perfectly, at least with a simple RoundCube
plugin.

For those who need the same as me, here a minimal example plug-in with
RoundCube:

==<
?php
class dovecot_ident extends rcube_plugin
{
function init()
{
$this->add_hook('storage_connect', [$this, 'add_ident']);
}

function add_ident($args)
{
$remoteIP = $_SERVER['REMOTE_ADDR'];
$identInfo = [ 'x-originating-ip' => $remoteIP ];

if ($args['ident']) {
$args['ident'] = array_merge($args['ident'], $identInfo);
} else {
$args['ident'] = $identInfo;
}
return $args;
}
}
?>
==

I am struggling to obtain answers from SOGo, but eventually I will get
there.

Maybe there is a way with imapproxy and an nginx setting ?

Kind regards,
André

-- 
André Rodier
HomeBox: https://github.com/progmaticltd/homebox