question about director_mail_servers setting
Hello, I have a question about director_mail_servers setting for director, I am aware that I can define a list of dovecot backend hostname there but I am not sure if the hashing apply to the hostname or underlying IP address of each hostname given? What if the IP of a specify hostname is changed? Will director pickup the new IP automatically? Will the same user hashed into the original IP still be mapped back to the new IP in this case? - Joe
Re: FTS-lucene errors : language not available for stemming
On 2020-05-19 16:28, Aki Tuomi wrote: Also if you were looking carefully what happens, you'd notice dovecot calls seteuid() before actually doing the indexing work. would be more sense in to have a dovecot shell that all commands must be issued in, like postgresql, setuid is nice still but its not well known it happends the same as apache starts as root for port under 1024 but after start a fork that is not running as root
Re: FTS-lucene errors : language not available for stemming
On 2020-05-19 16:22, Aki Tuomi wrote: Thats doveadm though, not indexer. and dovecot allows it
Re: FTS-lucene errors : language not available for stemming
On 2020-05-19 16:18, Stuart Henderson wrote: It does in the not entirely uncommon case where you have setup dovecot-fts-xapian, have multiple system users rather than a single uid owning all mailboxes, and need to index all mailboxes. PID USERNAME PRI NICE SIZE RES STATE WAIT TIMECPU COMMAND 44468 root 20 11M 18M sleep/6 netio 1:41 68.36% doveadm index -A * With solr the indexing is done out-of-process and typically under a safe uid. no doveconf -n, no problem if you really have a bug it should be solved i think you could su --user=non-root-user doveadm index -A to not have it run as root, it works well for fangfrisch, dovecot should not allow commands as root without thinking of consequences
Re: Problems with userdb and mail deliveribility
Don't file has /etc/dovecot/password and rodeo is a available at /etc/mail/password ?? Best regards, Muthu On Sun, 17 May, 2020, 8:53 PM Chris Bennett, wrote: > I briefly got some mails delivered, but not anymore. > Attached are dovecot-info.log, dovecot-debug.log, > dovecot.log, maillog, /etc/mail/passwd > smtpd.conf, vaddr, vusers, vdomains > > Thanks for any help. > > Chris Bennett > > >
Re: FTS-lucene errors : language not available for stemming
On 19/05/2020 17:22 Aki Tuomi wrote: On 19/05/2020 17:18 Stuart Hendersonwrote: On 2020/05/19 17:04, Aki Tuomi wrote: On 19/05/2020 16:48 Stuart Henderson wrote: On 2020-05-19, David Gessel wrote: I'm getting some log errors with clucene that I am having no luck tracking down on the interwebs. This looks relevant: https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html I am considering switch to xapian (solr and java... pls noe) as the port is quite tempting from an ease of integration perspective, but the easiest solution would be to resolve these odd indexing errors. Anyone have a clue? dovecot-fts-xapian is easy to configure, but has a big downside compared to solr in that the indexer runs as root. Dovecot indexer does not run as root. --- Aki Tuomi It does in the not entirely uncommon case where you have setup dovecot-fts-xapian, have multiple system users rather than a single uid owning all mailboxes, and need to index all mailboxes. PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND 44468 root 2 0 11M 18M sleep/6 netio 1:41 68.36% doveadm index -A * With solr the indexing is done out-of-process and typically under a safe uid. Thats doveadm though, not indexer. --- Aki Tuomi Also if you were looking carefully what happens, you'd notice dovecot calls seteuid() before actually doing the indexing work. --- Aki Tuomi
Re: FTS-lucene errors : language not available for stemming
On 19/05/2020 17:18 Stuart Hendersonwrote: On 2020/05/19 17:04, Aki Tuomi wrote: On 19/05/2020 16:48 Stuart Henderson wrote: On 2020-05-19, David Gessel wrote: I'm getting some log errors with clucene that I am having no luck tracking down on the interwebs. This looks relevant: https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html I am considering switch to xapian (solr and java... pls noe) as the port is quite tempting from an ease of integration perspective, but the easiest solution would be to resolve these odd indexing errors. Anyone have a clue? dovecot-fts-xapian is easy to configure, but has a big downside compared to solr in that the indexer runs as root. Dovecot indexer does not run as root. --- Aki Tuomi It does in the not entirely uncommon case where you have setup dovecot-fts-xapian, have multiple system users rather than a single uid owning all mailboxes, and need to index all mailboxes. PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND 44468 root 2 0 11M 18M sleep/6 netio 1:41 68.36% doveadm index -A * With solr the indexing is done out-of-process and typically under a safe uid. Thats doveadm though, not indexer. --- Aki Tuomi
Re: FTS-lucene errors : language not available for stemming
On 2020/05/19 17:04, Aki Tuomi wrote: > > On 19/05/2020 16:48 Stuart Henderson wrote: > > > On 2020-05-19, David Gessel wrote: > > I'm getting some log errors with clucene that I am having no luck > tracking down on the > interwebs. > > This looks relevant: > > https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html > > > I am considering switch to xapian (solr and java... pls noe) as the > port is quite tempting from an ease of integration perspective, but > the > easiest solution would be to resolve these odd indexing errors. > Anyone > have a clue? > > dovecot-fts-xapian is easy to configure, but has a big downside compared > to solr in that the indexer runs as root. > > Dovecot indexer does not run as root. > > --- > Aki Tuomi > It does in the not entirely uncommon case where you have setup dovecot-fts-xapian, have multiple system users rather than a single uid owning all mailboxes, and need to index all mailboxes. PID USERNAME PRI NICE SIZE RES STATE WAIT TIMECPU COMMAND 44468 root 20 11M 18M sleep/6 netio 1:41 68.36% doveadm index -A * With solr the indexing is done out-of-process and typically under a safe uid.
Re: Problem with tika
Greetings, Is the anyone who could please help me shed light on this? Arjen On 06-03-2020 09:52, Arjen Heidinga wrote: > Hello all, > > For some time now we've bin using solr as a search engine (working > great). I have added tika for searching inside documents, however it > keeps crashing when indexing. When just indexing mails that arrive works > good, but a reindex on all mail constantly crashes with below stacktrace. > > When I observe the packets with wireshark I see a HTTP-flow going to > tika and suddenly, midway a document dovecot (or the server) sends a > RST. Sometimes after thousands of documents, sometimes after a few. > Usually after a few hundred. > > It appears there are less errors using http than https. > > > > Relevant config: > > OS: CentOS6, fully updated > > plugin { > fts = solr > batch_size = 1 > fts_solr = > url=https://username:passw...@solr-01.vevida.net:443/solr/dovecot/ > #fts_tika = https://solr-01.vevida.net:443/tika/ > batch_size = 1000 > fts_autoindex=yes > soft_commit=no > } > > # dovecot --version > 2.3.9.3 (9f41b88fa) > > # Configure options: > > --docdir=%{_docdir}/dovecot \ > --disable-static \ > --with-nss \ > --with-shadow \ > --with-pam \ > --with-gssapi=plugin \ > --with-ldap=plugin \ > --with-sql=plugin \ > --with-pgsql \ > --with-sqlite \ > --with-zlib \ > --with-bzlib \ > --with-lzma \ > --with-libcap \ > --with-ssl=openssl \ > --with-ssldir=%{ssldir} \ > --with-solr \ > --with-docs > > # It is compiled agains the latest openssl > > # Tika and Solr: Both latest versions. > > #Stacktrace: > > doveadm(i...@samenmetrenske.nl): Info: Sent: Caching mails seq=1..161 > > doveadm(i...@samenmetrenske.nl): Panic: file http-client-request.c: line > 1173 (http_client_request_send_more): assertion failed: > (req->payload_input != NULL) > > doveadm(info@xxx.x): Error: Raw backtrace: > /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x2f) > [0x7f95d805acbf] -> > /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x26) [0x7f95d805add6] > -> /usr/lib64/dovecot/libdovecot.so.0(+0xe90ba) [0x7f95d80660ba] -> > /usr/lib64/dovecot/libdovecot.so.0(+0xe9161) [0x7f95d8066161] -> > /usr/lib64/dovecot/libdovecot.so.0(+0x41158) [0x7f95d7fbe158] -> > /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_more+0x424) > [0x7f95d8005094] -> > /usr/lib64/dovecot/libdovecot.so.0(http_client_connection_output+0x11a) > [0x7f95d800a24a] -> > /usr/lib64/dovecot/libssl_iostream_openssl.so(+0x8f6a) [0x7f95d57a2f6a] > -> /usr/lib64/dovecot/libdovecot.so.0(+0x114483) [0x7f95d8091483] -> > /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x61) > [0x7f95d807e581] -> > /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xdc) > [0x7f95d808076c] -> > /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5c) > [0x7f95d807e67c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) > [0x7f95d807e8c8] -> /usr/lib64/dovecot/libdovecot.so.0(+0x89105) > [0x7f95d8006105] -> > /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_payload+0x1f) > [0x7f95d80063cf] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xd31d) > [0x7f95d6ad931d] -> > /usr/lib64/dovecot/lib20_fts_plugin.so(fts_parser_more+0x1a) > [0x7f95d6ad83ca] -> > /usr/lib64/dovecot/lib20_fts_plugin.so(fts_build_mail+0x761) > [0x7f95d6ad6401] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x114ca) > [0x7f95d6add4ca] -> > /usr/lib64/dovecot/libdovecot-storage.so.0(mail_precache+0x2a) > [0x7f95d835ab4a] -> doveadm(+0x31e75) [0x55e7c1052e75] -> > doveadm(+0x321fb) [0x55e7c10531fb] -> doveadm(+0x2c321) [0x55e7c104d321] > -> doveadm(+0x2c577) [0x55e7c104d577] -> > doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x1e8) [0x55e7c104ec38] -> > doveadm(doveadm_cmd_run_ver2+0x52e) [0x55e7c105fafe] -> > doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55e7c105fb97] -> > doveadm(main+0x21a) [0x55e7c1062aca] -> > /lib64/libc.so.6(__libc_start_main+0x100) [0x7f95d7983d20] -> > doveadm(+0x1c479) [0x55e7c103d479] > signature.asc Description: OpenPGP digital signature
Re: FTS-lucene errors : language not available for stemming
On 19/05/2020 16:48 Stuart Hendersonwrote: On 2020-05-19, David Gessel wrote: I'm getting some log errors with clucene that I am having no luck tracking down on the interwebs. This looks relevant: https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html I am considering switch to xapian (solr and java... pls noe) as the port is quite tempting from an ease of integration perspective, but the easiest solution would be to resolve these odd indexing errors. Anyone have a clue? dovecot-fts-xapian is easy to configure, but has a big downside compared to solr in that the indexer runs as root. Dovecot indexer does not run as root. --- Aki Tuomi
Re: FTS-lucene errors : language not available for stemming
On 2020-05-19, David Gessel wrote: > I'm getting some log errors with clucene that I am having no luck tracking > down on the interwebs. This looks relevant: https://www.mail-archive.com/dovecot@dovecot.org/msg66366.html > I am considering switch to xapian (solr and java... pls noe) as the > port is quite tempting from an ease of integration perspective, but the > easiest solution would be to resolve these odd indexing errors. Anyone > have a clue? dovecot-fts-xapian is easy to configure, but has a big downside compared to solr in that the indexer runs as root.
Re: FTS-lucene errors : language not available for stemming
On 19.05.20 15:15, David Gessel wrote: I'm getting some log errors with clucene that I am having no luck tracking down on the interwebs. Errors: May 19 05:05:16 indexer-worker(ges...@blackrosetech.com)<62971>: Error: lucene index /mail/blackrosetech.com/gessel//lucene-indexes: IndexWriter::addDocument() failed (#4): language not available for stemming May 19 05:05:16 indexer-worker: Error: May 19 05:05:16 indexer-worker(ges...@blackrosetech.com)<62971>: Error: Mailbox Security: Mail search failed: Internal error occurred. Refer to server log for more information. [2020-05-19 05:05:16] May 19 05:05:16 indexer-worker(ges...@blackrosetech.com)<62971>: Error: Mailbox Security: Transaction commit failed: FTS transaction commit failed: transaction context (attempted to index 1 messages (UIDs 152736..152736)) Config: FreeBSD 11.3-RELEASE-p8 #0 r360490 dovecot-2.3.10_3 clucene-2.3.3.4_19 py37-pystemmer-2.0.0.1 py37-snowballstemmer-1.2.1 icu-67.1,1 plugin { #setting_name = value expire = Trash mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size fts_autoindex=yes #zlib_save_level = 6 # 1..9 #zlib_save = gz # or bz2 } plugin { fts = lucene # Lucene-specific settings, good ones are: fts_lucene = whitespace_chars=@. mime_parts } I am considering switch to xapian (solr and java... pls noe) as the port is quite tempting from an ease of integration perspective, but the easiest solution would be to resolve these odd indexing errors. Anyone have a clue? I ran into the same problem a few weeks back. The workaround I found was to add no_snowball to fts_lucene. It disables the snowball algorithm.
FTS-lucene errors : language not available for stemming
I'm getting some log errors with clucene that I am having no luck tracking down on the interwebs. Errors: May 19 05:05:16 indexer-worker(ges...@blackrosetech.com)<62971>: Error: lucene index /mail/blackrosetech.com/gessel//lucene-indexes: IndexWriter::addDocument() failed (#4): language not available for stemming May 19 05:05:16 indexer-worker: Error: May 19 05:05:16 indexer-worker(ges...@blackrosetech.com)<62971>: Error: Mailbox Security: Mail search failed: Internal error occurred. Refer to server log for more information. [2020-05-19 05:05:16] May 19 05:05:16 indexer-worker(ges...@blackrosetech.com)<62971>: Error: Mailbox Security: Transaction commit failed: FTS transaction commit failed: transaction context (attempted to index 1 messages (UIDs 152736..152736)) Config: FreeBSD 11.3-RELEASE-p8 #0 r360490 dovecot-2.3.10_3 clucene-2.3.3.4_19 py37-pystemmer-2.0.0.1 py37-snowballstemmer-1.2.1 icu-67.1,1 plugin { #setting_name = value expire = Trash mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size fts_autoindex=yes #zlib_save_level = 6 # 1..9 #zlib_save = gz # or bz2 } plugin { fts = lucene # Lucene-specific settings, good ones are: fts_lucene = whitespace_chars=@. mime_parts } I am considering switch to xapian (solr and java... pls noe) as the port is quite tempting from an ease of integration perspective, but the easiest solution would be to resolve these odd indexing errors. Anyone have a clue? -David
doveadm backup, question
Hello everyone. I am an inexperienced user of dovecot and I faced probably a simple problem, but I still do not understand how to solve it. The issue is related to using doveadm backup - I want to backup mailboxes; make this copy on the same machine (hereinafter, files with backup copies will be taken from this machine into the storage by a separate mechanism, this is not related to the issue). I am using dovecot 2.2.27 (c0f36b0) on Debian GNU/Linux 9 (stretch). The options related to storing mail and attachments are as follows: mail_location = mdbox:/var/mail/%d/%n mail_attachment_dir = /var/mail/%d/%n/attach Thus, the mail of u...@domain.tld is stored in /var/mail/domain.tld/user/, and mail attachments inside this directory, for example: /var/mail/domain.tld/user/attach. When I start copying (as root while in test mode): #doveadm backup -u "u...@domain.tld" sdbox:/backup/domain.tld/user, copying passes, but, as I understand it, the attachments are not copied (I am judging so far by the volume of the initial storage and the received one, it should not differ VERY much, but it differs by 10 times, it seems that it’s just the total amount of attachments). Tell me, plz, how to run doveadm backup correctly so that everything is copied so that you can subsequently deploy it from the backup? Thanks.
Re: Sieve addheader directive works only partially
Hi Stephan, Sorry for the delay - this was on version 2.3.4.1 (f79e8e7e4). Relevant bits from the config: # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.4 () # OS: Linux 4.19.0-9-amd64 x86_64 Debian 10.4 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader imapsieve vnd.dovecot.imapsieve sieve_extensions = +editheader sieve_global_extensions = +vnd.dovecot.pipe sieve_pipe_bin_dir = /usr/bin sieve_plugins = sieve_imapsieve sieve_extprograms Thanks Peter > On May 15, 2020, at 3:49 PM, Stephan Bosch wrote: > > > > Op 13-5-2020 om 20:10 schreef Peter Folta: >> require “editheader”; >> >> if address :is "From” “special-sen...@example.com” { >> addheader “X-My-Custom-Header” “Header Value"; >> } >> >> I have enabled the editheader extension in my Dovecot configuration: >> >> plugin { >> … >> sieve_extensions = +editheader >> } > > What version is this? Please provide output of `dovecot -n`. > > This smells a lot like an age-old bug that was fixed long ago. > > It works fine with master. > > Regards, > > Stephan.