date:20190919

Re: fts_solr: Error: fts_solr: received invalid uid '0'

2019-09-19 Thread Daniel Miller via dovecot


On 9/19/2019 6:28 AM, Fabian via dovecot wrote:


Thanks for your response! No we are not limiting Soli’s memory usage. After your tip, we've 
also upgraded the memory to 32GB. But the behavior remains the same. I have also already 
considered that Dovecot may index the UID incorrectly. But if I search the index directly, I 
don't find any entries with UID = 0, so I have no idea where this "fts_solr: received 
invalid uid '0"" message might come from.

In our test environment we actually indexed only one user. The user's mailbox 
contains about 100.000 mails. This means that there is not really much data in 
the index.

Are there any other hints or tips regarding this „invalid uid ‚0‘"-message?

Logfile:



Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: peer 172.17.10.12:8983: Creating 1 new connections to handle 
requests (already 0 usable, connecting t$


Your post has truncated the lines (right margin). Re-post with the full 
lines.


--
Daniel

Second private namespace

2019-09-19 Thread André Rodier via dovecot

Hello,

I have Dovecot working, with a private namespace only.

I would like to know if the following is feasible.

- Add a second private namespace, in another directory scheme, for
instance /mnt/backup/homebox//Mails/...
- This namespace would be read-only. Perhaps the index and flags should
be stored in a different folder with r/w access (e.g. /var, or even
/tmp, it's not important)

Now, as a matter of fact, this folder would be mounted on demand, via
systemd-automount and fuse and is likely to be remote. This might
implies some flags or caching on Dovecot performance.

Thanks for your insights,
André

Re: fts_solr: Error: fts_solr: received invalid uid '0'

2019-09-19 Thread Fabian via dovecot

Hi, 

> Am 16.09.2019 um 09:39 schrieb Daniel Miller :
> 
> On 9/13/2019 1:21 AM, Fabian via dovecot wrote:
>> Hi,
>> we are trying to add full text search functionality with Solr to our Doveoct 
>> setup. Our Versions:
>> OS: Debian 9
>> Tried versions:
>> - Dovecot 2.2.7 with Solr 3.6
>> - Dovecot 2.3.4 with Solr 8.2
>> (2.2.7 from offical Debian repository, 2.3.4 from backports)
>> Search is working mostly of the time perfrectly smooth. But sometimes 
>> following message appears in mail.err:
>> dovecot: imap(username)<16189>: Error: 
>> fts_solr: received invalid uid '0'
>> If this error occurs our webmail frontend delivers most of the time a 
>> timeout. Sometimes the search only takes really long.
>> Are  there any ideas why this error occurs? We are not able to reproduce the 
>> error in such a way that it would always be reproducible. However, we can 
>> reproduce the behavior in some form over and over again - but we do not know 
>> exactly what is decisive.
> 
> Are you limiting Solr's memory usage? How much available memory is on your 
> server?
> 
> To shortcut the conversation - if you don't have at least 16G of *free* RAM 
> it's time to upgrade. My own server has 32G installed - I used to have 16G. 
> My own Solr problems basically disappeared after adding RAM. And I only serve 
> a few users - my own mailstore is the largest as I keep most of my mails. If 
> you're serving 20+ users you'd probably benefit from doubling to at least 64G.
> 

Thanks for your response! No we are not limiting Soli’s memory usage. After 
your tip, we've also upgraded the memory to 32GB. But the behavior remains the 
same. I have also already considered that Dovecot may index the UID 
incorrectly. But if I search the index directly, I don't find any entries with 
UID = 0, so I have no idea where this "fts_solr: received invalid uid '0"" 
message might come from.

In our test environment we actually indexed only one user. The user's mailbox 
contains about 100.000 mails. This means that there is not really much data in 
the index.

Are there any other hints or tips regarding this „invalid uid ‚0‘"-message?

Logfile:
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: peer 172.17.10.12:8983: Linked queue 
http://172.17.10.12:8983 (1 queues linked)
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: queue http://172.17.10.12:8983: Started new connection 
to 172.17.10.12:8983
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: request [Req1: GET 
http://172.17.10.12/solr/dovecot/select?wt=xml&fl=box,uid,score&rows=10&sort=b$
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: Waiting for 1 requests to finish
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: peer 172.17.10.12:8983: Creating 1 new connections to 
handle requests (already 0 usable, connecting t$
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: peer 172.17.10.12:8983: Making new connection 1 of 1 (0 
connections exist, 0 pending)
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: conn 172.17.10.12:8983 [0]: (172.17.10.12:8983): 
Connecting
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: conn 172.17.10.12:8983 [0]: (172.17.10.12:8983): Waiting 
for connect (fd=26) to finish for max 0 msecs
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: conn 172.17.10.12:8983 [0]: HTTP connection created (1 
parallel connections exist)
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: conn 172.17.10.12:8983 [0]: (172.17.10.12:8983): Client 
connected (fd=26)
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: conn 172.17.10.12:8983 [0]: Connected
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: conn 172.17.10.12:8983 [0]: Ready for requests
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: peer 172.17.10.12:8983: Successfully connected (1 
connections exist, 0 pending)
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client: peer 172.17.10.12:8983: Successfully connected (1 
connections exist, 0 pending)
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: peer 172.17.10.12:8983: Using 1 idle connections to 
handle 1 requests (1 total connections ready)
Sep 16 08:35:27 mailservertest dovecot: imap(user01)<30204><+IjNzqWS2s2sEQoK>: 
Debug: http-client[1]: queue http://172.17.10.12:8983: Con

Re: Recent Dovecot on old operating system

2019-09-19 Thread Maciej Milaszewski IQ PL via dovecot

Hi
For test in my smal testing lab i downlod latest dovecot-2.2.36.4 and
debian 7.x

./configure --prefix=/usr/local/dovecot-2.2.36.4 --sysconfdir=/etc
--with-mysql --with-ssl=openssl --with-solr --with-storages=maildir,imapc

and make

working ok

debian 7:
dovecot-director

debian 8:
dovecot node

and tested like

debian 8:
dovecot-director

debian 7:
dovecot node

All works fine

On 19.09.2019 12:28, Gerald via dovecot wrote:
> Hi,
>
> sorry for the dumb question and please ignore this post if you think it's far 
> beyond; i know it's not the way to go, but for reasons ...
>
> Has anyone running a self compiled recent dovecot (2.2.36.4) on Debian-7 and 
> does it work?
>  Or thinks it should work.
>
> Surprisingly it actually compiles flawlessly on Debian-7, but i wonder wether 
> it will become a complete mess replacingthe existing dovecot (2.1.7) with the 
> new one, serving a few dozens users and any kind of mail clients.
>
> Libraries of the compiled Dovecot:
> $ ldd ./src/master/.libs/dovecotlinux-vdso.so.1 =>  
> (0x7ffeacfb1000)
> libdovecot.so.0 => 
> /usr/local/00-DBAI/dovecot/lib/dovecot/libdovecot.so.0 (0x7f557b2a9000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f557af1c000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f557ad18000)
> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f557ab1)
> /lib64/ld-linux-x86-64.so.2 (0x7f557b5dc000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
> (0x7f557a8f4000)
>
> libc6 versions:Debian-7: 2.13-38Debian-9: 2.24-11
>
> thanks so far, gerald 
>
>
>
>
>

Recent Dovecot on old operating system

2019-09-19 Thread Gerald via dovecot

Hi,

sorry for the dumb question and please ignore this post if you think it's far 
beyond; i know it's not the way to go, but for reasons ...

Has anyone running a self compiled recent dovecot (2.2.36.4) on Debian-7 and 
does it work?
 Or thinks it should work.

Surprisingly it actually compiles flawlessly on Debian-7, but i wonder wether 
it will become a complete mess replacingthe existing dovecot (2.1.7) with the 
new one, serving a few dozens users and any kind of mail clients.

Libraries of the compiled Dovecot:
$ ldd ./src/master/.libs/dovecotlinux-vdso.so.1 =>  (0x7ffeacfb1000)
libdovecot.so.0 => 
/usr/local/00-DBAI/dovecot/lib/dovecot/libdovecot.so.0 (0x7f557b2a9000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f557af1c000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f557ad18000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f557ab1)
/lib64/ld-linux-x86-64.so.2 (0x7f557b5dc000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x7f557a8f4000)

libc6 versions:Debian-7: 2.13-38Debian-9: 2.24-11

thanks so far, gerald

Re: Dovecot 2.3.7.2 lucene segfault and coredump

2019-09-19 Thread Domenico Pastore via dovecot

Hi,

i have find other information of the coredump.

My mail_plugin is:
mail_plugins = quota fts fts_lucene mail_log notify virtual

When I using quota and fts_lucene the coredump is generated.
If i use only the quota plugin or fts_lucene plugin, lmtp store work correctly.

Have you an ideas?

Br,
Domenico

Domenico Pastore
Senior Cloud Engineer
T 06.98269600 | M 347.1474270 | F 06.98269680
Par-Tec | beyond the IT domain
Via Cristoforo Colombo 163, 00147 Roma
CONFIDENZIALE: Questo messaggio ed i suoi allegati sono di carattere 
confidenziale. È vietato l'inoltro non autorizzato a destinatari diversi da 
quelli indicati nel messaggio originale. Se ricevuto per errore si prega di 
informare il mittente e cancellarlo immediatamente.
Il 16/09/19 17:20, Domenico Pastore via dovecot ha scritto:

Hi Stephan,

thanks for your replay.

This is bt full output:


(gdb) bt full
#0  clear (this=0x0) at 
/usr/src/debug/clucene-core-2.3.3.4/src/core/CLucene/util/VoidMap.h:123
No locals.
#1  lucene::search::FieldSortedHitQueue::_shutdown () at 
/usr/src/debug/clucene-core-2.3.3.4/src/core/CLucene/search/FieldSortedHitQueue.cpp:60
No locals.
#2  0x7f7dc5eabe96 in _lucene_shutdown () at 
/usr/src/debug/clucene-core-2.3.3.4/src/core/CLucene/StdHeader.cpp:33
No locals.
#3  0x7f7dc973bfbb in module_dir_deinit (modules=) at 
module-dir.c:578
_data_stack_cur_id = 4
module = 0x55ddb26268f0
rev = 0x55ddb25bd050
i = 
count = 6
#4  0x7f7dc973c001 in module_dir_unload 
(modules=modules@entry=0x7f7dc9d2a380 ) at 
module-dir.c:590
module = 
next = 
#5  0x7f7dc9a26264 in mail_storage_service_deinit 
(_ctx=_ctx@entry=0x55ddb25d9848) at mail-storage-service.c:1730
ctx = 0x55ddb2612c18
#6  0x55ddb0e0dd35 in doveadm_mail_cmd_exec (ctx=ctx@entry=0x55ddb25d9808, 
wildcard_user=wildcard_user@entry=0x0) at doveadm-mail.c:673
cctx = 
cli = 
ret = 
error = 0x7f7dc96762f7 <__x86_return_thunk+5> 
"\363\220\017\256\350\353\371H\215d$\b\303\350\a"
#7  0x55ddb0e0ebbd in doveadm_cmd_ver2_to_mail_cmd_wrapper (cctx=) at doveadm-mail.c:1144
mctx = 0x55ddb25d9808
wildcard_user = 
fieldstr = 0x55ddb25b9318 "\315\"N\305}\177"
pargv = {arr = {buffer = 0x55ddb25d9a38, element_size = 8}, v = 
0x55ddb25d9a38, v_modifiable = 0x55ddb25d9a38}
full_args = {arr = {buffer = , element_size = 8}, v = 
, v_modifiable = }
i = 
cli = 
tcp_server = 
mail_cmd = {alloc = 0x7f7dc54e1d00 , name = 
0x7f7dc54e22b6 "quota get", usage_args = 0x7f7dc54e2291 ""}
dashdash = 0x55ddb0e4a7ca "--"
args_pos = 1
#8  0x55ddb0e1fc98 in doveadm_cmd_run_ver2 (argc=argc@entry=1, 
argv=argv@entry=0x55ddb25b9158, cctx=cctx@entry=0x77c52670) at 
doveadm-cmd.c:574
param = 
pargv = {arr = {buffer = 0x55ddb25b9418, element_size = 104}, v = 
0x55ddb25b9418, v_modifiable = 0x55ddb25b9418}
opts = {arr = {buffer = , element_size = 32}, v = 
, v_modifiable = }
pargc = 4
c = -1
---Type  to continue, or q  to quit---
li = -252221570
pool = 0x55ddb25b9248
optbuf = 0x55ddb25b9260
__func__ = "doveadm_cmd_run_ver2"
#9  0x55ddb0e24020 in doveadm_cmd_server_run_ver2 (cctx=0x77c52670, 
argv=0x55ddb25b9158, argc=1, conn=0x55ddb260dbf8) at client-connection-tcp.c:178
No locals.
#10 doveadm_cmd_handle (cctx=0x77c52670, argv=0x55ddb25b9158, argc=1, 
cmd_name=, conn=0x55ddb260dbf8) at client-connection-tcp.c:342
mail_cmd = 
mctx = 
cmd_ver2 = 
prev_ioloop = 0x55ddb25c1cc0
cmd = 0x0
#11 client_handle_command (args=, conn=0x55ddb260dbf8) at 
client-connection-tcp.c:415
cctx = {cmd = 0x55ddb25d73e8, argc = 4, argv = 0x55ddb25b9450, username 
= 0x55ddb25b9119 
"testcli...@newsvilpec.babel.it",
 local_ip = {family = 2, u = {ip6 = {__in6_u = {
  __u6_addr8 = "\300\240\n\003", '\000' , 
__u6_addr16 = {41152, 778, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {51028160, 0, 0, 
0}}}, ip4 = {s_addr = 51028160}}}, remote_ip = {family = 2, u = {
  ip6 = {__in6_u = {__u6_addr8 = "\n\000\b\200", '\000' , __u6_addr16 = {10, 32776, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {2148007946, 
0, 0, 0}}}, ip4 = {s_addr = 2148007946}}},
  local_port = 924, remote_port = 41491, conn_type = 
DOVEADM_CONNECTION_TYPE_TCP, input = 0x55ddb25d2dc0, output = 0x55ddb25d8d00}
flags = 
cmd_name = 
argc = 
#12 client_connection_tcp_input (conn=0x55ddb260dbf8) at 
client-connection-tcp.c:561
args = 
_data_stack_cur_id = 3
line = 
ok = true
minor = 0
#13 0x7f7dc973155f in io_loop_call_io (io=0x55ddb25e7b20) at ioloop.c:704
ioloop = 0x55ddb25c1cc0
t_id = 2
__func__ = "io_loop_call_io"
#14 0x7f7dc9732fa3 in io_loop_handler_run_internal 
(iol

Re: fts-elastic plugin

2019-09-19 Thread Aki Tuomi via dovecot



> On 19/09/2019 11:24 Filip Hanes via dovecot  wrote:
> 
> 
> Hi all,
> I have recently worked on fts plugin for ElasticSearch.
> https://github.com/filiphanes/fts-elastic
> It is forked from fts-elasticsearch as you can see in PR 
> https://github.com/atkinsj/fts-elasticsearch/pull/21
> with following changes (maybe some more)
> 
>   * no duplicate json keys in indexed documents, so compatible with ES 7.x
>
>   * 1 elastic index for all users with routing by username
>
>   * _id field similar to fts-solr (uid/box_guid/user)
>   * better json response parsing
>
>   * no need to use _source
>   * added rawlog_dir to config and some more
>
>   * renamed all elasticsearch to elastic for shorter names
>
>   * fixed http_client leaks
>
>   * fixed some compile warnings
> I would appreciate if someone with better experience in c and dovecot code 
> would review my code and suggest/push changes.
> 
> In my opinion fts-elastic is now functionally on level as currently 
> recommended FTS plugin: fts-solr.
> I would like to implement proper rescan inspired by fts-lucene so it would 
> become the most complete fts plugin for dovecot.
> --
> Filip Hanes
>

Added link to your repository to our FTS documentation. Thank you for your 
contribution!

Aki

Re: fts-elastic plugin

2019-09-19 Thread Tom Sommer via dovecot


On 2019-09-19 10:24, Filip Hanes via dovecot wrote:


Hi all,
I have recently worked on fts plugin for ElasticSearch.
https://github.com/filiphanes/fts-elastic
It is forked from fts-elasticsearch as you can see in PR 
https://github.com/atkinsj/fts-elasticsearch/pull/21


In my opinion fts-elastic is now functionally on level as currently 
recommended FTS plugin: fts-solr.
I would like to implement proper rescan inspired by fts-lucene so it 
would become the most complete fts plugin for dovecot. --


Awesome. Thank you for this.

---
Tom

fts-elastic plugin

2019-09-19 Thread Filip Hanes via dovecot

Hi all,
I have recently worked on fts plugin for ElasticSearch.
https://github.com/filiphanes/fts-elastic
It is forked from fts-elasticsearch as you can see in PR
https://github.com/atkinsj/fts-elasticsearch/pull/21
with following changes (maybe some more)

   - no duplicate json keys in indexed documents, so compatible with ES 7.x
   - 1 elastic index for all users with routing by username
   - _id field similar to fts-solr (uid/box_guid/user)
   - better json response parsing
   - no need to use _source
   - added rawlog_dir to config and some more
   - renamed all elasticsearch to elastic for shorter names
   - fixed http_client leaks
   - fixed some compile warnings

I would appreciate if someone with better experience in c and dovecot code
would review my code and suggest/push changes.

In my opinion fts-elastic is now functionally on level as currently
recommended FTS plugin: fts-solr.
I would like to implement proper rescan inspired by fts-lucene so it would
become the most complete fts plugin for dovecot.
--
Filip Hanes

Re: fts_solr: Error: fts_solr: received invalid uid '0'

Second private namespace

Re: fts_solr: Error: fts_solr: received invalid uid '0'

Re: Recent Dovecot on old operating system

Recent Dovecot on old operating system

Re: Dovecot 2.3.7.2 lucene segfault and coredump

Re: fts-elastic plugin

Re: fts-elastic plugin

fts-elastic plugin

9 matches

Site Navigation

Mail list logo

Footer information