Re: Fwd: SOLR/Index?

[Larry Rosenman via dovecot]

Aki (via private chat) told me about adding
 override_fields = domain=lerctr.org

to my PAM passdb, and /etc/passwd userdb.

This solves the problem nicely.


On Mon, Apr 15, 2019 at 2:30 PM Tanstaafl via dovecot 
wrote:

> On 4/15/2019, 6:59:59 AM, Larry Rosenman via dovecot
>  wrote:
> > If I login to roundcube with @lerctr.org  it
> > finds the autoindexed mail.
> >
> > So, if I make everyone always authenticate as @lerctr.org
> >  we should be fine.
>
> You can configure roundcube to always use the fqdn...
>


-- 
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: Fwd: SOLR/Index?

[Tanstaafl via dovecot]

On 4/15/2019, 6:59:59 AM, Larry Rosenman via dovecot
 wrote:
> If I login to roundcube with @lerctr.org  it
> finds the autoindexed mail.
> 
> So, if I make everyone always authenticate as @lerctr.org
>  we should be fine.

You can configure roundcube to always use the fqdn...

Re: No CRLF in Pigeonhole's header?

[Aki Tuomi via dovecot]

> On 15 April 2019 21:11 Wojciech Sychut via dovecot  
> wrote:
> 
>  
> Hello,
> 
> I have been told on Freenode's #dovecot to ask here. I have Exim 4.92
> complaining about "Non-CRLF-terminated header, under CHUNKING: message
> abandoned" with redirects made by Pigeonhole 0.5.5 (with Dovecot
> 2.3.5.1). Please find relevant part of the traffic below, and help me
> determine whether Exim is correct on this.
>    0x:  4500 0954  4000 8006  7f00 0001  E..T..@.
>    0x0010:  7f00 0001 6ea2 0019 8982 e279 d090 b820  n..y
>    0x0020:  8018 021e 0749  0101 080a c345 2b51  .I...E+Q
>    0x0030:  b382 853c 4d41 494c 2046 524f 4d3a 3c72 ...    0x0040:  4073 3066 2e6d 642e 6465 7664 6576 6465  @sxf.md.devdevde
>    0x0050:  762e 6e65 743e 0d0a 5243 5054 2054 4f3a v.net>..RCPT.TO:
>    0x0060:  3c63 6f6e 7461 6374 4072 6567 6973 2e74     0x0070:  6563 683e 0d0a 4244 4154 2032 3235 3420 ech>..BDAT.2254.
>    0x0080:  4c41 5354 0d0a 582d 5369 6576 653a 2050  LAST..X-Sieve:.P
>    0x0090:  6967 656f 6e68 6f6c 6520 5369 6576 6520  igeonhole.Sieve.
>    0x00a0:  302e 352e 3520 2832 3438 3362 3038 3529  0.5.5.(2483b085)
>    0x00b0:  0a58 2d53 6965 7665 2d52 6564 6972 6563  .X-Sieve-Redirec
>    0x00c0:  7465 642d 4672 6f6d 3a20 7240 7330 662e  ted-From:.r@sxf.
> 
> With Dovecot 2.2.36 + Pigeonhole 0.4.24.1 and required minimum of
> version-related configuration changes - redirection works properly.
> Related Exim code fragment:
> https://github.com/Exim/exim/blob/master/src/src/receive.c#L
> 
> 
> Thank you,
> -- 
> Wojtek Sychut

Hi!

This is fixed with 
https://github.com/dovecot/core/compare/7b4cc472%5E..7a456f25.patch and will be 
part of 2.3.6 release.

Aki

No CRLF in Pigeonhole's header?

[Wojciech Sychut via dovecot]

Hello,

I have been told on Freenode's #dovecot to ask here. I have Exim 4.92
complaining about "Non-CRLF-terminated header, under CHUNKING: message
abandoned" with redirects made by Pigeonhole 0.5.5 (with Dovecot
2.3.5.1). Please find relevant part of the traffic below, and help me
determine whether Exim is correct on this.
  0x:  4500 0954  4000 8006  7f00 0001  E..T..@.
  0x0010:  7f00 0001 6ea2 0019 8982 e279 d090 b820  n..y
  0x0020:  8018 021e 0749  0101 080a c345 2b51  .I...E+Q
  0x0030:  b382 853c 4d41 494c 2046 524f 4d3a 3c72 .RCPT.TO:
  0x0060:  3c63 6f6e 7461 6374 4072 6567 6973 2e74 ..BDAT.2254.
  0x0080:  4c41 5354 0d0a 582d 5369 6576 653a 2050  LAST..X-Sieve:.P
  0x0090:  6967 656f 6e68 6f6c 6520 5369 6576 6520  igeonhole.Sieve.
  0x00a0:  302e 352e 3520 2832 3438 3362 3038 3529  0.5.5.(2483b085)
  0x00b0:  0a58 2d53 6965 7665 2d52 6564 6972 6563  .X-Sieve-Redirec
  0x00c0:  7465 642d 4672 6f6d 3a20 7240 7330 662e  ted-From:.r@sxf.

With Dovecot 2.2.36 + Pigeonhole 0.4.24.1 and required minimum of
version-related configuration changes - redirection works properly.
Related Exim code fragment:
https://github.com/Exim/exim/blob/master/src/src/receive.c#L


Thank you,
--
Wojtek Sychut

LDAP iterate (dovecoc 2.2.27)

[Debreczeni Attila via dovecot]

Dear Dovecot Team,

We would like to use the LDAP iterate for dovecot replication, but I got
this error:

dovecot: auth-worker(10149): Error: ldap(*):
ldap_search(base=ou=mail,o=asd,c=TLD filter=(objectClass=inetMailUser)
failed: Size limit exceeded
dovecot: replicator: Error: User listing returned failure
dovecot: replicator: Error: listing users failed, can't replicate
existing data

When I check the "doveadm replicator status":
...
Total number of known users   1200

But we have more than 10.000 users. When I run the ldapsearch manually,
I get the full list of the mail addresses.

Who can I fix it?

Thank you,
Attila

# dovecot --version
2.2.27 (c0f36b0)

# dovecot -n | filtered
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.8
auth_worker_max_count = 150
default_vsz_limit = 512 M
doveadm_password =  # hidden, use -P to show it
doveadm_port = ***
mail_gid = vmail
mail_plugins = " quota notify replication"
namespace inbox {
 ...
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  mail_replica = tcp:REMOTE_IP
}
replication_dsync_parameters = -d -N -1 -l 30 -U
service aggregator {
  fifo_listener replication-notify-fifo {
    user = vmail
  }
  unix_listener replication-notify {
    user = vmail
  }
}
service auth {
  inet_listener {
    address = LOCAL_IP
    port = LOCAL_PORT
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0600
    user = vmail
  }
}
service dict {
  unix_listener dict {
    group = vmail
    mode = 0600
    user = vmail
  }
}
service lmtp {
  inet_listener lmtp {
    address = LOCAL_IP
    port = LOCAL_PORT
  }
}

service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
    mode = 0600
  }
}

userdb {
  args = /etc/dovecot/dovecot-domain-ldap.conf.ext
  driver = ldap
  result_success = continue-ok
}
userdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
  result_failure = return-fail
}
protocol imap {
...
}
protocol pop3 {
...
}


# grep -v '^ *\(#.*\)\?$' dovecot-ldap.conf.ext
uris = ldaps://ldap.URL
dn = uid=auth,o=domain,c=TLD
dnpass = 
sasl_bind = no
tls_ca_cert_dir = /etc/ssl/certs
tls_require_cert = demand
ldap_version = 3
base = ou=mail,o=asd,c=TLD
deref = never
scope = subtree
user_attrs = =home=/dev/null/%Ld/%L{ldap:uid},
mailQuota=quota_rule=*:bytes=%$
user_filter = (objectClass=inetMailUser)
pass_attrs = userPassword=password
pass_filter = (objectClass=inetMailUser)
iterate_attrs = mail=user
iterate_filter = (objectClass=inetMailUser)
default_pass_scheme = PLAIN

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 16:34, Larry Rosenman via dovecot wrote:
> yes, either with or without the domain.
>
>
> On Mon, Apr 15, 2019 at 9:33 AM John Fawcett via dovecot
> mailto:dovecot@dovecot.org>> wrote:
>
> On 15/04/2019 16:05, Larry Rosenman via dovecot wrote:
>> I have normal system users, some and some virtual domains (2
>> different flavors).
>>
>> I have this job that runs from cron on my own userid to archive mail
>> I want Dovecot to use my system password for mail authentication.
>> I, therefore, use PAM to authenticate system users
>> Everything works great, modulo getting the auto-index to be
>> visible using RoundCube (which if I log into roundcube using
>> @ it does). 
>> by default if I'm logged in as my normal OS user, default doveadm
>> commands (as issued from the shell or from my archive script) use
>> the user I'm logged in as *WITHOUT A DOMAIN*.
>>
>> I want to default PAM auth'd users to append @lerctr.org
>>  (DOMAIN) to the OS user.
>>
> when they log in to roundcube as the OS user, right?
>
> John
>
>
>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> 
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

So if all works fine when you specify user@domain in roundcube, but you
want to be able to specify only username in roundcube and have roundcube
login as user@domain, then one solution would be to configure roundcube
to add a default domain to bare usernames. As far as I remember (it's a
while since I used roundcube) it is one of the configuration options.

John

Re: maildirlock fails

[Martynas Bendorius via dovecot]

It seems something like this has been reported already, however, I was unable 
to find any changelog entries or patches: 
http://dovecot.2317879.n4.nabble.com/maildirlock-time-unit-td65122.html

--
Best regards,
Martynas Bendorius


> On 12 Apr 2019, at 01:08, Martynas Bendorius  wrote:
> 
> Hello,
> 
> Maildirlock seems to panic on locking:
> [root@centos7 home]# /usr/libexec/dovecot/maildirlock 
> "/home/user/imap/domain.com/email/Maildir" 10
> Panic: BUG: No IOs or timeouts set. Not waiting for infinity.
> Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xd90ee) 
> [0x7f5bf02f10ee] -> /usr/lib/dovecot/libdovecot.so.0(+0xd9131) 
> [0x7f5bf02f1131] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) 
> [0x7f5bf0256efd] -> /usr/lib/dovecot/libdovecot.so.0(+0xf078c) 
> [0x7f5bf030878c] -> 
> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x36) 
> [0x7f5bf030b1f6] -> 
> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x56) [0x7f5bf03099c6] 
> -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f5bf0309be8] -> 
> /usr/libexec/dovecot/maildirlock(main+0x24a) [0x558ae0e032ca] -> 
> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5befe6d3d5] -> 
> /usr/libexec/dovecot/maildirlock(+0x13a2) [0x558ae0e033a2]
> 
> Is this a known issue?
> 
> Thank you!
> 
> --
> Best regards,
> Martynas Bendorius
> 
> 

Re: Fwd: SOLR/Index?

[Larry Rosenman via dovecot]

yes, either with or without the domain.


On Mon, Apr 15, 2019 at 9:33 AM John Fawcett via dovecot <
dovecot@dovecot.org> wrote:

> On 15/04/2019 16:05, Larry Rosenman via dovecot wrote:
>
> I have normal system users, some and some virtual domains (2 different
> flavors).
>
> I have this job that runs from cron on my own userid to archive mail
> I want Dovecot to use my system password for mail authentication.
> I, therefore, use PAM to authenticate system users
> Everything works great, modulo getting the auto-index to be visible using
> RoundCube (which if I log into roundcube using @ it does).
> by default if I'm logged in as my normal OS user, default doveadm commands
> (as issued from the shell or from my archive script) use the user I'm
> logged in as *WITHOUT A DOMAIN*.
>
> I want to default PAM auth'd users to append @lerctr.org (DOMAIN) to the
> OS user.
>
> when they log in to roundcube as the OS user, right?
>
> John
>


-- 
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: rsync sdbox, no new mails?

[Josef 'Jeff' Sipek via dovecot]

On Mon, Apr 15, 2019 at 14:45:31 +0200, Jakob Curdes via dovecot wrote:
> Hello, I had to restore a mailserver from a backup to replace a system 
> with problems that is still live. I did a 1:1 restore as far as possible 
> from a backup, then rsynced the /var/vmail/ directory, where we have all 
> the mailboxes, in sdbox format.
> So both directories should be the same. After syncing, I started 
> dovecot. I can see the mails until to the point of the backup, but not 
> the newer ones ,although they seem to be present in the "cur" directory.

"cur" is used by maildir.  sdbox does not have a "cur" dir.

Jeff.

> I thought with sdbox it is all in the index files that have been 
> transferred with the mails? Or am I missing a place that has to be synced?
> 
> Thank you for hints, Jakob
> 
> 
> 
> 

-- 
Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it.
- Brian W. Kernighan 

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 16:05, Larry Rosenman via dovecot wrote:
> I have normal system users, some and some virtual domains (2 different
> flavors).
>
> I have this job that runs from cron on my own userid to archive mail
> I want Dovecot to use my system password for mail authentication.
> I, therefore, use PAM to authenticate system users
> Everything works great, modulo getting the auto-index to be visible
> using RoundCube (which if I log into roundcube using @
> it does). 
> by default if I'm logged in as my normal OS user, default doveadm
> commands (as issued from the shell or from my archive script) use the
> user I'm logged in as *WITHOUT A DOMAIN*.
>
> I want to default PAM auth'd users to append @lerctr.org
>  (DOMAIN) to the OS user.
>
when they log in to roundcube as the OS user, right?

John

Re: rsync sdbox, no new mails?

[Jakob Curdes via dovecot]

Forget it,

 a) it was maildir (set via a SQL config that overrides the setting in 
10-mail.conf)

b) the client did not display the new messages. All fine.

JC

Re: FTS delays

[Josef 'Jeff' Sipek via dovecot]

On Sun, Apr 14, 2019 at 21:09:54 +0800, Joan Moreau wrote:
...
> THe "loop" part seems the most urgent : It breaks everything (search
> timeout 100% of the time) 

Any luck with git-bisect?

Jeff.

> 
> On 2019-04-06 09:56, Joan Moreau via dovecot wrote:
> 
> > For the point 1, this is not "suboptimal", it is plain wrong (results are 
> > damn wrong ! and this is not related to the backend, but the FTS logic in 
> > Dovecot core)
> > 
> > For the point 2 , this has been discussed already numerous times but 
> > without action. The dovecot core shall be the one re-submitting the emails 
> > to scan, not the backend to try to figure out where and which are the 
> > emails to be re-scaned 
> > 
> > For the point 3, I will do a bit of research in the existing code and will 
> > get back to you 
> > 
> > For the point 4, this is random. FTS backend (xapian, lucene, solr, 
> > whatever..) returns X, then dovecot core choose to select only Y emails. 
> > THis is a clear bug. 
> > 
> > On 2019-04-05 20:08, Josef 'Jeff' Sipek via dovecot wrote: 
> > On Fri, Apr 05, 2019 at 19:33:57 +0800, Joan Moreau via dovecot wrote: Hi 
> > 
> > If you plan to fix the FTS part of Dovecot, I will be very gratefull. 
> > I'm trying to figure out what is causing the 3rd issue you listed, so we can
> > decide how severe it is and therefore how quickly it needs to be fixed.  At
> > the moment we are unable to reproduce it, and therefore we cannot fix it.
> > 
> > Not sure this is related to any specific commit but rahter the overall
> > design 
> > Ok.
> > 
> > The list of bugs so far 
> > 
> > 1 - Double call to fts plugins with inconsistent parameter (first call
> > diferent from second call for the same request) 
> > Understood.  It is my understanding that this is simply suboptimal rather
> > than causing crashes/etc.
> > 
> > 2 - "Rescan" features for now consists of deleting indexes. SHall be
> > resending emails to rescan to the fts plugin instead 
> > I'm not sure I follow.  The rescan operation is invoked on the fts backend
> > and it is up to the implementation to somehow ensure that after it is done
> > the fts index is up to date.  The easiest way to implement it is to simply
> > delete the fts index and re-index all the mails.  That is what currently
> > happens in the solr backend.
> > 
> > The lucene fts backend does a more complicated matching of the fts index
> > with the emails.  Finally, the deprecated squat backend seem to ignore the
> > rescan requests (its rescan vfunc is NULL).
> > 
> > 3 - the loop when body search (just do a "doveadm search -u user@domain
> > mailbox inbox text whatevertexte") 
> > 
> > Refer to my email to Timo on 2019-04-03 18:30 on the same thread for bug
> > details 
> > 
> > (especially the loop) 
> > This seems to be the most important of the 4 issues you listed, so I'd like
> > to focus on this one for now.
> > 
> > As I mentioned, we cannot reproduce this ourselves.  So, we need your help
> > to narrow things down.  Therefore, can you give us the commit hashes of
> > revisions that you know are good and which are bad?  You can use git-bisect
> > to narrow the range down.
> > 
> > 4 - Most notably, I notice that header search usually does not care
> > about fts plugin (even with fts_enforced) and rely on some internal
> > search , which si total non-sense 
> > You're right, that doesn't seem to make sense.  Can you provide a test case?
> > 
> > Jeff.
> > 
> > Let me know how can I help on thos 4 points 
> > 
> > On 2019-04-05 18:37, Josef 'Jeff' Sipek wrote:
> > 
> > On Fri, Apr 05, 2019 at 17:45:36 +0800, Joan Moreau wrote: 
> > 
> > I am on master (very latest) 
> > 
> > No clue exactly when this problem appears, but 
> > 
> > 1 - the "request twice the fts plugin instead of once" issue has always
> > been there (since my first RC release of fts-xapian) 
> > Ok, good to know.
> > 
> > 2 - the body/text loop has appeared recently (maybe during the month of
> > March) 
> > Our testing doesn't seem to be able to reproduce this.  Can you try to
> > git-bisect this to find which commit broke it?
> > 
> > Thanks,
> > 
> > Jeff.
> > 
> > On 2019-04-05 16:36, Josef 'Jeff' Sipek via dovecot wrote:
> > 
> > On Wed, Apr 03, 2019 at 19:02:52 +0800, Joan Moreau via dovecot wrote: 
> > 
> > issue seems in the Git version : 
> > Which git revision?
> > 
> > Before you updated to the broken revision, which revision/version were you
> > running?
> > 
> > Can you try it with 5f6e39c50ec79ba8847b2fdb571a9152c71cd1b6 (the commit
> > just before the fts_enforced=body introduction)?  That's the only recent fts
> > change.
> > 
> > Thanks,
> > 
> > Jeff.
> > 
> > On 2019-04-03 18:58, @lbutlr via dovecot wrote:
> > 
> > On 3 Apr 2019, at 04:30, Joan Moreau via dovecot  
> > wrote: 
> > 
> > doveadm search -u j...@grosjo.net mailbox inbox text milan 
> > Did that search over my list mail and got 83 results, not able to duplicate 
> > your issue.
> > 
> > What version of dovecot and have you tried to 

LDAP iterate (dovecoc 2.2.27)

[Debreczeni Attila via dovecot]

Dear Dovecot Team,

We would like to use the LDAP iterate for dovecot replication, but I got
this error:

dovecot: auth-worker(10149): Error: ldap(*):
ldap_search(base=ou=mail,o=asd,c=TLD filter=(objectClass=inetMailUser)
failed: Size limit exceeded
dovecot: replicator: Error: User listing returned failure
dovecot: replicator: Error: listing users failed, can't replicate
existing data

When I check the "doveadm replicator status":
...
Total number of known users   1200

But we have more than 10.000 users. When I run the ldapsearch manually,
I get the full list of the mail addresses.

Who can I fix it?

Thank you,
Attila

# dovecot --version
2.2.27 (c0f36b0)

# dovecot -n | filtered
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.8
auth_worker_max_count = 150
default_vsz_limit = 512 M
doveadm_password =  # hidden, use -P to show it
doveadm_port = ***
mail_gid = vmail
mail_plugins = " quota notify replication"
namespace inbox {
 ...
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  mail_replica = tcp:REMOTE_IP
}
replication_dsync_parameters = -d -N -1 -l 30 -U
service aggregator {
  fifo_listener replication-notify-fifo {
    user = vmail
  }
  unix_listener replication-notify {
    user = vmail
  }
}
service auth {
  inet_listener {
    address = LOCAL_IP
    port = LOCAL_PORT
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0600
    user = vmail
  }
}
service dict {
  unix_listener dict {
    group = vmail
    mode = 0600
    user = vmail
  }
}
service lmtp {
  inet_listener lmtp {
    address = LOCAL_IP
    port = LOCAL_PORT
  }
}

service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
    mode = 0600
  }
}

userdb {
  args = /etc/dovecot/dovecot-domain-ldap.conf.ext
  driver = ldap
  result_success = continue-ok
}
userdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
  result_failure = return-fail
}
protocol imap {
...
}
protocol pop3 {
...
}


# grep -v '^ *\(#.*\)\?$' dovecot-ldap.conf.ext
uris = ldaps://ldap.URL
dn = uid=auth,o=domain,c=TLD
dnpass = 
sasl_bind = no
tls_ca_cert_dir = /etc/ssl/certs
tls_require_cert = demand
ldap_version = 3
base = ou=mail,o=asd,c=TLD
deref = never
scope = subtree
user_attrs = =home=/dev/null/%Ld/%L{ldap:uid},
mailQuota=quota_rule=*:bytes=%$
user_filter = (objectClass=inetMailUser)
pass_attrs = userPassword=password
pass_filter = (objectClass=inetMailUser)
iterate_attrs = mail=user
iterate_filter = (objectClass=inetMailUser)
default_pass_scheme = PLAIN

Re: Fwd: SOLR/Index?

[Larry Rosenman via dovecot]

I have normal system users, some and some virtual domains (2 different
flavors).

I have this job that runs from cron on my own userid to archive mail
I want Dovecot to use my system password for mail authentication.
I, therefore, use PAM to authenticate system users
Everything works great, modulo getting the auto-index to be visible using
RoundCube (which if I log into roundcube using @ it does).
by default if I'm logged in as my normal OS user, default doveadm commands
(as issued from the shell or from my archive script) use the user I'm
logged in as *WITHOUT A DOMAIN*.

I want to default PAM auth'd users to append @lerctr.org (DOMAIN) to the OS
user.



On Mon, Apr 15, 2019 at 7:34 AM John Fawcett via dovecot <
dovecot@dovecot.org> wrote:

> On 15/04/2019 13:43, Larry Rosenman via dovecot wrote:
> > Again, this doesn't help with doveadm running as the local user, and
> > also doesn't help with
> > the PAM authentication.
> >
> > passdb {
> >   driver = pam
> >   #[session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=20]
> >   #[cache_key=] []
> >   args = failure_show_msg=yes session=yes max_requests=20
> >   skip = authenticated
> > }
> >
> > How can I default the domain for PAM authentication?
> > (I've set auth_default_realm and it doesn't help in this case).
>
> Larry
>
> I guess I don't understand enough about your setup or what is not now
> working.
>
> My understanding was that everything is now working when logging in as
> user@domain, but that you would like to login as user and have dovecot
> treat that as though you had logged in as user@domain, but at this point
> I admit I may have misinterpreted your emails.
>
> What's also not clear for me is the purpose in your setup of the three
> passdb methods (sql, static, and pam) and two userdb methods (sql and
> passwd-file). That's why I've pointed you to the docs and I'm hestitant
> to give specific advice that may leave you worse off. Others on the list
> may have more insights.
>
> John
>
>

-- 
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

rsync sdbox, no new mails?

[Jakob Curdes via dovecot]

Hello, I had to restore a mailserver from a backup to replace a system 
with problems that is still live. I did a 1:1 restore as far as possible 
from a backup, then rsynced the /var/vmail/ directory, where we have all 
the mailboxes, in sdbox format.
So both directories should be the same. After syncing, I started 
dovecot. I can see the mails until to the point of the backup, but not 
the newer ones ,although they seem to be present in the "cur" directory.
I thought with sdbox it is all in the index files that have been 
transferred with the mails? Or am I missing a place that has to be synced?


Thank you for hints, Jakob

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 13:43, Larry Rosenman via dovecot wrote:
> Again, this doesn't help with doveadm running as the local user, and
> also doesn't help with
> the PAM authentication.
>
> passdb {
>   driver = pam
>   #[session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=20]
>   #[cache_key=] []
>   args = failure_show_msg=yes session=yes max_requests=20
>   skip = authenticated
> }
>
> How can I default the domain for PAM authentication?
> (I've set auth_default_realm and it doesn't help in this case).

Larry

I guess I don't understand enough about your setup or what is not now
working.

My understanding was that everything is now working when logging in as
user@domain, but that you would like to login as user and have dovecot
treat that as though you had logged in as user@domain, but at this point
I admit I may have misinterpreted your emails.

What's also not clear for me is the purpose in your setup of the three
passdb methods (sql, static, and pam) and two userdb methods (sql and
passwd-file). That's why I've pointed you to the docs and I'm hestitant
to give specific advice that may leave you worse off. Others on the list
may have more insights.

John

Re: Fwd: SOLR/Index?

[Larry Rosenman via dovecot]

Again, this doesn't help with doveadm running as the local user, and also
doesn't help with
the PAM authentication.

passdb {
  driver = pam
  #[session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=20]
  #[cache_key=] []
  args = failure_show_msg=yes session=yes max_requests=20
  skip = authenticated
}

How can I default the domain for PAM authentication?
(I've set auth_default_realm and it doesn't help in this case).

# System users (NSS, /etc/passwd, or similiar). In many systems nowadays
this
# uses Name Service Switch, which is configured in /etc/nsswitch.conf.
userdb {
  # 
  driver = passwd-file
  # [blocking=no]
  args = username_format=%Ln /etc/passwd
  #override_fields = user=%Ln /etc
  #name = %Ln

  # Override fields from passwd
  #override_fields = home=/home/virtual/%u
  #skip = found
}



On Mon, Apr 15, 2019 at 6:31 AM John Fawcett via dovecot <
dovecot@dovecot.org> wrote:

> On 15/04/2019 12:59, Larry Rosenman via dovecot wrote:
>
> forgot to reply all.
>
> -- Forwarded message -
> From: Larry Rosenman 
> Date: Mon, Apr 15, 2019 at 5:58 AM
> Subject: Re: SOLR/Index?
> To: John Fawcett 
>
>
> the local users (myself, my wife, a friend) can authenticate EITHER as
>  or @lerctr.org.
>
> switching to all virtual users is NOT going to happen.
>
> If I login to roundcube with @lerctr.org it finds the
> autoindexed mail.
>
> So, if I make everyone always authenticate as @lerctr.org we should
> be fine.
>
> and change my script to do doveadm -u @ instead of depending
> on the local user running the cron job.
>
> question: Is there some way to have dovecot change what it sees to be
> @lerctr.org when they login as ?
>
>
> Dovecot is very configurable, but it can also take some time, effort and
> testing to get the configuration you want. Personally I don't mix user
> types since it takes out an element of complexity.
>
> For your case you might find it useful to look into auth_default_realm
>
> that can specify a domain name when one is not supplied.
>
> https://wiki2.dovecot.org/DomainLost
>
> Other things that might be useful: there is a method for returning a
> "user" field from the userdb query or passdb query which will change the
> username. Or there is another setting that can overriding values of fields
> returned by the userdb.
>
> https://wiki.dovecot.org/UserDatabase
>
> Hope it helps!
>
> John
>


-- 
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 12:59, Larry Rosenman via dovecot wrote:
> forgot to reply all.
>
> -- Forwarded message -
> From: *Larry Rosenman* mailto:larry...@gmail.com>>
> Date: Mon, Apr 15, 2019 at 5:58 AM
> Subject: Re: SOLR/Index?
> To: John Fawcett mailto:j...@voipsupport.it>>
>
>
> the local users (myself, my wife, a friend) can authenticate EITHER as
>  or @lerctr.org .
>
> switching to all virtual users is NOT going to happen. 
>
> If I login to roundcube with @lerctr.org  it
> finds the autoindexed mail.
>
> So, if I make everyone always authenticate as @lerctr.org
>  we should be fine.
>
> and change my script to do doveadm -u @ instead of
> depending on the local user running the cron job. 
>
> question: Is there some way to have dovecot change what it sees to be
> @lerctr.org  when they login as ?
>
>
Dovecot is very configurable, but it can also take some time, effort and
testing to get the configuration you want. Personally I don't mix user
types since it takes out an element of complexity.

For your case you might find it useful to look into auth_default_realm

that can specify a domain name when one is not supplied.

https://wiki2.dovecot.org/DomainLost

Other things that might be useful: there is a method for returning a
"user" field from the userdb query or passdb query which will change the
username. Or there is another setting that can overriding values of
fields returned by the userdb.

https://wiki.dovecot.org/UserDatabase

Hope it helps!

John

Re: Fwd: SOLR/Index?

[Larry Rosenman via dovecot]

These are REAL local users, authenticated via PAM


On Mon, Apr 15, 2019 at 6:26 AM Johan Huldtgren via dovecot <
dovecot@dovecot.org> wrote:

>
>
> On 2019/04/15 06:59, Larry Rosenman via dovecot wrote:
> > forgot to reply all.
> >
> > -- Forwarded message -
> > From: *Larry Rosenman* mailto:larry...@gmail.com>>
> > Date: Mon, Apr 15, 2019 at 5:58 AM
> > Subject: Re: SOLR/Index?
> > To: John Fawcett mailto:j...@voipsupport.it>>
> >
> >
> > the local users (myself, my wife, a friend) can authenticate EITHER as
>  or @lerctr.org .
> >
> > switching to all virtual users is NOT going to happen.
> >
> > If I login to roundcube with @lerctr.org  it
> finds the autoindexed mail.
> >
> > So, if I make everyone always authenticate as @lerctr.org <
> http://lerctr.org> we should be fine.
> >
> > and change my script to do doveadm -u @ instead of
> depending on the local user running the cron job.
> >
> > question: Is there some way to have dovecot change what it sees to be
> @lerctr.org  when they login as ?
>
> FWIW, when I switched all users to be virtual to let the handful of people
> who had been logging in as 
> not have to switch to @example.com I created a second passdb
> entry which points to different sql.conf file
> which automatically appends this specific domain, thereby letting them
> type in  but getting logged in as
> @example.com so my dovecot logs always show @
> example.com. Not sure if this helps you get around
> your issue.
>
> .jh
>
> > On Mon, Apr 15, 2019 at 5:54 AM John Fawcett via dovecot <
> dovecot@dovecot.org > wrote:
> >
> > On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:
> >> ⌂63% [l...@thebighonker.lerctr.org:~  l...@thebighonker.lerctr.org:~>] $ grep fts1970 mail/INBOX
> >> ⌂67% [l...@thebighonker.lerctr.org:~  l...@thebighonker.lerctr.org:~>] 1 $ mail -s "test fts1970" l...@lerctr.org
> 
> >> test fts1970
> >>
> >> test fts1970
> >> .
> >> EOT
> >> [l...@thebighonker.lerctr.org:~ 
> >> ]
> $ mailq
> >> [l...@thebighonker.lerctr.org:~ 
> >> ]
> $ grep fts1970 mail/INBOX
> >> Subject: test fts1970
> >> test fts1970
> >> test fts1970
> >>
> >>
> >> Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <=
> l...@lerctr.org  U=ler P=local S=388
> >> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect
> from local
> >> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(
> l...@lerctr.org/49364 ): save: box=INBOX,
> uid=175402, msgid= e1hfxvd-000csq...@thebighonker.lerctr.org>>, size=640, vsize=660,
> from=Larry Rosenman mailto:l...@lerctr.org>>,
> subject=test fts1970, flags=()
> >> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(
> l...@lerctr.org/49364 ): sieve: msgid=<
> e1hfxvd-000csq...@thebighonker.lerctr.org  e1hfxvd-000csq...@thebighonker.lerctr.org>>: stored mail into mailbox
> 'INBOX' (subject=test fts1970 from=l...@lerctr.org 
> size=660)
> >> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect
> from local: Client has quit the connection (state=READY)
> >> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler <
> l...@lerctr.org > R=localuser T=dovecot_lmtp S=404
> C="250 2.0.0 mailto:l...@lerctr.org>>
> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
> >> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6
> Completed QT=0s
> >> Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker(
> l...@lerctr.org/49366 ): Indexed 1 messages
> in INBOX (UIDs 175402..175402)
> >>
> >>
> >> ⌂81% [l...@thebighonker.lerctr.org:~  l...@thebighonker.lerctr.org:~>] $ doveadm search mailbox INBOX  body
> 'fts1970'
> >> ⌂83% [l...@thebighonker.lerctr.org:~  l...@thebighonker.lerctr.org:~>] $
> >>
> >>
> >> ⌂65% [l...@thebighonker.lerctr.org:~  l...@thebighonker.lerctr.org:~>] 75 $ doveadm search -u l...@lerctr.org
>   mailbox INBOX body 'fts1970'
> >> a53a143be44bda5bd483bbe98eac 175402
> >> [l...@thebighonker.lerctr.org:~ 
> >> ]
> $ doveadm index -q INBOX
> >> [l...@thebighonker.lerctr.org:~ 
> >> ]
> $ doveadm search mailbox INBOX  body 'fts1970'
> >> [l...@thebighonker.lerctr.org:~ 
> >> ]
> $ doveadm fts rescan
> >> [l...@thebighonker.lerctr.org:~ 
> >> ]
> $ doveadm index -q INBOX
> >> [l...@thebighonker.lerctr.org:~ 
> >> ]
> $ doveadm search mailbox INBOX  body 'fts1970'
> >> a53a143be44bda5bd483bbe98eac 175402
> >> 

Re: Fwd: SOLR/Index?

[Johan Huldtgren via dovecot]

On 2019/04/15 06:59, Larry Rosenman via dovecot wrote:
> forgot to reply all.
> 
> -- Forwarded message -
> From: *Larry Rosenman* mailto:larry...@gmail.com>>
> Date: Mon, Apr 15, 2019 at 5:58 AM
> Subject: Re: SOLR/Index?
> To: John Fawcett mailto:j...@voipsupport.it>>
> 
> 
> the local users (myself, my wife, a friend) can authenticate EITHER as 
>  or @lerctr.org .
> 
> switching to all virtual users is NOT going to happen. 
> 
> If I login to roundcube with @lerctr.org  it finds 
> the autoindexed mail.
> 
> So, if I make everyone always authenticate as @lerctr.org 
>  we should be fine.
> 
> and change my script to do doveadm -u @ instead of depending on 
> the local user running the cron job. 
> 
> question: Is there some way to have dovecot change what it sees to be 
> @lerctr.org  when they login as ?

FWIW, when I switched all users to be virtual to let the handful of people who 
had been logging in as 
not have to switch to @example.com I created a second passdb entry 
which points to different sql.conf file
which automatically appends this specific domain, thereby letting them type in 
 but getting logged in as
@example.com so my dovecot logs always show @example.com. 
Not sure if this helps you get around
your issue.

.jh

> On Mon, Apr 15, 2019 at 5:54 AM John Fawcett via dovecot  > wrote:
> 
> On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:
>> ⌂63% [l...@thebighonker.lerctr.org:~ 
>> ] $ grep fts1970 mail/INBOX
>> ⌂67% [l...@thebighonker.lerctr.org:~ 
>> ] 1 $ mail -s "test fts1970" 
>> l...@lerctr.org 
>> test fts1970
>>
>> test fts1970
>> .
>> EOT
>> [l...@thebighonker.lerctr.org:~ ] 
>> $ mailq
>> [l...@thebighonker.lerctr.org:~ ] 
>> $ grep fts1970 mail/INBOX
>> Subject: test fts1970
>> test fts1970
>> test fts1970
>>
>>
>> Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <= 
>> l...@lerctr.org  U=ler P=local S=388
>> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from 
>> local
>> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364 
>> ): save: box=INBOX, uid=175402, 
>> msgid=> >, size=640, vsize=660, 
>> from=Larry Rosenman mailto:l...@lerctr.org>>, subject=test 
>> fts1970, flags=()
>> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364 
>> ): sieve: 
>> msgid=> >: stored mail into 
>> mailbox 'INBOX' (subject=test fts1970 from=l...@lerctr.org 
>>  size=660)
>> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect from 
>> local: Client has quit the connection (state=READY)
>> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler 
>> mailto:l...@lerctr.org>> R=localuser T=dovecot_lmtp S=404 
>> C="250 2.0.0 mailto:l...@lerctr.org>> 
>> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
>> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed 
>> QT=0s
>> Apr 15 04:29:03 thebighonker dovecot[2507]: 
>> indexer-worker(l...@lerctr.org/49366 ): 
>> Indexed 1 messages in INBOX (UIDs 175402..175402)
>>
>>
>> ⌂81% [l...@thebighonker.lerctr.org:~ 
>> ] $ doveadm search mailbox INBOX  
>> body 'fts1970'
>> ⌂83% [l...@thebighonker.lerctr.org:~ 
>> ] $
>>
>>
>> ⌂65% [l...@thebighonker.lerctr.org:~ 
>> ] 75 $ doveadm search -u 
>> l...@lerctr.org   mailbox INBOX body 'fts1970'
>> a53a143be44bda5bd483bbe98eac 175402
>> [l...@thebighonker.lerctr.org:~ ] 
>> $ doveadm index -q INBOX
>> [l...@thebighonker.lerctr.org:~ ] 
>> $ doveadm search mailbox INBOX  body 'fts1970'
>> [l...@thebighonker.lerctr.org:~ ] 
>> $ doveadm fts rescan
>> [l...@thebighonker.lerctr.org:~ ] 
>> $ doveadm index -q INBOX
>> [l...@thebighonker.lerctr.org:~ ] 
>> $ doveadm search mailbox INBOX  body 'fts1970'
>> a53a143be44bda5bd483bbe98eac 175402
>> [l...@thebighonker.lerctr.org:~ ] 
>> $ doveadm search -u l...@lerctr.org   mailbox INBOX 
>> body 'fts1970'
>> a53a143be44bda5bd483bbe98eac 175402
>> [l...@thebighonker.lerctr.org:~ 

Fwd: SOLR/Index?

[Larry Rosenman via dovecot]

forgot to reply all.

-- Forwarded message -
From: Larry Rosenman 
Date: Mon, Apr 15, 2019 at 5:58 AM
Subject: Re: SOLR/Index?
To: John Fawcett 


the local users (myself, my wife, a friend) can authenticate EITHER as
 or @lerctr.org.

switching to all virtual users is NOT going to happen.

If I login to roundcube with @lerctr.org it finds the
autoindexed mail.

So, if I make everyone always authenticate as @lerctr.org we should
be fine.

and change my script to do doveadm -u @ instead of depending
on the local user running the cron job.

question: Is there some way to have dovecot change what it sees to be
@lerctr.org when they login as ?



On Mon, Apr 15, 2019 at 5:54 AM John Fawcett via dovecot <
dovecot@dovecot.org> wrote:

> On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:
>
> ⌂63% [l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> ⌂67% [l...@thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970"
> l...@lerctr.org
> test fts1970
>
> test fts1970
> .
> EOT
> [l...@thebighonker.lerctr.org:~] $ mailq
> [l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> Subject: test fts1970
> test fts1970
> test fts1970
>
>
> Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <=
> l...@lerctr.org U=ler P=local S=388
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from local
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364):
> save: box=INBOX, uid=175402, msgid=<
> e1hfxvd-000csq...@thebighonker.lerctr.org>, size=640, vsize=660,
> from=Larry Rosenman , subject=test fts1970, flags=()
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364):
> sieve: msgid=: stored mail
> into mailbox 'INBOX' (subject=test fts1970 from=l...@lerctr.org size=660)
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect from
> local: Client has quit the connection (state=READY)
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler <
> l...@lerctr.org> R=localuser T=dovecot_lmtp S=404 C="250 2.0.0 <
> l...@lerctr.org> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s
> Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker(
> l...@lerctr.org/49366): Indexed 1 messages in INBOX (UIDs 175402..175402)
>
>
> ⌂81% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> ⌂83% [l...@thebighonker.lerctr.org:~] $
>
>
> ⌂65% [l...@thebighonker.lerctr.org:~] 75 $ doveadm search -u l...@lerctr.org
> mailbox INBOX body 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
> [l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $ doveadm search -u l...@lerctr.org
> mailbox INBOX body 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $
>
> So, yes, your hypothesis is correct.
>
> Question: How can I make it consistent?
>
> I have a script that runs on the first of the month that does archiving,
> and I have similar issues in that namespace:
> ⌂67% [l...@thebighonker.lerctr.org:~] $ cat bin/archive-mail
> #!/bin/sh
> PATH=$PATH:/usr/local/bin
> #Expects to be run after midnight on the first of the month
> #  to archive all the previous months mail
> #Date Run:
> TODAY=`date "+%Y-%m-%d"`
> #last month in /MM
> YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"`
> #1st of last month as 01-Mon-
> FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"`
> echo 'TODAY=' ${TODAY}
> echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH}
> echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH}
> # get a list of all the mailboxes with at least one real message
> doveadm -f tab mailbox status vsize \* 2>/dev/null |
> sed -e 1d | sort -k 1,1 |
> awk  'BEGIN {FS="\t"} {if ($2 > 0)  print $1}' |
> while read i
> do
>echo `date` start ${i}
>doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}"
>doveadm -f tab mailbox status messages "${i}"
>doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \
> "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH}
>doveadm -f tab mailbox status messages "${i}"
>echo `date` done  ${i}
> done
> ⌂64% [l...@thebighonker.lerctr.org:~] $
>
>
> The Exim config can be provided as well if necessary.
>
> ler & l...@lerctr.org *ARE THE SAME MAILBOX*
>
> At the moment it looks as though you have two sets of emails indexed in
> solr. One is indexed under username (the one you are running mannually and
> apparently the one used by roundcube too, but that's to be verified) and
> another set being indexed by autoindex = yes option using the full email
> address. Once you've got it working as you require, then you may 

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:
> ⌂63% [l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> ⌂67% [l...@thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970"
> l...@lerctr.org 
> test fts1970
>
> test fts1970
> .
> EOT
> [l...@thebighonker.lerctr.org:~] $ mailq
> [l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> Subject: test fts1970
> test fts1970
> test fts1970
>
>
> Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <=
> l...@lerctr.org  U=ler P=local S=388
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from
> local
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364
> ): save: box=INBOX, uid=175402,
> msgid= >, size=640,
> vsize=660, from=Larry Rosenman  >, subject=test fts1970, flags=()
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364
> ): sieve:
> msgid= >: stored mail into
> mailbox 'INBOX' (subject=test fts1970 from=l...@lerctr.org
>  size=660)
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect
> from local: Client has quit the connection (state=READY)
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler
> mailto:l...@lerctr.org>> R=localuser T=dovecot_lmtp
> S=404 C="250 2.0.0 mailto:l...@lerctr.org>>
> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s
> Apr 15 04:29:03 thebighonker dovecot[2507]:
> indexer-worker(l...@lerctr.org/49366 ):
> Indexed 1 messages in INBOX (UIDs 175402..175402)
>
>
> ⌂81% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX 
> body 'fts1970'
> ⌂83% [l...@thebighonker.lerctr.org:~] $
>
>
> ⌂65% [l...@thebighonker.lerctr.org:~] 75 $ doveadm search -u
> l...@lerctr.org   mailbox INBOX body 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
> [l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $ doveadm search -u l...@lerctr.org
>   mailbox INBOX body 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $
>
> So, yes, your hypothesis is correct.
>
> Question: How can I make it consistent?  
>
> I have a script that runs on the first of the month that does
> archiving, and I have similar issues in that namespace:
> ⌂67% [l...@thebighonker.lerctr.org:~] $ cat bin/archive-mail
> #!/bin/sh
> PATH=$PATH:/usr/local/bin
> #Expects to be run after midnight on the first of the month
> #  to archive all the previous months mail
> #Date Run:
> TODAY=`date "+%Y-%m-%d"`
> #last month in /MM
> YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"`
> #1st of last month as 01-Mon-
> FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"`
> echo 'TODAY=' ${TODAY}
> echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH}
> echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH}
> # get a list of all the mailboxes with at least one real message
> doveadm -f tab mailbox status vsize \* 2>/dev/null |
>         sed -e 1d | sort -k 1,1 |
>         awk  'BEGIN {FS="\t"} {if ($2 > 0)  print $1}' |
> while read i
> do
>    echo `date` start ${i}
>    doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}"
>    doveadm -f tab mailbox status messages "${i}"
>    doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \
>             "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH}
>    doveadm -f tab mailbox status messages "${i}"
>    echo `date` done  ${i}
> done
> ⌂64% [l...@thebighonker.lerctr.org:~] $
>
>
> The Exim config can be provided as well if necessary.
>
> ler & l...@lerctr.org  *ARE THE SAME MAILBOX*
>
At the moment it looks as though you have two sets of emails indexed in
solr. One is indexed under username (the one you are running mannually
and apparently the one used by roundcube too, but that's to be verified)
and another set being indexed by autoindex = yes option using the full
email address. Once you've got it working as you require, then you may
want to clean out solr and reindex with just one of them just to reduce
volumes.

Your setup seems to have a mix of users from mysql and from /etc/passwd.
Not sure if your mysql users are all mapped to real users or they have
their own mailboxes with domain included. Your solution will depend on
what you really need and if the setup is working correctly you may not
want to tweak it too much or other things may 

Re: SOLR/Index?

[Larry Rosenman via dovecot]

⌂63% [l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
⌂67% [l...@thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970"
l...@lerctr.org
test fts1970

test fts1970
.
EOT
[l...@thebighonker.lerctr.org:~] $ mailq
[l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
Subject: test fts1970
test fts1970
test fts1970


Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <= l...@lerctr.org
U=ler P=local S=388
Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from local
Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364):
save: box=INBOX, uid=175402, msgid=<
e1hfxvd-000csq...@thebighonker.lerctr.org>, size=640, vsize=660, from=Larry
Rosenman , subject=test fts1970, flags=()
Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364):
sieve: msgid=: stored mail into
mailbox 'INBOX' (subject=test fts1970 from=l...@lerctr.org size=660)
Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect from
local: Client has quit the connection (state=READY)
Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler <
l...@lerctr.org> R=localuser T=dovecot_lmtp S=404 C="250 2.0.0 <
l...@lerctr.org> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s
Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker(
l...@lerctr.org/49366): Indexed 1 messages in INBOX (UIDs 175402..175402)


⌂81% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
'fts1970'
⌂83% [l...@thebighonker.lerctr.org:~] $


⌂65% [l...@thebighonker.lerctr.org:~] 75 $ doveadm search -u l...@lerctr.org
mailbox INBOX body 'fts1970'
a53a143be44bda5bd483bbe98eac 175402
[l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
[l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
'fts1970'
[l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
[l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
[l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
'fts1970'
a53a143be44bda5bd483bbe98eac 175402
[l...@thebighonker.lerctr.org:~] $ doveadm search -u l...@lerctr.org  mailbox
INBOX body 'fts1970'
a53a143be44bda5bd483bbe98eac 175402
[l...@thebighonker.lerctr.org:~] $

So, yes, your hypothesis is correct.

Question: How can I make it consistent?

I have a script that runs on the first of the month that does archiving,
and I have similar issues in that namespace:
⌂67% [l...@thebighonker.lerctr.org:~] $ cat bin/archive-mail
#!/bin/sh
PATH=$PATH:/usr/local/bin
#Expects to be run after midnight on the first of the month
#  to archive all the previous months mail
#Date Run:
TODAY=`date "+%Y-%m-%d"`
#last month in /MM
YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"`
#1st of last month as 01-Mon-
FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"`
echo 'TODAY=' ${TODAY}
echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH}
echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH}
# get a list of all the mailboxes with at least one real message
doveadm -f tab mailbox status vsize \* 2>/dev/null |
sed -e 1d | sort -k 1,1 |
awk  'BEGIN {FS="\t"} {if ($2 > 0)  print $1}' |
while read i
do
   echo `date` start ${i}
   doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}"
   doveadm -f tab mailbox status messages "${i}"
   doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \
"${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH}
   doveadm -f tab mailbox status messages "${i}"
   echo `date` done  ${i}
done
⌂64% [l...@thebighonker.lerctr.org:~] $


The Exim config can be provided as well if necessary.

ler & l...@lerctr.org *ARE THE SAME MAILBOX*


On Mon, Apr 15, 2019 at 4:05 AM John Fawcett via dovecot <
dovecot@dovecot.org> wrote:

> On 15/04/2019 10:59, Larry Rosenman via dovecot wrote:
>
> I'll run a full test when I'm back in front of areal computer vs. My
> phone.(in a few hours)
>
> Get Outlook for Android 
>
> --
> *From:* dovecot 
>  on behalf of John Fawcett via dovecot
>  
> *Sent:* Monday, April 15, 2019 3:57:08 AM
> *To:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>
> On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:
>
> It always shows the autoindex. And yes built from sources.  I'm the
> FreeBSD port maintainer for mail/docecot.  This has been happening for
> several releases.
>
> Get Outlook for Android 
>
> --
> *From:* dovecot 
>  on behalf of John Fawcett via dovecot
>  
> *Sent:* Monday, April 15, 2019 2:06:55 AM
> *To:* dovecot@dovecot.org
> *Subject:* Re: SOLR/Index?
>
> On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
>
> Note the hits after the fts rescan/index.
>
> Get Outlook for Android 
>
> --
> *From:* Aki Tuomi 
> 
> *Sent:* Monday, April 15, 2019 12:55:07 AM
> *To:* Larry Rosenman; John Fawcett
> *Cc:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>
>
>
> On 15.4.2019 3.33, Larry 

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 10:59, Larry Rosenman via dovecot wrote:
> I'll run a full test when I'm back in front of areal computer vs. My
> phone.(in a few hours)
>
> Get Outlook for Android 
>
> 
> *From:* dovecot  on behalf of John
> Fawcett via dovecot 
> *Sent:* Monday, April 15, 2019 3:57:08 AM
> *To:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>  
> On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:
>> It always shows the autoindex. And yes built from sources.  I'm the
>> FreeBSD port maintainer for mail/docecot.  This has been happening
>> for several releases.
>>
>> Get Outlook for Android 
>>
>> 
>> *From:* dovecot  on behalf of John
>> Fawcett via dovecot 
>> *Sent:* Monday, April 15, 2019 2:06:55 AM
>> *To:* dovecot@dovecot.org
>> *Subject:* Re: SOLR/Index?
>>  
>> On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
>>> Note the hits after the fts rescan/index.
>>>
>>> Get Outlook for Android 
>>>
>>> 
>>> *From:* Aki Tuomi 
>>> *Sent:* Monday, April 15, 2019 12:55:07 AM
>>> *To:* Larry Rosenman; John Fawcett
>>> *Cc:* Dovecot Mailing List
>>> *Subject:* Re: SOLR/Index?
>>>  
>>>
>>>
>>> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
 ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
 lists/freebsd/ports-commiters  body 'sysutils'
 [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
 [l...@thebighonker.lerctr.org:~] $ doveadm index -q
 lists/freebsd/ports-commiters
 ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
 Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login:
 Disconnected (auth failed, 1 attempts in 2 secs): user=,
 method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS:
 Connection closed, session=
 Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
 user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
 lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
 session=
 Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
 out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
 Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
 address 23.100.68.192
 Apr 14 19:30:55 thebighonker exim[14846]:
 H=(DaVinci-MWare.prophet21lab.com
 ) [23.100.68.192]:52130
 I=[192.147.25.65]:25 sender verify defer for >>> >: host lookup did not complete
 Apr 14 19:30:55 thebighonker exim[14846]:
 H=(DaVinci-MWare.prophet21lab.com
 ) [23.100.68.192]:52130
 I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
 temporarily rejected RCPT mailto:jpo...@why.net>>:
 Could not complete sender verify
 Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
 user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
 lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
 session=
 Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
 out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
 Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
 address 80.253.235.35
 Apr 14 19:31:19 thebighonker dovecot[2507]:
 indexer-worker(ler/14919): Indexed 1578 messages in
 lists/freebsd/ports-commiters (UIDs 21067..22644)
 ^C
 [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
 lists/freebsd/ports-commiters  body 'sysutils/'
>>>
>>>
>>> Just minor nit, but you are searching for 'sysutils' first, then
>>> 'sysutils/'. FTS does not do substring searches by default.
>>>
>>> Aki
>>>

 -- 
 Larry Rosenman                     http://www.lerctr.org/~ler
 
 Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
 
 US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>>
>> Larry
>>
>> just to be sure: are you running a standard unmodified 2.3.5.1
>> version which you built from source code?
>>
>> I can see that first you search for sysutils, then do a rescan and
>> reindex (which is shown in the log) and then you are able to find
>> sysutils/.
>>
>> It is better when doing these tests to search for the same string
>> before and after, just to eliminate too many different factors in the
>> test.
>>
>> Nevertheless I did not see your logging for what happens when you
>> receive a test message containing sysutils/. Dovecot should be
>> outputing info about autoindexing given your setup. Does it do that
>> or does it give some other message? Can you show those logs?
>>
>> John
>>
> Larry
>
> Did you notice any difference between the logging for auto 

Re: SOLR/Index?

[Larry Rosenman via dovecot]

I'll run a full test when I'm back in front of areal computer vs. My phone.(in 
a few hours)

Get Outlook for Android


From: dovecot  on behalf of John Fawcett via 
dovecot 
Sent: Monday, April 15, 2019 3:57:08 AM
To: Dovecot Mailing List
Subject: Re: SOLR/Index?

On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:
It always shows the autoindex. And yes built from sources.  I'm the FreeBSD 
port maintainer for mail/docecot.  This has been happening for several releases.

Get Outlook for Android


From: dovecot  
on behalf of John Fawcett via dovecot 

Sent: Monday, April 15, 2019 2:06:55 AM
To: dovecot@dovecot.org
Subject: Re: SOLR/Index?

On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
Note the hits after the fts rescan/index.

Get Outlook for Android


From: Aki Tuomi 
Sent: Monday, April 15, 2019 12:55:07 AM
To: Larry Rosenman; John Fawcett
Cc: Dovecot Mailing List
Subject: Re: SOLR/Index?



On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
⌂72% [l...@thebighonker.lerctr.org:~] $ 
doveadm search mailbox lists/freebsd/ports-commiters  body 'sysutils'
[l...@thebighonker.lerctr.org:~] $ 
doveadm fts rescan
[l...@thebighonker.lerctr.org:~] $ 
doveadm index -q lists/freebsd/ports-commiters
⌂64% [l...@thebighonker.lerctr.org:~] $ 
tail -f /var/log/maillog
Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth 
failed, 1 attempts in 2 secs): user=, method=PLAIN, rip=180.180.217.124, 
lip=192.147.25.65, TLS: Connection closed, session=
Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=, 
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, 
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, 
session=
Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out 
in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 
23.100.68.192
Apr 14 19:30:55 thebighonker exim[14846]: 
H=(DaVinci-MWare.prophet21lab.com) 
[23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for 
mailto:i...@duke.org>>: host lookup did not complete
Apr 14 19:30:55 thebighonker exim[14846]: 
H=(DaVinci-MWare.prophet21lab.com) 
[23.100.68.192]:52130 I=[192.147.25.65]:25 
F=mailto:i...@duke.org>> temporarily rejected RCPT 
mailto:jpo...@why.net>>: Could not complete sender verify
Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=, 
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, 
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, 
session=
Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 
out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 
80.253.235.35
Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 
1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644)
^C
[l...@thebighonker.lerctr.org:~] 130 $ 
doveadm search mailbox lists/freebsd/ports-commiters  body 'sysutils/'


Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. 
FTS does not do substring searches by default.

Aki

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: 
larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version which 
you built from source code?

I can see that first you search for sysutils, then do a rescan and reindex 
(which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before and 
after, just to eliminate too many different factors in the test.

Nevertheless I did not see your logging for what happens when you receive a 
test message containing sysutils/. Dovecot should be outputing info about 
autoindexing given your setup. Does it do that or does it give some other 
message? Can you show those logs?

John

Larry

Did you notice any difference between the logging for auto indexing and the 
logging for indexing that you triggered manually? Would you mind posting the 
auto indexing logging for a message to that same user (ler)?

best regards

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:
> It always shows the autoindex. And yes built from sources.  I'm the
> FreeBSD port maintainer for mail/docecot.  This has been happening for
> several releases.
>
> Get Outlook for Android 
>
> 
> *From:* dovecot  on behalf of John
> Fawcett via dovecot 
> *Sent:* Monday, April 15, 2019 2:06:55 AM
> *To:* dovecot@dovecot.org
> *Subject:* Re: SOLR/Index?
>  
> On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
>> Note the hits after the fts rescan/index.
>>
>> Get Outlook for Android 
>>
>> 
>> *From:* Aki Tuomi 
>> *Sent:* Monday, April 15, 2019 12:55:07 AM
>> *To:* Larry Rosenman; John Fawcett
>> *Cc:* Dovecot Mailing List
>> *Subject:* Re: SOLR/Index?
>>  
>>
>>
>> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
>>> ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
>>> lists/freebsd/ports-commiters  body 'sysutils'
>>> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
>>> [l...@thebighonker.lerctr.org:~] $ doveadm index -q
>>> lists/freebsd/ports-commiters
>>> ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
>>> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected
>>> (auth failed, 1 attempts in 2 secs): user=, method=PLAIN,
>>> rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed,
>>> session=
>>> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
>>> session=
>>> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
>>> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
>>> address 23.100.68.192
>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>> H=(DaVinci-MWare.prophet21lab.com
>>> ) [23.100.68.192]:52130
>>> I=[192.147.25.65]:25 sender verify defer for >> >: host lookup did not complete
>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>> H=(DaVinci-MWare.prophet21lab.com
>>> ) [23.100.68.192]:52130
>>> I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
>>> temporarily rejected RCPT mailto:jpo...@why.net>>:
>>> Could not complete sender verify
>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
>>> session=
>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
>>> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
>>> address 80.253.235.35
>>> Apr 14 19:31:19 thebighonker dovecot[2507]:
>>> indexer-worker(ler/14919): Indexed 1578 messages in
>>> lists/freebsd/ports-commiters (UIDs 21067..22644)
>>> ^C
>>> [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
>>> lists/freebsd/ports-commiters  body 'sysutils/'
>>
>>
>> Just minor nit, but you are searching for 'sysutils' first, then
>> 'sysutils/'. FTS does not do substring searches by default.
>>
>> Aki
>>
>>>
>>> -- 
>>> Larry Rosenman                     http://www.lerctr.org/~ler
>>> 
>>> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
>>> 
>>> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>
> Larry
>
> just to be sure: are you running a standard unmodified 2.3.5.1 version
> which you built from source code?
>
> I can see that first you search for sysutils, then do a rescan and
> reindex (which is shown in the log) and then you are able to find
> sysutils/.
>
> It is better when doing these tests to search for the same string
> before and after, just to eliminate too many different factors in the
> test.
>
> Nevertheless I did not see your logging for what happens when you
> receive a test message containing sysutils/. Dovecot should be
> outputing info about autoindexing given your setup. Does it do that or
> does it give some other message? Can you show those logs?
>
> John
>
Larry

Did you notice any difference between the logging for auto indexing and
the logging for indexing that you triggered manually? Would you mind
posting the auto indexing logging for a message to that same user (ler)?

best regards

John

Re: SOLR/Index?

[Larry Rosenman via dovecot]

It always shows the autoindex. And yes built from sources.  I'm the FreeBSD 
port maintainer for mail/docecot.  This has been happening for several releases.

Get Outlook for Android


From: dovecot  on behalf of John Fawcett via 
dovecot 
Sent: Monday, April 15, 2019 2:06:55 AM
To: dovecot@dovecot.org
Subject: Re: SOLR/Index?

On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
Note the hits after the fts rescan/index.

Get Outlook for Android


From: Aki Tuomi 
Sent: Monday, April 15, 2019 12:55:07 AM
To: Larry Rosenman; John Fawcett
Cc: Dovecot Mailing List
Subject: Re: SOLR/Index?



On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
⌂72% [l...@thebighonker.lerctr.org:~] $ 
doveadm search mailbox lists/freebsd/ports-commiters  body 'sysutils'
[l...@thebighonker.lerctr.org:~] $ 
doveadm fts rescan
[l...@thebighonker.lerctr.org:~] $ 
doveadm index -q lists/freebsd/ports-commiters
⌂64% [l...@thebighonker.lerctr.org:~] $ 
tail -f /var/log/maillog
Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth 
failed, 1 attempts in 2 secs): user=, method=PLAIN, rip=180.180.217.124, 
lip=192.147.25.65, TLS: Connection closed, session=
Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=, 
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, 
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, 
session=
Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out 
in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 
23.100.68.192
Apr 14 19:30:55 thebighonker exim[14846]: 
H=(DaVinci-MWare.prophet21lab.com) 
[23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for 
mailto:i...@duke.org>>: host lookup did not complete
Apr 14 19:30:55 thebighonker exim[14846]: 
H=(DaVinci-MWare.prophet21lab.com) 
[23.100.68.192]:52130 I=[192.147.25.65]:25 
F=mailto:i...@duke.org>> temporarily rejected RCPT 
mailto:jpo...@why.net>>: Could not complete sender verify
Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=, 
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, 
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, 
session=
Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 
out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 
80.253.235.35
Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 
1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644)
^C
[l...@thebighonker.lerctr.org:~] 130 $ 
doveadm search mailbox lists/freebsd/ports-commiters  body 'sysutils/'


Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. 
FTS does not do substring searches by default.

Aki

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: 
larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version which 
you built from source code?

I can see that first you search for sysutils, then do a rescan and reindex 
(which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before and 
after, just to eliminate too many different factors in the test.

Nevertheless I did not see your logging for what happens when you receive a 
test message containing sysutils/. Dovecot should be outputing info about 
autoindexing given your setup. Does it do that or does it give some other 
message? Can you show those logs?

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
> Note the hits after the fts rescan/index.
>
> Get Outlook for Android 
>
> 
> *From:* Aki Tuomi 
> *Sent:* Monday, April 15, 2019 12:55:07 AM
> *To:* Larry Rosenman; John Fawcett
> *Cc:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>  
>
>
> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
>> ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
>> lists/freebsd/ports-commiters  body 'sysutils'
>> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
>> [l...@thebighonker.lerctr.org:~] $ doveadm index -q
>> lists/freebsd/ports-commiters
>> ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
>> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected
>> (auth failed, 1 attempts in 2 secs): user=, method=PLAIN,
>> rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed,
>> session=
>> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
>> session=
>> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
>> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
>> address 23.100.68.192
>> Apr 14 19:30:55 thebighonker exim[14846]:
>> H=(DaVinci-MWare.prophet21lab.com
>> ) [23.100.68.192]:52130
>> I=[192.147.25.65]:25 sender verify defer for > >: host lookup did not complete
>> Apr 14 19:30:55 thebighonker exim[14846]:
>> H=(DaVinci-MWare.prophet21lab.com
>> ) [23.100.68.192]:52130
>> I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
>> temporarily rejected RCPT mailto:jpo...@why.net>>:
>> Could not complete sender verify
>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
>> session=
>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
>> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
>> address 80.253.235.35
>> Apr 14 19:31:19 thebighonker dovecot[2507]:
>> indexer-worker(ler/14919): Indexed 1578 messages in
>> lists/freebsd/ports-commiters (UIDs 21067..22644)
>> ^C
>> [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
>> lists/freebsd/ports-commiters  body 'sysutils/'
>
>
> Just minor nit, but you are searching for 'sysutils' first, then
> 'sysutils/'. FTS does not do substring searches by default.
>
> Aki
>
>>
>> -- 
>> Larry Rosenman                     http://www.lerctr.org/~ler
>> 
>> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
>> 
>> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version
which you built from source code?

I can see that first you search for sysutils, then do a rescan and
reindex (which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before
and after, just to eliminate too many different factors in the test.

Nevertheless I did not see your logging for what happens when you
receive a test message containing sysutils/. Dovecot should be outputing
info about autoindexing given your setup. Does it do that or does it
give some other message? Can you show those logs?

John

Re: SOLR/Index?

[Larry Rosenman via dovecot]

Note the hits after the fts rescan/index.

Get Outlook for Android


From: Aki Tuomi 
Sent: Monday, April 15, 2019 12:55:07 AM
To: Larry Rosenman; John Fawcett
Cc: Dovecot Mailing List
Subject: Re: SOLR/Index?



On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
⌂72% [l...@thebighonker.lerctr.org:~] $ 
doveadm search mailbox lists/freebsd/ports-commiters  body 'sysutils'
[l...@thebighonker.lerctr.org:~] $ 
doveadm fts rescan
[l...@thebighonker.lerctr.org:~] $ 
doveadm index -q lists/freebsd/ports-commiters
⌂64% [l...@thebighonker.lerctr.org:~] $ 
tail -f /var/log/maillog
Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth 
failed, 1 attempts in 2 secs): user=, method=PLAIN, rip=180.180.217.124, 
lip=192.147.25.65, TLS: Connection closed, session=
Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: user=, 
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, 
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, 
session=
Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out 
in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP address 
23.100.68.192
Apr 14 19:30:55 thebighonker exim[14846]: 
H=(DaVinci-MWare.prophet21lab.com) 
[23.100.68.192]:52130 I=[192.147.25.65]:25 sender verify defer for 
mailto:i...@duke.org>>: host lookup did not complete
Apr 14 19:30:55 thebighonker exim[14846]: 
H=(DaVinci-MWare.prophet21lab.com) 
[23.100.68.192]:52130 I=[192.147.25.65]:25 
F=mailto:i...@duke.org>> temporarily rejected RCPT 
mailto:jpo...@why.net>>: Could not complete sender verify
Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: user=, 
method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, 
lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, 
session=
Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out in=169 
out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP address 
80.253.235.35
Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919): Indexed 
1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644)
^C
[l...@thebighonker.lerctr.org:~] 130 $ 
doveadm search mailbox lists/freebsd/ports-commiters  body 'sysutils/'


Just minor nit, but you are searching for 'sysutils' first, then 'sysutils/'. 
FTS does not do substring searches by default.

Aki

--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: 
larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106