Re: another problem with conversations db
Yes, updating reconstruct is on our roadmap. The biggest problem is that there are multiple file types across mailboxes and conversations and the changes can't be atomic, so if reconstruct discovers something partially done, it can't tell if the conversations were correct. The longer term grand plan is to reduce the number of db types and make conversations updates able to be atomic. Cheers, Bron. On Fri, Jan 25, 2019, at 22:30, Michael Menge wrote: > Hi, > > > Quoting Bron Gondwana : > > > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: > >> Hi Bron > >> > >> > >> Quoting Bron Gondwana : > >> > >> > Sorry I haven't been following along with this earlier. Can you post > >> > your imapd.conf and cyrus.conf as well as let her know if you run > >> > anything which directly messes with files on disk. > >> > >> Thanks for looking into it. I have attached the backend and replica > >> configs. There should be nothing messing with the files on disk > >> directly. Some times we need to restore mails from file based > >> backup (bacula) followed by a reconstruct but this was not the > >> case this time. > > > > Do you always rebuild the conversations db after doing the > > reconstruct? That will be necessary now. We switched to doing all > > our restores using IMAP append a while back so we're never fiddling > > the file system under Cyrus. > > > > We didn't have to restore any mails from backup since we enabled > conversation db > a few weeks ago. It is good to know that the rebuild is necessary, but > shouldn't > reconstruct also update the conversation db if it re-appends the message? > At least a hint should be put in the reconstruct man page, like the > one for "quota -f" > > > >> > > >> > Also, what operating system is this on and what Cyrus version? > >> > > >> > >> We are running a cyrus 3.0.8 compiled with the following Options > >> (./configure --enable-murder --enable-http --enable-calalarmd > >> --enable-replication --enable-backup --enable-idled > >> --enable-autocreate CFLAGS="-fPIC -g") > >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. > > > > They should be fine. I'll have a read of the config when I'm at a > > real computer. > > > >> > >> > Bron. > >> > > >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: > >> >> Hi, > >> >> > >> >> I have discovered an other problem with the conversations db: > >> >> > >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and > >> >> "IOERROR: conversations_audit on store:" > >> >> A look at the source code shows that these errors are logged after > >> >> "_sanity_check_counts" is called. > >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a > >> >> serious problem. Do I? > >> >> > >> >> This problem occurred for accounts where the rebuild of the > >> >> conversations db was successful. > >> >> > >> >> I don't want to rebuild the conversations db every few days. > >> >> > >> >> Any help is appreciated. > >> >> > >> >> Kind regards > >> >> > >> >> > >> >> Michael Menge > > > > > M.Menge Tel.: (49) 7071/29-70316 > Universität Tübingen Fax.: (49) 7071/29-5912 > Zentrum für Datenverarbeitung mail: > michael.me...@zdv.uni-tuebingen.de > Wächterstraße 76 > 72074 Tübingen > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: another problem with conversations db
Hi, Quoting Bron Gondwana : On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: Hi Bron Quoting Bron Gondwana : > Sorry I haven't been following along with this earlier. Can you post > your imapd.conf and cyrus.conf as well as let her know if you run > anything which directly messes with files on disk. Thanks for looking into it. I have attached the backend and replica configs. There should be nothing messing with the files on disk directly. Some times we need to restore mails from file based backup (bacula) followed by a reconstruct but this was not the case this time. Do you always rebuild the conversations db after doing the reconstruct? That will be necessary now. We switched to doing all our restores using IMAP append a while back so we're never fiddling the file system under Cyrus. We didn't have to restore any mails from backup since we enabled conversation db a few weeks ago. It is good to know that the rebuild is necessary, but shouldn't reconstruct also update the conversation db if it re-appends the message? At least a hint should be put in the reconstruct man page, like the one for "quota -f" > > Also, what operating system is this on and what Cyrus version? > We are running a cyrus 3.0.8 compiled with the following Options (./configure --enable-murder --enable-http --enable-calalarmd --enable-replication --enable-backup --enable-idled --enable-autocreate CFLAGS="-fPIC -g") in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. They should be fine. I'll have a read of the config when I'm at a real computer. > Bron. > > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: >> Hi, >> >> I have discovered an other problem with the conversations db: >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and >> "IOERROR: conversations_audit on store:" >> A look at the source code shows that these errors are logged after >> "_sanity_check_counts" is called. >> The log level LOG_ERR and the prefix IOERROR indicate that I have a >> serious problem. Do I? >> >> This problem occurred for accounts where the rebuild of the >> conversations db was successful. >> >> I don't want to rebuild the conversations db every few days. >> >> Any help is appreciated. >> >> Kind regards >> >> >> Michael Menge M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: another problem with conversations db
On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote: > Hi Bron > > > Quoting Bron Gondwana : > > > Sorry I haven't been following along with this earlier. Can you post > > your imapd.conf and cyrus.conf as well as let her know if you run > > anything which directly messes with files on disk. > > Thanks for looking into it. I have attached the backend and replica > configs. There should be nothing messing with the files on disk > directly. Some times we need to restore mails from file based > backup (bacula) followed by a reconstruct but this was not the > case this time. Do you always rebuild the conversations db after doing the reconstruct? That will be necessary now. We switched to doing all our restores using IMAP append a while back so we're never fiddling the file system under Cyrus. > > > > Also, what operating system is this on and what Cyrus version? > > > > We are running a cyrus 3.0.8 compiled with the following Options > (./configure --enable-murder --enable-http --enable-calalarmd > --enable-replication --enable-backup --enable-idled > --enable-autocreate CFLAGS="-fPIC -g") > in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. They should be fine. I'll have a read of the config when I'm at a real computer. > > > Bron. > > > > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: > >> Hi, > >> > >> I have discovered an other problem with the conversations db: > >> > >> Thousends of lines with "IOERROR: conversations_audit on load:" and > >> "IOERROR: conversations_audit on store:" > >> A look at the source code shows that these errors are logged after > >> "_sanity_check_counts" is called. > >> The log level LOG_ERR and the prefix IOERROR indicate that I have a > >> serious problem. Do I? > >> > >> This problem occurred for accounts where the rebuild of the > >> conversations db was successful. > >> > >> I don't want to rebuild the conversations db every few days. > >> > >> Any help is appreciated. > >> > >> Kind regards > >> > >> > >> Michael Menge > >> > >> > >> > >> > >> > >> > >> M.Menge Tel.: (49) 7071/29-70316 > >> Universität Tübingen Fax.: (49) 7071/29-5912 > >> Zentrum für Datenverarbeitung mail: > >> michael.me...@zdv.uni-tuebingen.de > >> Wächterstraße 76 > >> 72074 Tübingen > >> > >> > >> Cyrus Home Page: http://www.cyrusimap.org/ > >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > >> To Unsubscribe: > >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > > -- > > Bron Gondwana > > br...@fastmail.fm > > > > > M.Menge Tel.: (49) 7071/29-70316 > Universität Tübingen Fax.: (49) 7071/29-5912 > Zentrum für Datenverarbeitung mail: > michael.me...@zdv.uni-tuebingen.de > Wächterstraße 76 > 72074 Tübingen > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > *Attachments:* > * imapd_be.conf > * cyrus_be.conf > * cyrus_re.conf > * imapd_re.conf -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: another problem with conversations db
Hi Bron Quoting Bron Gondwana : Sorry I haven't been following along with this earlier. Can you post your imapd.conf and cyrus.conf as well as let her know if you run anything which directly messes with files on disk. Thanks for looking into it. I have attached the backend and replica configs. There should be nothing messing with the files on disk directly. Some times we need to restore mails from file based backup (bacula) followed by a reconstruct but this was not the case this time. Also, what operating system is this on and what Cyrus version? We are running a cyrus 3.0.8 compiled with the following Options (./configure --enable-murder --enable-http --enable-calalarmd --enable-replication --enable-backup --enable-idled --enable-autocreate CFLAGS="-fPIC -g") in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs. Bron. On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote: Hi, I have discovered an other problem with the conversations db: Thousends of lines with "IOERROR: conversations_audit on load:" and "IOERROR: conversations_audit on store:" A look at the source code shows that these errors are logged after "_sanity_check_counts" is called. The log level LOG_ERR and the prefix IOERROR indicate that I have a serious problem. Do I? This problem occurred for accounts where the rebuild of the conversations db was successful. I don't want to rebuild the conversations db every few days. Any help is appreciated. Kind regards Michael Menge M.Menge Tel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana br...@fastmail.fm M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen # Configuration for Backend/Failover Instance # Template MD5SUM: b5b04095d552bb1cbc2b178f7edfd1e2 conf/imapd_master.template servername: ma01.mail.localhost configdirectory: /srv/cyrus-be partition-default: /srv/cyrus-be partition-ssd: /srv/cyrus-be/ssd-part metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part metapartition_files: header index cache expunge squat annotations lock dav archivecache archivepartition-ssd: /srv/cyrus-hdd-be/archive/ssd-part archive_enabled: 1 archive_days: 31 archive_maxsize: 10240 proc_path: /srv/tmpfs/proc-be mboxname_lockpath: /srv/tmpfs/lock-be defaultpartition: ssd admins: mupdate_server: mupdate.mail.localhost mupdate_port: 3905 # mupdate_authname verwenden nicht mupdate_username # sonst wird login als root/cyrus versucht #mupdate_username: mupdate_authname: mupdate_password: proxy_authname: proxy_password: proxyservers: allowusermoves: 1 allowallsubscribe: 1 sync_host: sl01.mail.localhost sync_authname: sync_password: sync_port: 2005 guid_mode: sha1 sync_log: 1 sync_shutdown_file: /srv/cyrus-be/sync/shutdown sievedir: /srv/cyrus-be/sieve sieve_extensions: fileinto reject vacation imapflags notify include envelope body relational regex subaddress copy sieve_maxscriptsize: 150 sasl_pwcheck_method: saslauthd sasl_mech_list: plain login allowanonymouslogin: no reject8bit: no quotawarn: 90 timeout: 45 poptimeout: 10 dracinterval: 0 drachost: localhost lmtp_over_quota_perm_failure: 1 altnamespace: 1 #flushseenstate: 1 unixhierarchysep: 1 hashimapspool: 1 fulldirhash: 1 duplicatesuppression: 0 expunge_mode: delayed delete_mode: delayed deletedprefix: DELETED anysievefolder: 1 #annotation_db: skiplist #duplicate_db: skiplist #mboxlist_db: skiplist #ptscache_db: skiplist quota_db: quotalegacy #seenstate_db: skiplist subscriptions_db: flat xlist-sent: Mail.sent xlist-trash: Mail.trash xlist-drafts: Mail.drafts xlist-junk: Mail.v-spam xlist-spam: Mail.v-spam specialusealways: 1 syslog_prefix: be # https://bugzilla.cyrusimap.org/show_bug.cgi?id=3850 # Vermutlich gefixed #suppress_capabilities: LIST-EXTENDED tls_server_cert: /etc/pki/cyrus-imapd/cyrus-imapd.pem tls_server_key: /etc/pki/cyrus-imapd/cyrus-imapd.pem tls_client_ca_file: /etc/pki/tls/certs/ca-bundle.crt tls_ciphers: EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA:EECDH:EDH+AESGCM:EDH:+3DES:ECDH+AESGCM:ECDH+AES:ECDH:AES:HIGH:MEDIUM:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP:!3DES:!IDEA tls_prefer_server_ciphers: 1 # Änderungen 27.06.2018 reverseacls: 1 search_engine: squat delete_unsubscribe: 1 statuscache: 1 #tlscache_db deprecated
