Cyrus backups and deleted (users, mailboxes)

[Deborah Pickett]

Semi-related questions about how Cyrus backup servers deal with deleted 
stuff.


I have the following settings on the main mail server:

deletedprefix: DELETED
delete_mode: delayed
expunge_mode: delayed

My first question is about deleted folders.  Scenario: I delete a folder 
on the server, and it is renamed into the DELETED namespace as 
expected.  I cannot back these folders up to the backup server.  They 
are not enumerated during "XBACKUP * channel", and if I name the DELETED 
namespace explicitly I get the following output:


x xbackup DELETED/* rsync
* NO MAILBOX polyfoam.com.au!DELETED.rawmaterials.Trash.Downer 
EDI.5DAD45A0 (Mailbox is locked)
* NO MAILBOX polyfoam.com.au!DELETED.support.Has Spaces.5DCA22F4 
(Mailbox is locked)

x NO Mailbox is locked

(Which is not nearly the whole list either: it should have listed 
hundreds of deleted folders.)


Is this expected behaviour?  I suppose that my backup has a copy of this 
folder in its original location before it was deleted, so no one will 
miss these folders, (though I can imagine scenarios where a message 
arrives and is quickly put into a folder which is then deleted, and this 
message will never touch the backup server).


My second question is about deleted users.  Scenario: A user "alice" 
departs the company, and I delete their root mailbox on the main 
server.  It is no longer enumerated during "XBACKUP * channel".


What will happen to the backup file a/alice_XX on the backup 
server?  Will it stay there forever?  Do I need to manually delete it?  
Do I need to manually remove alice from /var/lib/cyrus/backups.db?  I 
suppose I can use "ctl_backups list -t" to detect such files.  Any other 
things I should be aware of?


--
*Deborah Pickett*
System Administrator
*Polyfoam Australia Pty Ltd*


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Backup compaction optimization in a block-level replication environment

[Deborah Pickett]

On 2019-11-11 11:10, ellie timoney wrote:

This setting might be helpful:

Thanks, I saw that setting but didn't really think through how it would
help me.  I'll experiment with it and report back.

That would be great, thanks!


Progress report: I started with very large chunks (minimum 64 MB, 
maximum 1024 MB) and a threshold of 8 chunks but I found that compaction 
was running every time, even on a backup file that hardly changed.  Not 
certain why this would be; my current theory is that in chunks that size 
there is almost always some benefit to compacting, so the threshold is 
passed easily.  There were 41 chunks in my %SHARED backup.


I'm now trying very small chunks (no minimum size, maximum 128 kB) with 
varying thresholds.  This is probably _too_ small (smaller than even 
some messages).  I'll bisect the difference and see if there is a sweet 
spot.


I've settled on rsync as the transport protocol for sending the backups 
off site.  Its rolling-checksum algorithm means that even if a chunk 
moves within the file it can still be transmitted efficiently, saving me 
from needing all that fragmentation guff I started this thread with.


Related: I had to apply the patch described in 
(https://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg47320.html), 
"backupd IOERROR reading backup files larger than 2GB", because during 
initial population of my backup, chunks tended to by multiple GB in size 
(my %SHARED user backup is 23 GB, compressed).  Will this patch be 
merged to a main line?


--
*Deborah Pickett*
System Administrator
*Polyfoam Australia Pty Ltd*


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.12 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.0.12

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

This release contains a fix for CVE-2019-18928, a session hijacking 
vulnerability in the httpd daemon.  If you compile cyrus with HTTP support 
enabled, your cyrus.conf contains SERVICES entries that run the httpd daemon, 
and you provide a proxy frontend service that reuses connections to the 
backend, you will need this upgrade.

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.12/cyrus-imapd-3.0.12.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.12/cyrus-imapd-3.0.12.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-3.0.12.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-3.0.12.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 
3.0.12:

https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.12.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 2.5.14 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 2.5.14

This release contains a fix for CVE-2019-18928, a session hijacking 
vulnerability in the httpd daemon.  If you compile cyrus with HTTP support 
enabled, your cyrus.conf contains SERVICES entries that run the httpd daemon, 
and you provide a proxy frontend service that reuses connections to the 
backend, you will need this upgrade.

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.14/cyrus-imapd-2.5.14.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.14/cyrus-imapd-2.5.14.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-2.5.14.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-2.5.14.tar.gz.sig

Please consult the release notes before upgrading to 2.5.14:

https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.14.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Centos 8

[Nikos Gatsis - Qbit]

Title: Untitled Document

  
  
Thank you very much!


On 14/11/2019 1:49 μ.μ., Patrick
  Boutilier wrote:

On
  11/14/19 7:38 AM, Nikos Gatsis - Qbit wrote:
  
  Hello list.


Does somebody knows witch version on cyrus-imapd centos 8
install?


Thank you in advance.

  
  
  
  yum list|grep cyrus-imapd
  
  
  cyrus-imapd.i686
  3.0.7-15.el8_0.1     AppStream
  
  
  cyrus-imapd.x86_64  
  3.0.7-15.el8_0.1     AppStream
  
  
  cyrus-imapd-utils.x86_64
  3.0.7-15.el8_0.1     AppStream
  
  
  cyrus-imapd-vzic.x86_64 
  3.0.7-15.el8_0.1     AppStream
  
  
  
  
  

Nikos.




Cyrus Home Page: http://www.cyrusimap.org/

List Archives/Info:
http://lists.andrew.cmu.edu/pipermail/info-cyrus/

To Unsubscribe:

https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

  
  
  
  
  
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
  
  
  
  Γατσής
  Νίκος - Gatsis Nikos
Web developer
tel.: 210.8256721 - 210.8256712
email: ngat...@qbit.gr
http://www.qbit.gr 
  


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Centos 8

[Patrick Boutilier]

On 11/14/19 7:38 AM, Nikos Gatsis - Qbit wrote:

Hello list.

Does somebody knows witch version on cyrus-imapd centos 8 install?

Thank you in advance.



yum list|grep cyrus-imapd

cyrus-imapd.i686 3.0.7-15.el8_0.1 
AppStream


cyrus-imapd.x86_64   3.0.7-15.el8_0.1 
AppStream


cyrus-imapd-utils.x86_64 3.0.7-15.el8_0.1 
AppStream


cyrus-imapd-vzic.x86_64  3.0.7-15.el8_0.1 
AppStream






Nikos.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


<>
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Centos 8

[Nikos Gatsis - Qbit]

Hello list.

Does somebody knows witch version on cyrus-imapd centos 8 install?

Thank you in advance.

Nikos.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [Help] Cyrus 2.4.17 segfault

[Geoff Winkless]

On Thu, 14 Nov 2019 at 08:44, Michael Menge
 wrote:

> here is the why only addresses are show and not function names
> you need the debug package for your cyrus imapd

To be fair, you also need to run

gdb /path/to/imapd coredump

as per

> "/var/spool/abrt/ccpp-2019-11-13-07:04:30-19177/coredump" is a core file.
> Please specify an executable to debug.

But yes, a debug version of imapd will make the dump more useful.

Geoff

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [Help] Cyrus 2.4.17 segfault

[Michael Menge]

Quoting Marco :


Hello Michael,
  I've generated the dump, but it isn't easy for me to understand  
the content...


[root@cyrus ccpp-2019-11-13-07:04:30-19177]# gdb coredump
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Missing separate debuginfo for the main executable file


here is the why only addresses are show and not function names
you need the debug package for your cyrus imapd

Try: yum --disablerepo='*' --enablerepo='*-debug*' install  
/usr/lib/debug/.build-id/59/d016b26d3d62fcd00eb1400ff1be9c0142ee7c

[New Thread 19177]
Core was generated by `imapd:'.
Program terminated with signal 7, Bus error.
#0  0x7f3af70cd122 in ?? ()
"/var/spool/abrt/ccpp-2019-11-13-07:04:30-19177/coredump" is a core file.
Please specify an executable to debug.
(gdb) bt full
#0  0x7f3af70cd122 in ?? ()
No symbol table info available.
#1  0x7f3af757fcb0 in ?? ()
No symbol table info available.
#2  0x7f3af70cf008 in ?? ()
No symbol table info available.
#3  0x7fff in ?? ()
No symbol table info available.
#4  0x7f3af757fcb0 in ?? ()
No symbol table info available.
#5  0x7f3af757fcc0 in ?? ()
No symbol table info available.
#6  0x7f3af757fcc8 in ?? ()
No symbol table info available.
#7  0x7fff6dec5770 in ?? ()
No symbol table info available.
#8  0x7f3af757c8b0 in ?? ()
No symbol table info available.
#9  0x0001 in ?? ()
No symbol table info available.
#10 0x7f3a0fbc in ?? ()
No symbol table info available.
#11 0x7fff6dec5820 in ?? ()
No symbol table info available.
#12 0x in ?? ()
No symbol table info available.






M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:  
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [Help] Cyrus 2.4.17 segfault

[Marco]

Hello Michael,
  I've generated the dump, but it isn't easy for me to understand the 
content...


[root@cyrus ccpp-2019-11-13-07:04:30-19177]# gdb coredump
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Missing separate debuginfo for the main executable file
Try: yum --disablerepo='*' --enablerepo='*-debug*' install 
/usr/lib/debug/.build-id/59/d016b26d3d62fcd00eb1400ff1be9c0142ee7c

[New Thread 19177]
Core was generated by `imapd:'.
Program terminated with signal 7, Bus error.
#0  0x7f3af70cd122 in ?? ()
"/var/spool/abrt/ccpp-2019-11-13-07:04:30-19177/coredump" is a core file.
Please specify an executable to debug.
(gdb) bt full
#0  0x7f3af70cd122 in ?? ()
No symbol table info available.
#1  0x7f3af757fcb0 in ?? ()
No symbol table info available.
#2  0x7f3af70cf008 in ?? ()
No symbol table info available.
#3  0x7fff in ?? ()
No symbol table info available.
#4  0x7f3af757fcb0 in ?? ()
No symbol table info available.
#5  0x7f3af757fcc0 in ?? ()
No symbol table info available.
#6  0x7f3af757fcc8 in ?? ()
No symbol table info available.
#7  0x7fff6dec5770 in ?? ()
No symbol table info available.
#8  0x7f3af757c8b0 in ?? ()
No symbol table info available.
#9  0x0001 in ?? ()
No symbol table info available.
#10 0x7f3a0fbc in ?? ()
No symbol table info available.
#11 0x7fff6dec5820 in ?? ()
No symbol table info available.
#12 0x in ?? ()
No symbol table info available.

All reconstruct on user/a was useless. Now we have resolved deleting 
user/a at all.


Thank you
Marco

Il 13/11/2019 09:51, Michael Menge ha scritto:

Hi Marco,

can you generate a coredump and generate a backtrace (e.g. with gdb
https://stackoverflow.com/questions/5115613/core-dump-file-analysis )
You might need to install some debug packages.




Quoting Marco :


Il 12/11/2019 22:58, ellie timoney ha scritto:

On Tue, Nov 12, 2019, at 2:51 AM, Marco wrote:

An user user/a has full ACL to another mailbox user/b. When the user/a
SELECT a folder on user/b where he has access the imap process crashes.


If you set up a couple of test accounts with the same sharing 
arrangement, do those crash in the same way?  Or is it specific to 
the two original accounts?


Hi Ellie,

 I made other tests... It's specific to the user/a account.

When user/a SELECT a shared folder where it has access, the imap 
process crashes.


I updated to cyrus-imapd-2.4.20 and the issue still happens.

I also tried to rebuild mailboxes.db with ctl_mboxlist -d -u, I tried 
to rebuild mailboxes.db with an old backup... I tried to delete the 
statuscache and to delete the skipstamp db... the problem still happens.


Thank you
Marco

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus




 


M.Menge    Tel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail: 
michael.me...@zdv.uni-tuebingen.de

Wächterstraße 76
72074 Tübingen


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus