Re: [Dovecot] IMAP SPECIAL-USE extension
On 2011-12-02 16:12, Timo Sirainen wrote:
It's implemented now in dovecot-2.1 hg. It also deprecates autocreate plugin
(but it still works the old way). The idea is that you can now do e.g.:
mailbox Spam {
auto = create # autocreate, but don't autosubscribe
special_use = \Junk
This looks very nice. Thanks Timo.
--
Aleksander 'A.L.E.C' Machniak
LAN Management System Developer [http://lms.org.pl]
Roundcube Webmail Developer [http://roundcube.net]
---
PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
Re: [Dovecot] Questions about single intance storage
Hello Timo. If we can not safely restore from backup a user's mailbox with SiS enabled, we can not enable SiS. Any plan to include this backup recovery tool in dovecot 2.0.X or 2.1? Regards Maria 5ยบ We use bacula to save indexes mdboxes, and we recover mailboxes using doveadm import when a user makes a fatal mistake wiping all her Inbox. If we enable SiS I am not really sure how can we safely restore a user's INBOX if that user has SiS attachments. Hm. Yes, that is problematic.. Even if you knew what SIS files were used, there's no simple way to restore those with proper refcounts. I think what really should be done is writing a tool that can create/restore backups, possibly de-SISing the attachments..
Re: [Dovecot] Questions about single intance storage
On 4.12.2011, at 16.10, Maria Arrea wrote: If we can not safely restore from backup a user's mailbox with SiS enabled, we can not enable SiS. Any plan to include this backup recovery tool in dovecot 2.0.X or 2.1? I'd first have to design it. And before designing it I'd need to look into how the backup softwares usually work.. If anyone has any ideas about this, I'd like to hear.
Re: [Dovecot] Questions about single intance storage
If we can not safely restore from backup a user's mailbox with SiS enabled, we can not enable SiS. Any plan to include this backup recovery tool in dovecot 2.0.X or 2.1? I'd first have to design it. And before designing it I'd need to look into how the backup softwares usually work.. If anyone has any ideas about this, I'd like to hear. BackupPC uses rsync by default for *nix boxes. No idea what SiS is, but I'm guessing you're running into the same problem as backing up any other open file with changing internal data that may be inconsistent. This is exactly what it's difficult (and pointless) to backup an open MySQL database or a SQL Server database. The snapshot of what's in memory doesn't always match what's on disk. The only ways I know around this are to periodically create a backup copy that *is* consistent and restorable and a utility to restore the backup back to the live storage format, or create a method for the software to flush it's buffers to disk then disconnect from the data file while the backup process is running. The first option takes ~2x the storage space, while the second option makes the user's data inaccessible during the backup. My apologies if I'm misunderstanding the problem and have been rambling for no purpose. 8-) Terry
Re: [Dovecot] Questions about single intance storage
On 4.12.2011, at 19.41, Terry Carmen wrote: If we can not safely restore from backup a user's mailbox with SiS enabled, we can not enable SiS. Any plan to include this backup recovery tool in dovecot 2.0.X or 2.1? I'd first have to design it. And before designing it I'd need to look into how the backup softwares usually work.. If anyone has any ideas about this, I'd like to hear. BackupPC uses rsync by default for *nix boxes. No idea what SiS is, but I'm guessing you're running into the same problem as backing up any other open file with changing internal data that may be inconsistent. Inconsistency is an issue, but it's not the biggest problem. It would be possible to write a tool that scans through all mails and makes sure everything is consistent after a restore. dbox mostly does this automatically already, but SIS would need a separate program to ensure its consistency. SIS is anyway a single instance attachment storage. So that lets say you send one 100 MB pdf to 10 people, and it's stored only once in disk under /attachments/aa/bb/aabccddeeff-etc. Each people would have their own unique link under /attachments/, but all of them would be hard linked to a common file. So the problem is mainly about restoring a single user's mails. The mail files are simple to restore, but then you need to figure out which attachments to restore. There's no simple way to know which attachment files belong to which users, so you need to scan through the mail files and see what attachments are referred to. Also backing up the attachment links could be problematic if the backup system doesn't support hard links. Each attachment always has at least 2 links, so if the backup doesn't realize that it at minimum duplicates the space used by attachments. So I was thinking that there probably could be some tool that during a user's backup it would write the attachments among the user's other files, so it would be easy to find all of the files needed for a restore. This would of course mean that backups can take a lot more space, because there's no SIS. Perhaps there could be some other alternatives .. like maybe not storing the attachments directly to backups, but add symlinks to them so they can be used to figure out what to restore. Or maybe the backing up wouldn't need a special tool, but the restoring tool could just read through the dbox files to see what attachments are also needed and write a list of them somewhere so they can be taken from backups as well. I'm not really sure what is the best way.
Re: [Dovecot] Questions about single intance storage
On 5.12.2011, at 0.07, Lorens Kockum wrote: Timo Sirainen wrote: And before designing it I'd need to look into how the backup softwares usually work.. If anyone has any ideas about this, I'd like to hear. Simple or even moderately efficient backup programs like rsync copy all the files. I'm mainly wondering if it's common for backup programs to support using a separate program to generate the backups. For example if there was a dovecot-backup binary that just dumps all (or new-since-last-backup) of the users' mails into stdout, which the backup program can use. Or perhaps in that case there wouldn't really be much of anything for the backup to do except to write it to tape.. Also backing up the attachment links could be problematic if the backup system doesn't support hard links. Each attachment always has at least 2 links, so if the backup doesn't realize that it at minimum duplicates the space used by attachments. rsync recognizes hard links with option -H, but at a very noticeable performance cost when dealing with millions of files. If the aa/bb/aabccddeeff-etc is unique across the whole mailstore, it would be easy to replace the hard link with a symlink, as you said: SIS was designed to work with hard links. They couldn't be replaced with symlinks without a redesign (which would be less efficient in normal operation). maybe not storing the attachments directly to backups, but add symlinks to them so they can be used to figure out what to restore. Or maybe the backing up wouldn't need a special tool, but the restoring tool could just read through the dbox files to see what attachments are also needed and write a list of them somewhere so they can be taken from backups as well. In the second way, you would have a separate hierarchy for multiple-recipient attachments, or would the attachment be really stored in the box of a recipient chosen at random? I meant that SIS would work exactly like it works now, with hard links and everything, but on top of that it would also create symlinks to the used files simply to make it easier to find what files are used. The annoying thing about that is that in error situations the symlinks can get out of sync with the reality. Just some random thoughts: professionally, I use Zimbra. Messages are stored in Maildir-equivalents. The time it takes to backup is a quite severe constraint on the backup technique. For example, compressing the backup files takes too long, so the zip files are not compressed. Instead, the individual mails are stored compressed on disk. Each backup zips up the mails in a few big backup files. You mean you first create uncompressed zip files (why not just tar?) of all the mails to the filesystem and the backup software then backups those zip files? An improvement could be to sort mails into backup zip files so that once a zip file is made, it stays the same. After all, if a mail is not deleted a month after it is read, then it will probably stay in the same state forever, or at least until the user starts a keep-me-under-quota cleaning-up spree. During this time, backing up that big zip file can just be a check to see if it is already OK in the backup, which is much quicker. I have no idea if this could be applied to Dovecot, but who knows. Dovecot's mdbox files already contain multiple messages in each file, so it should be a lot more efficient to do backups on those. And each message in an mdbox file can be compressed if zlib plugin is enabled. So I think that sounds quite a lot like what you propose.
