rados mailbox? (was Re: Ceph for email storage)
Although Ceph fs would technically work for storing mail with maildir, when you step back from the situation, Maildir + a distributed file system is a pretty terrible way to approach mail storage. Maildir was designed to work around the limited consistency of NFS, and manages that, but performs pretty horribly on almost any file system. Mostly this is due to the message-per-file approach and the fact that file systems' internal management of inodes and directories mean lots and lots of seeks, even to read message headers. Ceph's MDS will probably do better than most due to its embedded inodes, but it's hardly ideal. However, and idea that has been kicking around here is building a mail storage system directly on top of RADOS. In principle, it should be a relatively straightforward matter of implementing a library and plugging it into the storage backend for something like Dovecot, or any other mail system (delivery agent and/or IMAP/POP frontend) with a pluggable backend. (I think postfix has pluggable delivery agents, but that's about where my experience in this area runs out.) The basic idea is this: - each mail message is a rados object, and immutable. - each mailbox is an index of messages, stored in a rados object. - the index consists of omap records, one for each message. - the key is some unique id - the value is a copy of (a useful subset of) the message headers This has a number of nice properties: - you can efficiently list messages in the mailbox using the omap operations - you can (more) efficiently search messages (everything but the message body) based on the index contents (since it's all stored in one object) - you can efficiently grab recent messages with the omap ops (e.g., list keys > last_seen_msgid) - moving messages between folders involves updating the indices only; the messages objects need not be copied/moved. - no metadata bottleneck: mailbox indices are distributed across the entire cluster, just like the mail. - all the scaling benefits of rados for a growing mail system. I don't know enough about what exactly the mail storage backends need to support to know what issues will come up. Presumably there are several. E.g., if you delete a message, is the IMAP client expected to discover that efficiently? And do the mail storage backends attempt to do it efficiently? This also doesn't solve the problem of efficiently indexing/searching the bodies of messages, although I suspect that indexing could be efficiently implemented on top of this scheme. So, a non-trivial project, but probably one that can be prototyped without that much pain, and one that would perform and scale drastically better than existing solutions I'm aware of. I'm hoping there are some motivated hackers lurking who understand the pain that is maildir/mail infrastructure... sage On Wed, 4 Jul 2012, Mitsue Acosta Murakami wrote: > Hello, > > We are examining Ceph to use as email storage. In our current system, several > clients servers with different services (imap, smtp, etc) access a NFS storage > server. The mailboxes are stored in Maildir format, with many small files. We > use Amazon AWS EC2 for clients and storage server. In this scenario, we have > some questions about Ceph: > > 1. Is Ceph recommended for heavy write/read of small files? > > 2. Is there any problem in installing Ceph on Amazon instances? > > 3. Does Ceph already support quota? > > 4. What File System would you encourage us to use? > > > Thanks in advance, > > -- > Mitsue Acosta Murakami > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rados mailbox? (was Re: Ceph for email storage)
On 04-07-12 22:40, Sage Weil wrote: Although Ceph fs would technically work for storing mail with maildir, when you step back from the situation, Maildir + a distributed file system is a pretty terrible way to approach mail storage. Maildir was designed to work around the limited consistency of NFS, and manages that, but performs pretty horribly on almost any file system. Mostly this is due to the message-per-file approach and the fact that file systems' internal management of inodes and directories mean lots and lots of seeks, even to read message headers. Ceph's MDS will probably do better than most due to its embedded inodes, but it's hardly ideal. However, and idea that has been kicking around here is building a mail storage system directly on top of RADOS. In principle, it should be a relatively straightforward matter of implementing a library and plugging it into the storage backend for something like Dovecot, or any other mail system (delivery agent and/or IMAP/POP frontend) with a pluggable backend. (I think postfix has pluggable delivery agents, but that's about where my experience in this area runs out.) When you first told me the idea about a couple of months ago I took a look at the Dovecot code and it's not that trivial to implement. It seems that mbox and Maildir are pretty hardcoded in Dovecot, but there is an advantage: You can use Dovecot as your LDA/VDA (Local/Virtual Delivery Agent) for Postfix, so you'd only have to implement this library in Dovecot and you'd be able to handle IMAP, POP3 and Delivery of e-mails to RADOS. Source: http://wiki.dovecot.org/LDA/Postfix The basic idea is this: - each mail message is a rados object, and immutable. - each mailbox is an index of messages, stored in a rados object. - the index consists of omap records, one for each message. - the key is some unique id - the value is a copy of (a useful subset of) the message headers This has a number of nice properties: - you can efficiently list messages in the mailbox using the omap operations - you can (more) efficiently search messages (everything but the message body) based on the index contents (since it's all stored in one object) - you can efficiently grab recent messages with the omap ops (e.g., list keys > last_seen_msgid) - moving messages between folders involves updating the indices only; the messages objects need not be copied/moved. - no metadata bottleneck: mailbox indices are distributed across the entire cluster, just like the mail. - all the scaling benefits of rados for a growing mail system. I don't know enough about what exactly the mail storage backends need to support to know what issues will come up. Presumably there are several. E.g., if you delete a message, is the IMAP client expected to discover that efficiently? And do the mail storage backends attempt to do it efficiently? With IMAP a message gets marked as deleted until your do a "PURGE", that will actually remove the message, Problem with IMAP clients however is that there are a lot of bugs in them, especially outlook. But if you can somehow plug into Dovecot and only handle the calls that it's doing you should be fine. This also doesn't solve the problem of efficiently indexing/searching the bodies of messages, although I suspect that indexing could be efficiently implemented on top of this scheme. Nowadays most clients keep a local cache, at least Thunderbird does and uses that for local search. Much faster! Webmail clients like RoundCube have a local cache as well and applications like OpenXchange also have local caches. So, a non-trivial project, but probably one that can be prototyped without that much pain, and one that would perform and scale drastically better than existing solutions I'm aware of. Yes, MUCH better than Maildir over CephFS or NFS. I'm hoping there are some motivated hackers lurking who understand the pain that is maildir/mail infrastructure... Plenty of motivation, not enough time I think. Wido sage On Wed, 4 Jul 2012, Mitsue Acosta Murakami wrote: Hello, We are examining Ceph to use as email storage. In our current system, several clients servers with different services (imap, smtp, etc) access a NFS storage server. The mailboxes are stored in Maildir format, with many small files. We use Amazon AWS EC2 for clients and storage server. In this scenario, we have some questions about Ceph: 1. Is Ceph recommended for heavy write/read of small files? 2. Is there any problem in installing Ceph on Amazon instances? 3. Does Ceph already support quota? 4. What File System would you encourage us to use? Thanks in advance, -- Mitsue Acosta Murakami -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsu
Re: rados mailbox? (was Re: Ceph for email storage)
Very short answer to this. It can work if you direct all email requests for a particular mailbox to a single machine. You need to avoid locking between servers as much as possible. Messages will need to be indexed, period. Or else your life will suck. Dovecot has a nice writeup on this type of thing; not Ceph specific, but NFS related..it can be extrapolated to Ceph or any distributed storage: http://wiki.dovecot.org/NFS On 07/04/2012 03:40 PM, Sage Weil wrote: Although Ceph fs would technically work for storing mail with maildir, when you step back from the situation, Maildir + a distributed file system is a pretty terrible way to approach mail storage. Maildir was designed to work around the limited consistency of NFS, and manages that, but performs pretty horribly on almost any file system. Mostly this is due to the message-per-file approach and the fact that file systems' internal management of inodes and directories mean lots and lots of seeks, even to read message headers. Ceph's MDS will probably do better than most due to its embedded inodes, but it's hardly ideal. However, and idea that has been kicking around here is building a mail storage system directly on top of RADOS. In principle, it should be a relatively straightforward matter of implementing a library and plugging it into the storage backend for something like Dovecot, or any other mail system (delivery agent and/or IMAP/POP frontend) with a pluggable backend. (I think postfix has pluggable delivery agents, but that's about where my experience in this area runs out.) The basic idea is this: - each mail message is a rados object, and immutable. - each mailbox is an index of messages, stored in a rados object. - the index consists of omap records, one for each message. - the key is some unique id - the value is a copy of (a useful subset of) the message headers This has a number of nice properties: - you can efficiently list messages in the mailbox using the omap operations - you can (more) efficiently search messages (everything but the message body) based on the index contents (since it's all stored in one object) - you can efficiently grab recent messages with the omap ops (e.g., list keys > last_seen_msgid) - moving messages between folders involves updating the indices only; the messages objects need not be copied/moved. - no metadata bottleneck: mailbox indices are distributed across the entire cluster, just like the mail. - all the scaling benefits of rados for a growing mail system. I don't know enough about what exactly the mail storage backends need to support to know what issues will come up. Presumably there are several. E.g., if you delete a message, is the IMAP client expected to discover that efficiently? And do the mail storage backends attempt to do it efficiently? This also doesn't solve the problem of efficiently indexing/searching the bodies of messages, although I suspect that indexing could be efficiently implemented on top of this scheme. So, a non-trivial project, but probably one that can be prototyped without that much pain, and one that would perform and scale drastically better than existing solutions I'm aware of. I'm hoping there are some motivated hackers lurking who understand the pain that is maildir/mail infrastructure... sage On Wed, 4 Jul 2012, Mitsue Acosta Murakami wrote: Hello, We are examining Ceph to use as email storage. In our current system, several clients servers with different services (imap, smtp, etc) access a NFS storage server. The mailboxes are stored in Maildir format, with many small files. We use Amazon AWS EC2 for clients and storage server. In this scenario, we have some questions about Ceph: 1. Is Ceph recommended for heavy write/read of small files? 2. Is there any problem in installing Ceph on Amazon instances? 3. Does Ceph already support quota? 4. What File System would you encourage us to use? Thanks in advance, -- Mitsue Acosta Murakami -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rados mailbox? (was Re: Ceph for email storage)
Am 10.07.2012 07:45, schrieb Kristofer: > Very short answer to this. > > It can work if you direct all email requests for a particular mailbox to a > single machine. You need to avoid locking between servers as much as possible. > > Messages will need to be indexed, period. Or else your life will suck. > > Dovecot has a nice writeup on this type of thing; not Ceph specific, but NFS > related..it can be extrapolated to Ceph or any distributed storage: > http://wiki.dovecot.org/NFS >> >> - each mail message is a rados object, and immutable. >> - each mailbox is an index of messages, stored in a rados object. >> - the index consists of omap records, one for each message. >> - the key is some unique id >> - the value is a copy of (a useful subset of) the message headers >> >> This has a number of nice properties: >> >> - you can efficiently list messages in the mailbox using the omap >> operations >> - you can (more) efficiently search messages (everything but the message >> body) based on the index contents (since it's all stored in one object) >> - you can efficiently grab recent messages with the omap ops (e.g., list >> keys > last_seen_msgid) >> - moving messages between folders involves updating the indices only; the >> messages objects need not be copied/moved. >> - no metadata bottleneck: mailbox indices are distributed across the >> entire cluster, just like the mail. >> - all the scaling benefits of rados for a growing mail system. >> >> I don't know enough about what exactly the mail storage backends need to >> support to know what issues will come up. Presumably there are several. >> E.g., if you delete a message, is the IMAP client expected to discover >> that efficiently? And do the mail storage backends attempt to do it >> efficiently? >> >> This also doesn't solve the problem of efficiently indexing/searching the >> bodies of messages, although I suspect that indexing could be efficiently >> implemented on top of this scheme. >> >> So, a non-trivial project, but probably one that can be prototyped without >> that much pain, and one that would perform and scale drastically better >> than existing solutions I'm aware of. >> >> I'm hoping there are some motivated hackers lurking who understand the >> pain that is maildir/mail infrastructure... >> Maybe another idea which could be done with few effort would be to mostly reuse the code from dbmail and make a cephmail version out of it. -- Mit freundlichen Grüßen, Smart Weblications GmbH Martinsberger Str. 1 D-95119 Naila fon.: +49 9282 9638 200 fax.: +49 9282 9638 205 24/7: +49 900 144 000 00 - 0,99 EUR/Min* http://www.smart-weblications.de -- Sitz der Gesellschaft: Naila Geschäftsführer: Florian Wiessner HRB-Nr.: HRB 3840 Amtsgericht Hof *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
