Re: [OT] Re: What's the best possible email failover solution
On Tue, 22 Jun 2004, Bill Moran wrote: > The other option is to take what appears to be the best IMAP server out > there (Cyrus) and figure out a way to do real-time mirroring of the > mailboxes. I was wondering if it could be done with Coda, but I don't > know anything about Coda, and it doesn't look like I'll have time to > experiment in the near future. A previous responder to this thread has already pointed at the cam.ac.uk work which offers transactional replication for fast fail-over. -- jan grant, ILRT, University of Bristol. http://www.ilrt.bris.ac.uk/ Tel +44(0)117 9287088 Fax +44 (0)117 9287112 http://ioctl.org/jan/ "Impact of vulnerability: Run code of an attacker's choice Maximum Severity Rating: Moderate" -- M$ security bulletin ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Bill Moran wrote: "David E. Meier" <[EMAIL PROTECTED]> wrote: Like I said, we'll never know till someone tries it. It looks like Dovecot is going to try it eventually, but it seems like they have other priorities at this time. "Someone" already stores mails in a database: Oracle (Email Server and Collaboration Suite). I set up the Oracle Email Server 5.2 for a company I worked for earlier. And to express it nicely: It was a nightmare! Mails got stuck and rejected because the system was not capable of writing them into the database. Besides, the support for that system was also =0. We were probably the only ones daring to run the system ;-) I am glad I am running cyrus now. Extremly stable and fast. That system was not well thought through at all. I don't know how much work needs to be done for a database email store, but Oracle wasn't (isn't) able to do so. MS Exchange Server also stores email in a database. This has some great benefits, but also some horrible side effects. If your database gets corrupted, you are in deep doodoo, and back in the days when I administered an Exchange Server, that happened all too often. As someone mentioned, Cyrus takes a middle approach: it doesn't store the email in a database, but maintains a database that tracks the email to improve speed. It seems to scale well. It's a shame it wasn't an OSS project, so we could determine if keeping mail in a database is a bad idea, or if Oracle just did it poorly. The other option is to take what appears to be the best IMAP server out there (Cyrus) and figure out a way to do real-time mirroring of the mailboxes. I was wondering if it could be done with Coda, but I don't know anything about Coda, and it doesn't look like I'll have time to experiment in the near future. I played with Coda a few years ago, and it seems to have a lot of potential. I keep promising myself I will look at it again. I recently thought of using it for precisely this scenario, but I had only one server with two drives, and Coda didn't seem to be the way to go in that case. In fact, I just use rsync nightly, and figure I can live with the loss of one day's mail if something dies. There are other distributed filesystems out there in the Linux world, but (IIRC) not many for FreeBSD. I think it is worth looking at Coda for this purpose, although it may have some downside (e.g. not real-time enough) that makes it inappropriate. - Bob ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Chuck Swiger <[EMAIL PROTECTED]> wrote: > Bill Moran wrote: > > Christian Laursen <[EMAIL PROTECTED]> wrote: > >> If you are running FreeBSD 5, you should be able to make a filesystem snapshot > >> and rsync from there. > > > > I suppose I should have commented on that ;) > > > > We're not running FreeBSD 5 on these production machines yet ... but it's > > likely we will be soon, so I'm considering using snapshots. > > > > To my understanding, we still have to stop Cyrus while the snapshot is > > being created (to ensure consistency) but since a snapshot takes a lot > > less time than an rsync, this should be a big improvement. Once the > > snapshot is created, rsync can take as long as necessary. > > No, snapshots can be taken without significantly interrupting running > processes, although I'm not sure how long filesystem access gets blocked while > creating the snapshot. You could also detach a RAID-1 mirror of the data > (using vinum, ccd, whatever) and backup that, and then re-attach and resync > the mirror drive to the live volume. > > Both of these methods make taking a very current backup easy; they do not > provide live replication of the data, however. Sort of. The Cyrus docs recommend this, with the warning that "you can potentially have a couple of inconsistent mailboxes in the snapshot" Me, I'd rather risk 5 minutes of Cyrus being down in the middle of the night than risk an inconsistent backup. Knowing my luck, the inconsistent part would be the one folder that causes a big stir when it can't be restored. I figure, if a snapshot takes (let's say) 20 seconds to complete. You stop cyrus, take the snapshot, then restart Cyrus ... you can then take all the time you need to backup from the snapshot. You're looking at less than 5 minutes fo downtime. In our case, our customers are mostly regional, so doing this at 2:00 AM shouldn't disrupt much. -- Bill Moran Potential Technologies http://www.potentialtech.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Bill Moran wrote: Christian Laursen <[EMAIL PROTECTED]> wrote: If you are running FreeBSD 5, you should be able to make a filesystem snapshot and rsync from there. I suppose I should have commented on that ;) We're not running FreeBSD 5 on these production machines yet ... but it's likely we will be soon, so I'm considering using snapshots. To my understanding, we still have to stop Cyrus while the snapshot is being created (to ensure consistency) but since a snapshot takes a lot less time than an rsync, this should be a big improvement. Once the snapshot is created, rsync can take as long as necessary. No, snapshots can be taken without significantly interrupting running processes, although I'm not sure how long filesystem access gets blocked while creating the snapshot. You could also detach a RAID-1 mirror of the data (using vinum, ccd, whatever) and backup that, and then re-attach and resync the mirror drive to the live volume. Both of these methods make taking a very current backup easy; they do not provide live replication of the data, however. -- -Chuck ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Bill Moran <[EMAIL PROTECTED]> writes: > Christian Laursen <[EMAIL PROTECTED]> wrote: > [snip] > > > If you are running FreeBSD 5, you should be able to make a filesystem snapshot > > and rsync from there. > > I suppose I should have commented on that ;) > > We're not running FreeBSD 5 on these production machines yet ... but it's > likely we will be soon, so I'm considering using snapshots. > > To my understanding, we still have to stop Cyrus while the snapshot is > being created (to ensure consistency) but since a snapshot takes a lot > less time than an rsync, this should be a big improvement. Once the > snapshot is created, rsync can take as long as necessary. Actually it is not neccesary to stop Cyrus while the snapshot is created. If it tries to use the filesystem during that short period it will block until the snapshot is fully created. -- Christian Laursen ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Christian Laursen <[EMAIL PROTECTED]> wrote: > Bill Moran <[EMAIL PROTECTED]> writes: > > > Nico Meijer <[EMAIL PROTECTED]> wrote: > > > > > Hi Bill, > > > > > > > The other option is to take what appears to be the best IMAP server out > > > > there (Cyrus) and figure out a way to do real-time mirroring of the > > > > mailboxes. > > > > > > Depending on the size / number of messages: how about using rsync and > > > OpenBSD's CARP? > > > > > > True, it will not be realtime, but the synchronization (note the > > > "depending" above) might take place regularly. > > > > Problem is: rsyncing the mail directories takes about 20 minutes, and the > > only way to ensure that a good copy is achieved is to shut down Cyrus > > during the backup. This makes it a little prohibitive to be doing this > > very often. > > If you are running FreeBSD 5, you should be able to make a filesystem snapshot > and rsync from there. I suppose I should have commented on that ;) We're not running FreeBSD 5 on these production machines yet ... but it's likely we will be soon, so I'm considering using snapshots. To my understanding, we still have to stop Cyrus while the snapshot is being created (to ensure consistency) but since a snapshot takes a lot less time than an rsync, this should be a big improvement. Once the snapshot is created, rsync can take as long as necessary. -- Bill Moran Potential Technologies http://www.potentialtech.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Bill Moran <[EMAIL PROTECTED]> writes: > Nico Meijer <[EMAIL PROTECTED]> wrote: > > > Hi Bill, > > > > > The other option is to take what appears to be the best IMAP server out > > > there (Cyrus) and figure out a way to do real-time mirroring of the > > > mailboxes. > > > > Depending on the size / number of messages: how about using rsync and > > OpenBSD's CARP? > > > > True, it will not be realtime, but the synchronization (note the > > "depending" above) might take place regularly. > > Problem is: rsyncing the mail directories takes about 20 minutes, and the > only way to ensure that a good copy is achieved is to shut down Cyrus > during the backup. This makes it a little prohibitive to be doing this > very often. If you are running FreeBSD 5, you should be able to make a filesystem snapshot and rsync from there. -- Christian Laursen ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Nico Meijer <[EMAIL PROTECTED]> wrote: > Hi Bill, > > > The other option is to take what appears to be the best IMAP server out > > there (Cyrus) and figure out a way to do real-time mirroring of the > > mailboxes. > > Depending on the size / number of messages: how about using rsync and > OpenBSD's CARP? > > True, it will not be realtime, but the synchronization (note the > "depending" above) might take place regularly. Problem is: rsyncing the mail directories takes about 20 minutes, and the only way to ensure that a good copy is achieved is to shut down Cyrus during the backup. This makes it a little prohibitive to be doing this very often. -- Bill Moran Potential Technologies http://www.potentialtech.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Hi Bill, The other option is to take what appears to be the best IMAP server out there (Cyrus) and figure out a way to do real-time mirroring of the mailboxes. Depending on the size / number of messages: how about using rsync and OpenBSD's CARP? True, it will not be realtime, but the synchronization (note the "depending" above) might take place regularly. HTH... Nico ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
"David E. Meier" <[EMAIL PROTECTED]> wrote: > > Like I said, we'll never know till someone tries it. It looks like > > Dovecot is going to try it eventually, but it seems like they have > > other priorities at this time. > > "Someone" already stores mails in a database: Oracle (Email Server and > Collaboration Suite). I set up the Oracle Email Server 5.2 for a company I > worked for earlier. And to express it nicely: It was a nightmare! Mails > got stuck and rejected because the system was not capable of writing them > into the database. Besides, the support for that system was also =0. We > were probably the only ones daring to run the system ;-) I am glad I am > running cyrus now. Extremly stable and fast. > > That system was not well thought through at all. I don't know how much > work needs to be done for a database email store, but Oracle wasn't > (isn't) able to do so. It's a shame it wasn't an OSS project, so we could determine if keeping mail in a database is a bad idea, or if Oracle just did it poorly. The other option is to take what appears to be the best IMAP server out there (Cyrus) and figure out a way to do real-time mirroring of the mailboxes. I was wondering if it could be done with Coda, but I don't know anything about Coda, and it doesn't look like I'll have time to experiment in the near future. -- Bill Moran Potential Technologies http://www.potentialtech.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
> The other advantages is it would scale like nobody's business. Since the > data is in postgres, you could use multiple backends (replicated with > Slony) > and have the IMAP daemons contact different back ends if the load got > heavy. With a little work, the system could failover silently as well. It would be very nice indeed. > Like I said, we'll never know till someone tries it. It looks like > Dovecot is going to try it eventually, but it seems like they have > other priorities at this time. "Someone" already stores mails in a database: Oracle (Email Server and Collaboration Suite). I set up the Oracle Email Server 5.2 for a company I worked for earlier. And to express it nicely: It was a nightmare! Mails got stuck and rejected because the system was not capable of writing them into the database. Besides, the support for that system was also =0. We were probably the only ones daring to run the system ;-) I am glad I am running cyrus now. Extremly stable and fast. That system was not well thought through at all. I don't know how much work needs to be done for a database email store, but Oracle wasn't (isn't) able to do so. Dave ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Chuck Swiger <[EMAIL PROTECTED]> wrote: > Bill Moran wrote: > > Chuck Swiger <[EMAIL PROTECTED]> wrote: > >>[ I don't think that stuffing email into a database is a particularly good > >>idea since that means keeping large blobs of non-relational data floating > >>around, something that the filesystem can do a better job of handling... ] > [ ... ] > > During my research of the IMAP protocol, I determined that _the_best_ > > way to store email for high-performance would be to put them in a > > database. This is because IMAP doesn't see email as a big blob of > > text like POP does. It sees the headers as one thing, and the > > different MIME parts of the email each as a seperate thing that can > > be fetched independently of the other MIME parts. This is a pretty > > good layout for a one -> many relationship in a database. Fact is, > > every current IMAP server that I'm aware of has to break emails > > apart on the fly in order to server IMAP. > > There's nothing wrong with applying database concepts to email, and it sounds > like you want things which take advantage of database replication and > transaction management and so forth in order to gain reliability, so perhaps > you will find a DB better suited for your requirements than my comments above > suggest. > > I don't mind being wrong when the result works better for someone. However, > please remember that I know you are an optimist if you think I am a pessimist. > > :-) > > > Now, I could be wrong on this count, as I never wrote the mailserver, > > so my theory could ultimately be proven wrong, but I guess I just > > don't agree with the statement that SQL is a bad way to store email > > until someone has actually proven it. > > My concern has less to do with the suitability of using a database to store > mail as it has to do with database transactions becoming a potential > bottleneck on the system as a whole. Agreed ... and (now that I'm thinking about it) that's why I decided to write a mail server. IIRC, I started researching IMAP, and started wondering how well it would work with a database back end, so I decided to write one to see how _well_ it worked ... then reality intervened and I didn't get to have any fun ... > In the case of storing email in a DB, while you can break up a mail message > into headers plus seperate MIME components, are you really going to want to > decompose each and every mail message in a 3GB mail volume like that? > Although if you throw enough RAM at a DB so that the entire thing fits into > main memory, that can produce some spectacular results, and is almost doable > for this specific case. > > Anyway, consider each time someone reads a message from the DB, you'd have to > do two or three database transactions per message, maybe more, compared with > read()ing or mmap()ing a single file in an IMAPD and doing strnstr()s for MIME > boundary seperators in C. Remember that hitting the DB involves multiprocess > IPC and adds a lot of latency compared to what a filesystem-based IMAP daemon > does. Possibly. But most IMAP clients cache a lot of stuff, so it'll only grab most things once. The other advantages is it would scale like nobody's business. Since the data is in postgres, you could use multiple backends (replicated with Slony) and have the IMAP daemons contact different back ends if the load got heavy. With a little work, the system could failover silently as well. Like I said, we'll never know till someone tries it. It looks like Dovecot is going to try it eventually, but it seems like they have other priorities at this time. -- Bill Moran Potential Technologies http://www.potentialtech.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Jan Grant <[EMAIL PROTECTED]> wrote: > On Mon, 21 Jun 2004, Bill Moran wrote: > > > During my research of the IMAP protocol, I determined that _the_best_ > > way to store email for high-performance would be to put them in a > > database. This is because IMAP doesn't see email as a big blob of > > text like POP does. It sees the headers as one thing, and the > > different MIME parts of the email each as a seperate thing that can > > be fetched independently of the other MIME parts. This is a pretty > > good layout for a one -> many relationship in a database. Fact is, > > every current IMAP server that I'm aware of has to break emails > > apart on the fly in order to server IMAP. > > Have a closer look at the cyrus layout. Each message is in a single > file, true, but they are also preparsed to extract the data required for > common IMAP operations. The index files contain things like preformed > bodystructure responses and offsets to each mime piece. That would explain why Cyrus is so fast then. If only there was a way to do replication ... it'd be the perfect IMAP server. -- Bill Moran Potential Technologies http://www.potentialtech.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
On Mon, 21 Jun 2004, Bill Moran wrote: > During my research of the IMAP protocol, I determined that _the_best_ > way to store email for high-performance would be to put them in a > database. This is because IMAP doesn't see email as a big blob of > text like POP does. It sees the headers as one thing, and the > different MIME parts of the email each as a seperate thing that can > be fetched independently of the other MIME parts. This is a pretty > good layout for a one -> many relationship in a database. Fact is, > every current IMAP server that I'm aware of has to break emails > apart on the fly in order to server IMAP. Have a closer look at the cyrus layout. Each message is in a single file, true, but they are also preparsed to extract the data required for common IMAP operations. The index files contain things like preformed bodystructure responses and offsets to each mime piece. -- jan grant, ILRT, University of Bristol. http://www.ilrt.bris.ac.uk/ Tel +44(0)117 9287088 Fax +44 (0)117 9287112 http://ioctl.org/jan/ HP-unix: Open Sauce product, available in 57 distributions. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Just a thought, but couldn't you write the imapd process to act more like a web application server in the RDBMS scenario. You can cache data and limit the number of select statements executed on the actual data store. Although one wouldn't have something like cookies for sessions, the username and other characteristics of the message could be used to create a hash identifying the data in the imap server. Imap clients also tend to retrieve the headers only and then retrieve message bodies if someone "reads" the message. For most clients, caching the headers might be a good idea. Of course the timeout value couldn't be to large or they wouldn't get the newest messages in a reasonable time. You could also seperate the cache from the imap daemon similar to how livejournal.com uses a seperate caching service to limit the overhead on the mysql servers for large mail deployments. Its similar in the sense people want the most recent journal entries just like they want their new messages. The other advantage to a mail server implemented as a database is that one could add groupwise type functionality as pluggable modules that tied in with the data store. Its overkill, but could be rather neat. Lucas Holt [EMAIL PROTECTED] FoolishGames.com (Jewel Fan Site) JustJournal.com (Free blogging) ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: [OT] Re: What's the best possible email failover solution
Bill Moran wrote: Chuck Swiger <[EMAIL PROTECTED]> wrote: [ I don't think that stuffing email into a database is a particularly good idea since that means keeping large blobs of non-relational data floating around, something that the filesystem can do a better job of handling... ] [ ... ] During my research of the IMAP protocol, I determined that _the_best_ way to store email for high-performance would be to put them in a database. This is because IMAP doesn't see email as a big blob of text like POP does. It sees the headers as one thing, and the different MIME parts of the email each as a seperate thing that can be fetched independently of the other MIME parts. This is a pretty good layout for a one -> many relationship in a database. Fact is, every current IMAP server that I'm aware of has to break emails apart on the fly in order to server IMAP. There's nothing wrong with applying database concepts to email, and it sounds like you want things which take advantage of database replication and transaction management and so forth in order to gain reliability, so perhaps you will find a DB better suited for your requirements than my comments above suggest. I don't mind being wrong when the result works better for someone. However, please remember that I know you are an optimist if you think I am a pessimist. :-) Now, I could be wrong on this count, as I never wrote the mailserver, so my theory could ultimately be proven wrong, but I guess I just don't agree with the statement that SQL is a bad way to store email until someone has actually proven it. My concern has less to do with the suitability of using a database to store mail as it has to do with database transactions becoming a potential bottleneck on the system as a whole. I've spent a great deal of time in my day job dealing with dynamic websites, which mostly means ones driven by content generated by a database. In my experience, you want to provide static content as efficiently as possible, and reserve database transactions for persisting changes to state and answering relational queries. The most relevant comparison is one involving a site where people can search for images by keyword, which someone was also storing in the database. The idea works fine under light to moderate load, but it turns out that keeping just the "relational" part of the image data (name, keywords, etc) and a filesystem reference, and generating a link using that path for Apache to serve directly scales much better. --- In the case of storing email in a DB, while you can break up a mail message into headers plus seperate MIME components, are you really going to want to decompose each and every mail message in a 3GB mail volume like that? Although if you throw enough RAM at a DB so that the entire thing fits into main memory, that can produce some spectacular results, and is almost doable for this specific case. Anyway, consider each time someone reads a message from the DB, you'd have to do two or three database transactions per message, maybe more, compared with read()ing or mmap()ing a single file in an IMAPD and doing strnstr()s for MIME boundary seperators in C. Remember that hitting the DB involves multiprocess IPC and adds a lot of latency compared to what a filesystem-based IMAP daemon does. -- -Chuck ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
[OT] Re: What's the best possible email failover solution
Chuck Swiger <[EMAIL PROTECTED]> wrote: > Bill Moran wrote: > > Chuck Swiger <[EMAIL PROTECTED]> wrote: > [ ... ] > >> The latter uses one-message-per-file, and ought to work *much* better both in > >> terms of performance and stability, and in terms of playing nice with the way > >> rsync wants to back things up. > > > > Doesn't really matter. Fact is, the mail directories are something on the order > > of 3G. No matter how efficiently I store them, rsync is not going to be able > > to back them up fast enough to hit the level of redundancy I'm shooting for. > > You may well be right, as you aren't really talking about performing backups, > you're talking about creating a fully redundant storage which is kept > up-to-date in realtime. > > > Although Maildirs might work a little better, since I wouldn't have to stop > > the IMAP server during backup. > > That, and the granularity of one-message-per-file fits perfectly with rsync's > file-driven model. I don't know about that. Last I checked, many files resulted in rsync taking a lot of time, and a lot of memory to build a file list. I can't say whether the end result is more efficient or not, as I've never tested it, but rsync does intelligently move portions of files, instead of the whole file, when changes occur. > > It takes about 30 minutes to rsync the system to the backup server right now. > > That's perfectly acceptable for nightly backup purposes. This is a 1.5Ghz > > with 256M RAM and 80G ATA 100 HDDs. If the system runs rysnc continuously > > 24/7, I still have 30mins old data. > > Oh, yes. Just don't forget that if you do eliminate this time gap, you still > ought to have another system actually taking backups. Any change the system > encounters will be replicated to the redundant mail storage system in real > time, including bad changes. Hehe ... I've been trying to explain that to some of my less intelligent clients for a while now ... "Yes, when you do something wrong, it backs that up as well ..." Actually had a client ask me once why the backup didn't know the difference between something it should be backing up, and something that shouldn't be backed up. I told him if I knew how to do that, I'd be a lot richer! > >>[ I don't think that stuffing email into a database is a particularly good > >>idea since that means keeping large blobs of non-relational data floating > >>around, something that the filesystem can do a better job of handling... ] > > > > It's a good idea if I want real-time redundancy. I see where you're coming > > from, and it's true that a RDBMS isn't the best way to store emails. But, > > when you look at the features available, it's the best way for this > > circumstance. With something like Slony, I'd have real-time redundancy > > with (I'm expecting) only a minor performance drop. Although I can't be > > sure until I can put something together to test. Reliability is much more > > important than performance in this case. Who cares if their email takes > > and extra 60 seconds to deliver, as long as it doesn't get lost! If the > > email arrives fast, it's useless if the server fails and the email is > > lost because the SMTP server told the delivering server that it had > > arrived and then crashed before it could be backed up. > > I suspect that the relatively heavy weight of database transactions compared > with filesystem access is going to slow things down a fair amount, too, > particularly when running against a replicated DB. But reliability over > performance is a fine choice to make. :-) > > Using RAID improves fault-tolerance, but you still end up with a > single-point-of-failure at the system level; using database replication gives > you higher availability, which seems to be what you mean when you talk about > "reliability". Perhaps SAN or NAS concepts might be worth considering, as you > can set up a fully-redundant fibre channel configuration where the storage is > shared between two or more systems, thus with no single-point-of-failure. Problem there is cost ... that hardware is a bit pricey, compared to PC hardware, anyway. -- Bill Moran Potential Technologies http://www.potentialtech.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
[OT] Re: What's the best possible email failover solution
Chuck Swiger <[EMAIL PROTECTED]> wrote: > [ I don't think that stuffing email into a database is a particularly good > idea since that means keeping large blobs of non-relational data floating > around, something that the filesystem can do a better job of handling... ] Actually ... you got me thinking. I did some research about a year ago because I was going to write a mail server. It was mainly going to be an education project so I could learn some things. I'd forgotten about this until now. During my research of the IMAP protocol, I determined that _the_best_ way to store email for high-performance would be to put them in a database. This is because IMAP doesn't see email as a big blob of text like POP does. It sees the headers as one thing, and the different MIME parts of the email each as a seperate thing that can be fetched independently of the other MIME parts. This is a pretty good layout for a one -> many relationship in a database. Fact is, every current IMAP server that I'm aware of has to break emails apart on the fly in order to server IMAP. Now, I could be wrong on this count, as I never wrote the mailserver, so my theory could ultimately be proven wrong, but I guess I just don't agree with the statement that SQL is a bad way to store email until someone has actually proven it. -- Bill Moran Potential Technologies http://www.potentialtech.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
