1) Since latency requirements are low, why did performance drop so
much previously when you implemented a very simple mysql storage
backend? I glanced at the code a few weeks ago and whilst it's
surprisingly complicated right now to implement a backend, I was also
surprised that a database storage engine "sucked" I think you phrased
it? Possibly the code also placed the indexes on the DB? Certainly
this could very well kill performance? (Note I'm not arguing sql
storage is a good thing, I just want to understand the latency to
backend requirements)
Yes, it placed indexes also to SQL. That's slow. But even without it,
Dovecot code needs to be changed to access more mails in parallel
before the performance can be good for high-latency mail storages.
My expectation then is that with local index and sql message storage
that the performance should be very reasonable for a large class of
users... (ok, other problems perhaps arise)
2) I would be thinking that with some care, even very high latency
storage would be workable, eg S3/Gluster/MogileFs ? I would love to
see a backend using S3 - If nothing else I think it would quickly
highlight all the bottlenecks in any design...
Yes, S3 should be possible. With dbox it could even be used to store
the old mails and keep new mails in lower latency storage.
Mogile doesn't handle S3, but I always thought it would be terrific to
be able to have one copy of your data on fast local storage, but to be
able to use slower (sometimes cheaper) storage for backups or less
valuable data (eg older messages), ie replicating some data to other
storage boxes
CouchDB seems like it would still be more difficult than necessary to
scale. I'd really just want something that distributes the load and
disk usage evenly across all servers and allows easily plugging in
more servers and it automatically rebalances the load. CouchDB seems
like much of that would have to be done manually (or building scripts
to do it).
Ahh fair enough - I thought it being massively multi-master would allow
simply querying different machines for different users. Not a perfect
scale-out, but good enough for a whole class of requirements...
For the filesystem backend have you looked at the various log
structured filesystems appearing? Whenever I watch the debate
between Maildir vs Mailbox I always think that a hybrid is the best
solution because you are optimising for a write one, read many
situation, where you have an increased probability of having good
cache localisation on any given read.
To me this ends up looking like log structured storage... (which
feels like a hybrid between maildir/mailbox)
Hmm. I don't really see how it looks like log structured storage.. But
you do know that multi-dbox is kind of a maildir/mbox hybrid, right?
Well the access is largely append only, with some deletes and noise at
the writing end, but largely the older storage stays static with much
longer gaps between deletes (and extremely infrequent edits)
So maildir is optimised really for deletes, but improves random access
for a subset of operations. Mailbox is optimised for writes and seems
like it's generally fast for most operations except deletes (people do
worry about having a lot of eggs in one basket, but I think this is
really a symptom of other problems at work). Mailbox also has improved
packing for small messages and probably has improved cache locality on
certain read patterns
So one obvious hybrid would be a mailbox type structure which perhaps
splits messages up into variable sized sub mailboxes based on various
criteria, perhaps including message age, type of message or message
size...? The rapid write delete would happen at the head, perhaps even
as a maildir layout and gradually the storage would become larger and
ever more compressed mailboxes as the age/frequency of access/etc declines.
Perhaps this is exactly dbox?
It would also be interesting to consier separate message headers from
body content. Have heavy localisation of message headers, and slower
higher latency access to the message body. Would this improve access
speeds in general? Also the mime structure could be torn apart to store
attachments individually - the motivation being single instance storage
of large attachments with identical content... Anyway, these seem like
very speculative directions...
I haven't really done any explicit benchmarks, but there are a few
reasons why I think low-latency for indexes is really important:
I think low latency for indexes is a given. You appear to have
architected the system so that all responses are delivered from the
index and baring an increase in index efficiency the remaining time is
spent doing the initial generation and maintenance of those indexes. I
would have thought bar downloading an entire INBOX that the access time
of individual mails was very much secondary?
- If the goal is performance by allowing a scale-out in quantity of
servers then I guess you need to measure it carefully to make sure it
actually works? I haven't had the fortune to develop something that
big, but the general advice is that scaling out is hard to get right,
so assume you made a mistake in your design somewhere... Measure,
measure
I don't think it's all that much about performance of a single user,
but more about distributing the load more evenly in an easier way.
That's basically done by outsourcing the problem to the underlying
storage (database).
So perhaps something like CouchDB can work then? One user localises per
replica and you keep reusing that replica?
Yes, resolving conflicts due to split brain merging back is something
I really want to make work as well as it can. The backend database can
hopefully again help here (by noticing there was a conflict and
allowing the program to resolve it).
In general conflict resolution is thrown back to the application, so
likely this is going to become a dovecot problem. It seems that the
general class of problem is too hard to solve at the storage side
This is also one of its goals :) Even if I make a mistake in choosing
a bad database first, it should be somewhat easy to implement another
backend again. The backend FS API will be pretty simple. Basically
it's going to be:
I wouldn't get too held back by posix semantics. For sure they are
memorable, but definitely consider that transactions are the key to any
kind of database performance improvement and make sure you can batch
together stuff to make good use of the backend. Your "flush" command
seems to be the implicit end of transaction, but I guess give it plenty
of thought that you might have a super slow system (eg S3) and the
backend might want the flexibility to mark something "kind of done",
while uploading for 30 seconds in the background, then marking it
properly done once S3 actually acks the data saved?
- Finally I am a bit sad that offline distributed multi-master isn't
in the roadmap anymore... :-(
I think dsync can do that. It'll do two-way syncing between Dovecots
and resolves all conflicts. Is the syncing itself still done with very
high latencies, i.e. something like USB sticks? That's currently not
really working, but it probably wouldn't be too difficult.
What is dsync? There is a dsync.org which is some kind of directory
synchroniser?
Aha, google suggests that I might have missed an email from you
recently... Will read up...
OK, this sounds like a better implementation of the kind of thing we are
building here - likely this is the way ahead!
Cheers
Ed W
