On 10/03/2010 21:19, Timo Sirainen wrote:
On 10.8.2009, at 20.01, Timo Sirainen wrote:
(3.5. Implement async I/O filesystem backend.)
You know what I found out today? Linux doesn't support async IO for regular
buffered files. I had heard there were issues, but I thought it was
b) Use threads, either via some library or implement yourself. Each thread of
course uses some extra memory. Also enabling threads causes glibc to start
using a thread-safe version of malloc() (I think?), which slows things down
(unless that can be avoided, maybe by using clone() directly
On 10.8.2009, at 20.01, Timo Sirainen wrote:
(3.5. Implement async I/O filesystem backend.)
You know what I found out today? Linux doesn't support async IO for regular
buffered files. I had heard there were issues, but I thought it was mainly
about some annoying APIs and such. Anyone know if
Timo Sirainen put forth on 3/10/2010 3:19 PM:
On 10.8.2009, at 20.01, Timo Sirainen wrote:
(3.5. Implement async I/O filesystem backend.)
You know what I found out today? Linux doesn't support async IO for regular
buffered files. I had heard there were issues, but I thought it was mainly
Hi Folks,
How scalable is dovecot. Do you think a machine with a dual core 2.8Ghz
processor and 2GB of RAM would do a business with 600 users? This server
will have a 100Mbps link to the net, however it will be access by the
business using a couple of ADSL lines (Roughly 7Mbps each).
This
On Feb 12, 2010, at 4:57 PM, Jonathan Tripathy wrote:
How scalable is dovecot. Do you think a machine with a dual core
2.8Ghz processor and 2GB of RAM would do a business with 600 users?
This server will have a 100Mbps link to the net, however it will be
access by the business using a
Quoting Jonathan Tripathy jon...@abpni.co.uk:
How scalable is dovecot.
Very, but also it will vary by how you implement it (mbox or maildir,
local or remote disk, etc).
Do you think a machine with a dual core 2.8Ghz processor and 2GB of
RAM would do a business with 600 users?
Yes, unless
On Mon, 2009-08-10 at 14:33 -0700, Seth Mattinen wrote:
Nothing forces you to switch from maildir, if you're happy with it :)
But if you want to support millions of users, it's simpler to distribute
the storage and disk I/O evenly across hundreds of servers using a
database that was
paulmon wrote:
My current thinking if having the local delivery break messages up into
their component pieces, headers, from address, to address, spam scores, body
etc into various key:value relationships.
Whilst this looks appealing on the surface I think the details are going
to need some
On Mon, 2009-09-28 at 09:00 -0700, paulmon wrote:
My current thinking is a key/value store as you've proposed. Something like
Hadoop components or Project Voldamort. Voldamort might be a better fit
from what I've read.
My understanding of Hadoop is that it's more about distributed computing
On Mon, 2009-09-28 at 17:57 +0100, Ed W wrote:
My only request to Timo was to kind of consider that a bunch of these
ideas from the audience will almost certainly involve splitting up the
mime message into component parts and that the abstracted interface
should try not to throw away any
Timo Sirainen wrote:
On Mon, 2009-09-28 at 09:00 -0700, paulmon wrote:
My current thinking is a key/value store as you've proposed. Something like
Hadoop components or Project Voldamort. Voldamort might be a better fit
from what I've read.
My understanding of Hadoop is that it's
Timo Sirainen wrote:
On Mon, 2009-09-28 at 17:57 +0100, Ed W wrote:
My only request to Timo was to kind of consider that a bunch of these
ideas from the audience will almost certainly involve splitting up the
mime message into component parts and that the abstracted interface
should try
On Mon, 2009-09-28 at 18:35 +0100, Ed W wrote:
I would have thought that your API will prefer to request message parts
where it can (eg header, body, mime part), and just issue a read_bytes,
where that's what the client is asking for otherwise. This would allow
the storage engine to
On 9/28/2009, Ed W (li...@wildgooses.com) wrote:
In theory Thunderbird does this, but at least on my machine it just
repeatedly downloads the same message again and again in various ways
- it grinds to a halt every time I click on an email with a decent
sized attachment, even if I have already
On Mon, 2009-09-28 at 19:21 +0100, Ed W wrote:
In my mind this is more about what lib-storage API was supposed to
abstract out, whereas my filesystem API would be used simply for binary
data storage. The same FS API could be used to store both dbox files and
index files.
I guess in
Timo Sirainen wrote:
On Mon, 2009-09-28 at 19:21 +0100, Ed W wrote:
In my mind this is more about what lib-storage API was supposed to
abstract out, whereas my filesystem API would be used simply for binary
data storage. The same FS API could be used to store both dbox files and
index files.
On Mon, 2009-09-28 at 20:11 +0100, Ed W wrote:
lib-storage API has existed since Dovecot v1.0 and it's used to abstract
out access to maildir, mbox, dbox, cydir, etc. SQL would fit right there
with those.
OK, I thought that was what you were going to be simplifying...
Nope. It can
On 9/28/2009 4:24 PM, Jeff Grossman wrote:
In fact there are lots of IMAP improvements in v3... I can't wait until
all my extensions catch up, and I figure out how to customize the UI the
way I want (e.g., how in the world do I get rid of the stupid Tabs??)
You can't get rid of tabs per se,
Timo Sirainen wrote:
The SQL code was for v1.0 and the lib-storage API has been simplified
since then, maybe not hugely but still pretty much. Maybe some day I'll
see about updating the SQL code for v2.0 API.
Oh and some documentation about it would probably help a lot too. I
guess I should
Hi,
I´m reading the past topics related to archive and scalability of dovecot,
they are all very interesting. Here, I´m using two dovecot proxies in
front of five storages pairs, and we split the domain´s accounts among
those servers. So, we can share the i/o load and if one server goes down
only
On Tue, 2009-08-25 at 11:00 -0300, ferna...@dfcom.com.br wrote:
this is very weird and (at same time) very interesting approach. Instead
of put all messages into one maildir and this maildir into one server,
this maildir (?) is spplited among many servers - so, if one servers
fails the account
Hi Timo,
Yes it´s related, but I don´t understand '... You'll just need to
implement a filesystem that allows distributing a single user's mails to
multiple servers ...'.
My idea is just in the direction that we don´t need to care about
filesystem, we don´t need any distribuited filesystem...let
On Tue, 2009-08-25 at 12:12 -0300, ferna...@dfcom.com.br wrote:
Hi Timo,
Yes it´s related, but I don´t understand '... You'll just need to
implement a filesystem that allows distributing a single user's mails to
multiple servers ...'.
My idea is just in the direction that we don´t need to
I´m not sure if i´m being quite clear about my solution (pardon my
english)...
I´m not thinking in any filesystem abstraction layer, I´m afraid of this
kind of solution. If some corruption happens we can loose data - and the
crash-recovery is not easy.
I´m suggesting the opposite, to not have an
On Tue, 2009-08-25 at 12:41 -0300, ferna...@dfcom.com.br wrote:
I´m suggesting the opposite, to not have an abstraction layer. The
abstraction layer would be the proxy - dovecot does the abstraction layer.
When I´m telling 'proxy' I meaning something like the proxy feature of the
dovecot.
ferna...@dfcom.com.br wrote:
I´m reading the past topics related to archive and scalability of dovecot,
they are all very interesting. Here, I´m using two dovecot proxies in
front of five storages pairs, and we split the domain´s accounts among
those servers. So, we can share the i/o load and if
Hi Timo,
Do you have any experience using Dovecot and Glusterfs ? How about the
maturity of Glusterfs in a huge email environment ?
Fernando
On Tue, 2009-08-25 at 12:12 -0300, ferna...@dfcom.com.br wrote:
Hi Timo,
Yes it´s related, but I don´t understand '... You'll just need to
On Tue, 2009-08-25 at 14:13 -0300, ferna...@dfcom.com.br wrote:
Hi Timo,
Do you have any experience using Dovecot and Glusterfs ?
Not really.
How about the
maturity of Glusterfs in a huge email environment ?
No idea.
signature.asc
Description: This is a digitally signed message part
On Wed, 2009-08-12 at 18:18 -0400, Timo Sirainen wrote:
On Wed, 2009-08-12 at 14:54 -0700, Daniel L. Miller wrote:
If every attachment in a given message is individually scanned to
generate some unique identifier, and that identifier then used to
determine whether or not it exists in the
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Wed, 12 Aug 2009, Timo Sirainen wrote:
So yeah, either one are as reliable as the script :)
Well, I think this is not what the OP intended :)
This will work for only a little amount of mails I see, because:
a) forwarded messages differ,
b)
But for example, what I'd really like to be able to do is say something
like:
SiS_mode = binary,64K
So only binary attachments over 64KB in size would be checksummed and
single instance stored. I don't care bout the headers, or body text, or
tiny ( 64KB) sig attachments, or text
On 8/12/2009 3:58 PM, Timo Sirainen wrote:
- and I've never heard of any other mail server supporting such a
thing.
Exchange does... and is the single and only reason I have *considered*
switching to it shudder in the past few years...
I heard that the next Exchange version drops that
On 8/12/2009 4:02 PM, Timo Sirainen wrote:
Do you need per-MIME part single instance storage, or would per-email be
enough? Since the per-email can already done with hard links.
The only thing I can find about this on the wiki is where it says single
instance attachment storage (for dbox) is
On 8/12/2009, Ed W (li...@wildgooses.com) wrote:
It also had a bunch of limitations, it was basically only single
instance for CC recipients on a message (more or less). Quite a lot
of things such as certain types of virus scanning would (I think)
easily disable the single instance storage
On 8/12/2009, Daniel L. Miller (dmil...@amfes.com) wrote:
Under the structure I've proposed, net storage consumed by the
attachments should be one copy of attachment 1, and one copy of
attachment two, plus headers and any comments in the messages times
the number of recipients. Domino would
On Aug 13, 2009, at 8:13 AM, Charles Marcus wrote:
I'm also having trouble figuring out how using hard links (or their
equivalent) for messages with large attachments and having only one
instance of the attachment could cause *more* disk I/O than having
dozens/hundred of multiple copies of the
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Tue, 11 Aug 2009, Eric Jon Rostetter wrote:
For a massively scaled system, there may be sufficient performance to
put the queues elsewhere.
Which also allows that the queue can easily have multiple machines pushing
poping items.
But on a
On Aug 12, 2009, at 11:26 AM, Ed W wrote:
Hi
* Mail data on the other hand is just written once and usually read
maybe once or a couple of times. Caching mail data in memory probably
doesn't help all that much. Latency isn't such a horrible issue as
long
as multiple mails can be fetched at
On Aug 12, 2009, at 2:21 AM, Steffen Kaiser skdove...@smail.inf.fh-brs.de
wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Tue, 11 Aug 2009, Eric Jon Rostetter wrote:
For a massively scaled system, there may be sufficient performance to
put the queues elsewhere.
Which also allows
1) Since latency requirements are low, why did performance drop so
much previously when you implemented a very simple mysql storage
backend? I glanced at the code a few weeks ago and whilst it's
surprisingly complicated right now to implement a backend, I was also
surprised that a database
On Wed, 2009-08-12 at 17:46 +0100, Ed W wrote:
My expectation then is that with local index and sql message storage
that the performance should be very reasonable for a large class of
users... (ok, other problems perhaps arise)
If messages are stored to SQL in dummy blobs then the
CouchDB seems like it would still be more difficult than necessary to
scale. I'd really just want something that distributes the load and
disk usage evenly across all servers and allows easily plugging in
more servers and it automatically rebalances the load. CouchDB seems
like much of that
On Wed, 2009-08-12 at 18:42 +0100, Ed W wrote:
Something like that. In dbox you have one storage directory containing
all mailboxes' mails (so that copying can be done by simple index
updates). Then you have a bunch of files, each about n MB (configurable,
2 MB by default). Expunging
Oh, have you considered some optional api calls in the storage API?
The logic might be to assume that someone wanted to do something
clever and split the message up in some way, eg store headers
separately to bodies or bodies carved up into mime parts. The
motivation would be if there was a
Timo Sirainen wrote:
Also the mime structure could be torn apart to store
attachments individually - the motivation being single instance storage
of large attachments with identical content... Anyway, these seem like
very speculative directions...
Yes, this is also something in dbox's
On Wed, 2009-08-12 at 19:19 +0100, Ed W wrote:
I actually thought your idea of having a bunch of cut down IMAP type
servers as the backend storage talking to a bunch of beefier frontend
servers was quite an interesting idea!
Certainly though a simplification of the on-disk API would
On Wed, 2009-08-12 at 11:35 -0700, Daniel L. Miller wrote:
Timo Sirainen wrote:
Also the mime structure could be torn apart to store
attachments individually - the motivation being single instance storage
of large attachments with identical content... Anyway, these seem like
very
Daniel L. Miller wrote:
Timo Sirainen wrote:
Also the mime structure could be torn apart to store attachments
individually - the motivation being single instance storage of large
attachments with identical content... Anyway, these seem like very
speculative directions...
Yes, this is
On 8/12/2009, Timo Sirainen (t...@iki.fi) wrote:
Do you need per-MIME part single instance storage, or would per-email be
enough? Since the per-email can already done with hard links.
Our users are constantly in-line forwarding the same emails with (20+MB)
attachment(s) to different people, but
On 8/12/2009, Timo Sirainen (t...@iki.fi) wrote:
Do you need per-MIME part single instance storage, or would per-email be
enough? Since the per-email can already done with hard links.
The only thing I can find about this on the wiki is where it says single
instance attachment storage (for dbox)
On 8/12/2009, Daniel L. Miller (dmil...@amfes.com) wrote:
dbox-only is fine. I could care less about the storage method chosen
- filesystem, db, encrypted, whatever - but I believe the impact on
storage - and possibly indexes searching - would be huge.
It would be huge for us and anyone else
On Wed, 2009-08-12 at 15:42 -0400, Charles Marcus wrote:
- and I've never heard of any other mail server supporting such a
thing.
Exchange does... and is the single and only reason I have *considered*
switching to it shudder in the past few years...
I heard that the next Exchange version
On Wed, 2009-08-12 at 15:33 -0400, Charles Marcus wrote:
On 8/12/2009, Timo Sirainen (t...@iki.fi) wrote:
Do you need per-MIME part single instance storage, or would per-email be
enough? Since the per-email can already done with hard links.
The only thing I can find about this on the wiki
Timo Sirainen wrote:
On Wed, 2009-08-12 at 15:42 -0400, Charles Marcus wrote:
- and I've never heard of any other mail server supporting such a
thing.
Exchange does... and is the single and only reason I have *considered*
switching to it shudder in the past few years...
I
Original-Nachricht
Datum: Wed, 12 Aug 2009 12:34:40 -0700
Von: Daniel L. Miller dmil...@amfes.com
An: Dovecot Mailing List dovecot@dovecot.org
Betreff: Re: [Dovecot] Scalability plans: Abstract out filesystem and make it
someone else\'s problem
Timo Sirainen wrote
Steve wrote:
dbox-only is fine. I could care less about the storage method chosen -
filesystem, db, encrypted, whatever - but I believe the impact on
storage - and possibly indexes searching - would be huge.
On the personal greedy side, if you want to see a mass corporate
migration to
On Wed, 2009-08-12 at 14:54 -0700, Daniel L. Miller wrote:
If every attachment in a given message is individually scanned to
generate some unique identifier, and that identifier then used to
determine whether or not it exists in the database - this could have
HUGE effects. This now
On Wed, 2009-08-12 at 18:18 -0400, Timo Sirainen wrote:
Oh BTW. I think dbmail 2.3 does that. Then again I haven't yet seen a
stable dbmail version. But looks like they've released 2.3.6 recently
that I haven't tested yet.
Looks like it even does single instance header values:
The header
Timo Sirainen wrote:
On Wed, 2009-08-12 at 18:18 -0400, Timo Sirainen wrote:
Oh BTW. I think dbmail 2.3 does that. Then again I haven't yet seen a
stable dbmail version. But looks like they've released 2.3.6 recently
that I haven't tested yet.
Looks like it even does single instance
Ha! Fooled you! I'm going to reply to the original question instead of
SIS!
Timo Sirainen wrote:
* Index files are really more like memory dumps. They're already in an
optimal format for keeping them in memory, so they can be just mmap()ed
and used. Doing some kind of translation to another
On Aug 12, 2009, at 6:54 PM, Daniel L. Miller wrote:
Do the indexes contain any of the header information?
Yes.
In particular, since I know nothing of the communication between
IMAP clients servers in general, is the information that is shown
in typical client mail lists (subject,
Timo Sirainen wrote:
On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
Nothing forces you to switch from maildir, if you're happy with it :)
But if you want to support millions of users, it's simpler to distribute
the storage and disk I/O evenly across hundreds of servers using a
database
On Aug 11, 2009, at 2:16 AM, Seth Mattinen wrote:
Show me a clustered filesystem that can guarantee that each file is
stored in at least 3 different data centers and can scale linearly by
simply adding more servers (let's say at least up to thousands).
Easy, AFS. It is known to support tens
Timo Sirainen schrieb:
On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
Nothing forces you to switch from maildir, if you're happy with it :)
But if you want to support millions of users, it's simpler to distribute
the storage and disk I/O evenly across hundreds of servers using a
Robert Schetterer wrote:
Timo Sirainen schrieb:
On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
Nothing forces you to switch from maildir, if you're happy with it :)
But if you want to support millions of users, it's simpler to distribute
the storage and disk I/O evenly across hundreds of
I was more thinking about thousands of servers, not clients. Each server should
contribute to the amount of storage you have. Buying huge storages is more
expensive. Also it would be nice if you could just keep plugging in more
servers to get more storage space, disk I/O and CPU and the system
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Mon, 10 Aug 2009, Timo Sirainen wrote:
4. Implement a multi-master filesystem backend for index files. The idea
would be that all servers accessing the same mailbox must be talking to
each others via network and every time something is changed,
Quoting Seth Mattinen se...@rollernet.us:
Queue directories and clusters don't
mix well, but a read-heavy maildir/dbox environment shouldn't suffer the
same problem.
Why don't queue directories and clusters mix well? Is this a performance
issue only, or something worse?
~Seth
--
Eric
On Aug 11, 2009, at 10:32 AM, Steffen Kaiser wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Mon, 10 Aug 2009, Timo Sirainen wrote:
4. Implement a multi-master filesystem backend for index files. The
idea
would be that all servers accessing the same mailbox must be
talking to
On Tue, 2009-08-11 at 09:38 -0700, Seth Mattinen wrote:
Why don't queue directories and clusters mix well? Is this a performance
issue only, or something worse?
It depends on the locking scheme used by the filesystem. Working queue
directories (the ones where stuff comes and goes
Quoting Timo Sirainen t...@iki.fi:
It depends on the locking scheme used by the filesystem. Working queue
directories (the ones where stuff comes and goes rapidly) is best suited
for a local FS anyway.
And when a server and its disk dies, the emails get lost :(
It would appear he is not
Timo Sirainen wrote:
This is something I figured out a few months ago, mainly because this
one guy at work (hi, Stu) kept telling me my multi-master replication
plan sucked and we should use some existing scalable database. (I guess
it didn't go exactly like that, but that's the result anyway.)
On Mon, 2009-08-10 at 14:33 -0700, Seth Mattinen wrote:
Timo Sirainen wrote:
This is something I figured out a few months ago, mainly because this
one guy at work (hi, Stu) kept telling me my multi-master replication
plan sucked and we should use some existing scalable database. (I guess
Timo Sirainen wrote:
On Mon, 2009-08-10 at 14:33 -0700, Seth Mattinen wrote:
Timo Sirainen wrote:
This is something I figured out a few months ago, mainly because this
one guy at work (hi, Stu) kept telling me my multi-master replication
plan sucked and we should use some existing scalable
Seth Mattinen wrote:
Ick, some people (myself included) hate the idea of storing mail in a
database versus simple and almost impossible to screw up plain text
files of maildir. Cyrus already does the whole mail-in-database thing.
Why do you think 'maildir' isn't a database?
Or to you does
Curtis Maloney wrote:
Seth Mattinen wrote:
Ick, some people (myself included) hate the idea of storing mail in a
database versus simple and almost impossible to screw up plain text
files of maildir. Cyrus already does the whole mail-in-database thing.
Why do you think 'maildir' isn't a
On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
Nothing forces you to switch from maildir, if you're happy with it :)
But if you want to support millions of users, it's simpler to
distribute
the storage and disk I/O evenly across hundreds of servers using a
database that was designed for
78 matches
Mail list logo
