>b) Use threads, either via some library or implement yourself. Each thread of
>course uses some extra memory. Also enabling >threads causes glibc to start
>using a thread-safe version of malloc() (I think?), which slows things down
>(unless that can be avoided, >maybe by using clone() directly
On 10/03/2010 21:19, Timo Sirainen wrote:
On 10.8.2009, at 20.01, Timo Sirainen wrote:
(3.5. Implement async I/O filesystem backend.)
You know what I found out today? Linux doesn't support async IO for regular
buffered files. I had heard there were issues, but I thought it was mainl
Timo Sirainen put forth on 3/10/2010 3:19 PM:
> On 10.8.2009, at 20.01, Timo Sirainen wrote:
>
>> (3.5. Implement async I/O filesystem backend.)
>
> You know what I found out today? Linux doesn't support async IO for regular
> buffered files. I had heard there were issues, but I thought it was m
On 10.8.2009, at 20.01, Timo Sirainen wrote:
> (3.5. Implement async I/O filesystem backend.)
You know what I found out today? Linux doesn't support async IO for regular
buffered files. I had heard there were issues, but I thought it was mainly
about some annoying APIs and such. Anyone know if
On Mon, 2009-09-28 at 22:20 +0100, Ed W wrote:
> Timo Sirainen wrote:
> >
> > The SQL code was for v1.0 and the lib-storage API has been simplified
> > since then, maybe not hugely but still pretty much. Maybe some day I'll
> > see about updating the SQL code for v2.0 API.
> >
> > Oh and some docum
Timo Sirainen wrote:
The SQL code was for v1.0 and the lib-storage API has been simplified
since then, maybe not hugely but still pretty much. Maybe some day I'll
see about updating the SQL code for v2.0 API.
Oh and some documentation about it would probably help a lot too. I
guess I should wri
On 9/28/2009 1:44 PM, Charles Marcus wrote:
On 9/28/2009 4:24 PM, Jeff Grossman wrote:
In fact there are lots of IMAP improvements in v3... I can't wait until
all my extensions catch up, and I figure out how to customize the UI the
way I want (e.g., how in the world do I get rid of the stupi
On 9/28/2009, Charles Marcus (cmar...@media-brokers.com) wrote:
> > You can't get rid of tabs per se, but you can make it so you don't use
> > > them. I hate tabs personally also. Go to Options, Advanced, Reading
> > > and Display, and select Open Messages In: An Existing Window or A New
> > > Wi
On 9/28/2009 4:24 PM, Jeff Grossman wrote:
>> In fact there are lots of IMAP improvements in v3... I can't wait until
>> all my extensions catch up, and I figure out how to customize the UI the
>> way I want (e.g., how in the world do I get rid of the stupid Tabs??)
> You can't get rid of tabs per
On 9/28/2009 11:43 AM, Charles Marcus wrote:
On 9/28/2009, Ed W (li...@wildgooses.com) wrote:
In theory Thunderbird does this, but at least on my machine it just
repeatedly downloads the same message again and again in various ways
- it grinds to a halt every time I click on an email with a
On Mon, 2009-09-28 at 20:11 +0100, Ed W wrote:
> > lib-storage API has existed since Dovecot v1.0 and it's used to abstract
> > out access to maildir, mbox, dbox, cydir, etc. SQL would fit right there
> > with those.
> >
>
> OK, I thought that was what you were going to be simplifying...
Nope.
Timo Sirainen wrote:
On Mon, 2009-09-28 at 19:21 +0100, Ed W wrote:
In my mind this is more about what lib-storage API was supposed to
abstract out, whereas my filesystem API would be used simply for binary
data storage. The same FS API could be used to store both dbox files and
index files.
On Mon, 2009-09-28 at 19:21 +0100, Ed W wrote:
> > In my mind this is more about what lib-storage API was supposed to
> > abstract out, whereas my filesystem API would be used simply for binary
> > data storage. The same FS API could be used to store both dbox files and
> > index files.
> >
>
>
On 9/28/2009, Ed W (li...@wildgooses.com) wrote:
> In theory Thunderbird does this, but at least on my machine it just
> repeatedly downloads the same message again and again in various ways
> - it grinds to a halt every time I click on an email with a decent
> sized attachment, even if I have alre
Timo Sirainen wrote:
On Mon, 2009-09-28 at 18:35 +0100, Ed W wrote:
I would have thought that your API will prefer to request message parts
where it can (eg header, body, mime part), and just issue a read_bytes,
where that's what the client is asking for otherwise. This would allow
the sto
On Mon, 2009-09-28 at 18:35 +0100, Ed W wrote:
> I would have thought that your API will prefer to request message parts
> where it can (eg header, body, mime part), and just issue a read_bytes,
> where that's what the client is asking for otherwise. This would allow
> the storage engine to opt
Timo Sirainen wrote:
On Mon, 2009-09-28 at 17:57 +0100, Ed W wrote:
My only request to Timo was to kind of consider that a bunch of these
ideas from the audience will almost certainly involve splitting up the
mime message into component parts and that the abstracted interface
should try not
Timo Sirainen wrote:
On Mon, 2009-09-28 at 09:00 -0700, paulmon wrote:
My current thinking is a key/value store as you've proposed. Something like
Hadoop components or Project Voldamort. Voldamort might be a better fit
from what I've read.
My understanding of Hadoop is that it's mor
On Mon, 2009-09-28 at 17:57 +0100, Ed W wrote:
> My only request to Timo was to kind of consider that a bunch of these
> ideas from the audience will almost certainly involve splitting up the
> mime message into component parts and that the abstracted interface
> should try not to throw away any
On Mon, 2009-09-28 at 09:00 -0700, paulmon wrote:
> My current thinking is a key/value store as you've proposed. Something like
> Hadoop components or Project Voldamort. Voldamort might be a better fit
> from what I've read.
My understanding of Hadoop is that it's more about distributed computi
paulmon wrote:
My current thinking if having the local delivery break messages up into
their component pieces, headers, from address, to address, spam scores, body
etc into various key:value relationships.
Whilst this looks appealing on the surface I think the details are going
to need some b
On Mon, 2009-08-10 at 14:33 -0700, Seth Mattinen wrote:
> Nothing forces you to switch from maildir, if you're happy with it :)
> But if you want to support millions of users, it's simpler to distribute
> the storage and disk I/O evenly across hundreds of servers using a
> database that was de
On Wed, 2009-08-12 at 18:18 -0400, Timo Sirainen wrote:
> On Wed, 2009-08-12 at 14:54 -0700, Daniel L. Miller wrote:
> > If every attachment in a given message is individually scanned to
> > generate some unique identifier, and that identifier then used to
> > determine whether or not it exists i
> But for example, what I'd really like to be able to do is say something
> like:
>
> SiS_mode = binary,64K
>
> So only binary attachments over 64KB in size would be checksummed and
> single instance stored. I don't care bout the headers, or body text, or
> tiny (< 64KB) sig attachments, or text
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Wed, 12 Aug 2009, Timo Sirainen wrote:
So yeah, either one are as reliable as the script :)
Well, I think this is not what the OP intended :)
This will work for only a little amount of mails I see, because:
a) forwarded messages differ,
b) re
On Aug 13, 2009, at 8:13 AM, Charles Marcus wrote:
I'm also having trouble figuring out how using hard links (or their
equivalent) for messages with large attachments and having only one
instance of the attachment could cause *more* disk I/O than having
dozens/hundred of multiple copies of the m
On 8/12/2009, Daniel L. Miller (dmil...@amfes.com) wrote:
> Under the structure I've proposed, net storage consumed by the
> attachments should be one copy of attachment 1, and one copy of
> attachment two, plus headers and any comments in the messages times
> the number of recipients. Domino woul
On 8/12/2009, Ed W (li...@wildgooses.com) wrote:
> It also had a bunch of limitations, it was basically only single
> instance for CC recipients on a message (more or less). Quite a lot
> of things such as certain types of virus scanning would (I think)
> easily disable the single instance storage
On 8/12/2009 4:02 PM, Timo Sirainen wrote:
>>> Do you need per-MIME part single instance storage, or would per-email be
>>> enough? Since the per-email can already done with hard links.
>> The only thing I can find about this on the wiki is where it says single
>> instance attachment storage (for
On 8/12/2009 3:58 PM, Timo Sirainen wrote:
>>> - and I've never heard of any other mail server supporting such a
>>> thing.
>> Exchange does... and is the single and only reason I have *considered*
>> switching to it in the past few years...
> I heard that the next Exchange version drops that fe
On Aug 12, 2009, at 6:54 PM, Daniel L. Miller wrote:
Do the indexes contain any of the header information?
Yes.
In particular, since I know nothing of the communication between
IMAP clients & servers in general, is the information that is shown
in typical client mail lists (subject, send
Ha! Fooled you! I'm going to reply to the original question instead of
SIS!
Timo Sirainen wrote:
* Index files are really more like memory dumps. They're already in an
optimal format for keeping them in memory, so they can be just mmap()ed
and used. Doing some kind of translation to another
Timo Sirainen wrote:
On Wed, 2009-08-12 at 18:18 -0400, Timo Sirainen wrote:
Oh BTW. I think dbmail 2.3 does that. Then again I haven't yet seen a
stable dbmail version. But looks like they've released 2.3.6 recently
that I haven't tested yet.
Looks like it even does single instance he
On Wed, 2009-08-12 at 18:18 -0400, Timo Sirainen wrote:
> Oh BTW. I think dbmail 2.3 does that. Then again I haven't yet seen a
> stable dbmail version. But looks like they've released 2.3.6 recently
> that I haven't tested yet.
Looks like it even does single instance header values:
> The header
On Wed, 2009-08-12 at 14:54 -0700, Daniel L. Miller wrote:
> If every attachment in a given message is individually scanned to
> generate some unique identifier, and that identifier then used to
> determine whether or not it exists in the database - this could have
> HUGE effects. This now addr
Steve wrote:
dbox-only is fine. I could care less about the storage method chosen -
filesystem, db, encrypted, whatever - but I believe the impact on
storage - and possibly indexes & searching - would be huge.
On the personal greedy side, if you want to see a mass corporate
migration to Do
Original-Nachricht
> Datum: Wed, 12 Aug 2009 12:34:40 -0700
> Von: "Daniel L. Miller"
> An: Dovecot Mailing List
> Betreff: Re: [Dovecot] Scalability plans: Abstract out filesystem and make it
> someone else\'s problem
> Timo Sirainen wrote:
> > On Wed, 2009-08-12 at 11:35 -0
Timo Sirainen wrote:
On Wed, 2009-08-12 at 15:42 -0400, Charles Marcus wrote:
- and I've never heard of any other mail server supporting such a
thing.
Exchange does... and is the single and only reason I have *considered*
switching to it in the past few years...
I heard that t
On Wed, 2009-08-12 at 15:33 -0400, Charles Marcus wrote:
> On 8/12/2009, Timo Sirainen (t...@iki.fi) wrote:
> > Do you need per-MIME part single instance storage, or would per-email be
> > enough? Since the per-email can already done with hard links.
>
> The only thing I can find about this on the
On Wed, 2009-08-12 at 15:42 -0400, Charles Marcus wrote:
> > - and I've never heard of any other mail server supporting such a
> > thing.
>
> Exchange does... and is the single and only reason I have *considered*
> switching to it in the past few years...
I heard that the next Exchange version d
On 8/12/2009, Daniel L. Miller (dmil...@amfes.com) wrote:
> dbox-only is fine. I could care less about the storage method chosen
> - filesystem, db, encrypted, whatever - but I believe the impact on
> storage - and possibly indexes & searching - would be huge.
It would be huge for us and anyone e
Timo Sirainen wrote:
On Wed, 2009-08-12 at 11:35 -0700, Daniel L. Miller wrote:
Timo Sirainen wrote:
Also the mime structure could be torn apart to store
attachments individually - the motivation being single instance storage
of large attachments with identical content... Anyway, thes
On 8/12/2009, Timo Sirainen (t...@iki.fi) wrote:
> Do you need per-MIME part single instance storage, or would per-email be
> enough? Since the per-email can already done with hard links.
The only thing I can find about this on the wiki is where it says single
instance attachment storage (for dbox
On 8/12/2009, Timo Sirainen (t...@iki.fi) wrote:
> Do you need per-MIME part single instance storage, or would per-email be
> enough? Since the per-email can already done with hard links.
Our users are constantly in-line forwarding the same emails with (20+MB)
attachment(s) to different people, bu
Daniel L. Miller wrote:
Timo Sirainen wrote:
Also the mime structure could be torn apart to store attachments
individually - the motivation being single instance storage of large
attachments with identical content... Anyway, these seem like very
speculative directions...
Yes, this is a
On Wed, 2009-08-12 at 11:35 -0700, Daniel L. Miller wrote:
> Timo Sirainen wrote:
> >> Also the mime structure could be torn apart to store
> >> attachments individually - the motivation being single instance storage
> >> of large attachments with identical content... Anyway, these seem like
>
On Wed, 2009-08-12 at 19:19 +0100, Ed W wrote:
> I actually thought your idea of having a bunch of cut down IMAP type
> servers as the backend storage talking to a bunch of beefier frontend
> servers was quite an interesting idea!
>
> Certainly though a simplification of the on-disk API would en
Timo Sirainen wrote:
Also the mime structure could be torn apart to store
attachments individually - the motivation being single instance storage
of large attachments with identical content... Anyway, these seem like
very speculative directions...
Yes, this is also something in dbox's f
Oh, have you considered some "optional" api calls in the storage API?
The logic might be to assume that someone wanted to do something
clever and split the message up in some way, eg store headers
separately to bodies or bodies carved up into mime parts. The
motivation would be if there was a c
Oh, have you considered some "optional" api calls in the storage API?
The logic might be to assume that someone wanted to do something
clever and split the message up in some way, eg store headers
separately to bodies or bodies carved up into mime parts. The
motivation would be if there was a c
Ed W wrote:
>
> Actually I use maildir, but apart from delete performance which is
> usually rare, mailbox seems better for nearly all use patterns
>
> Seems like if it were possible to "solve" delete performance then
> mailbox becomes the preferred choice for many requirements (also lets
> solve
On Wed, 2009-08-12 at 18:42 +0100, Ed W wrote:
> > Something like that. In dbox you have one storage directory containing
> > all mailboxes' mails (so that copying can be done by simple index
> > updates). Then you have a bunch of files, each about n MB (configurable,
> > 2 MB by default). Expungin
CouchDB seems like it would still be more difficult than necessary to
scale. I'd really just want something that distributes the load and
disk usage evenly across all servers and allows easily plugging in
more servers and it automatically rebalances the load. CouchDB seems
like much of that w
On Wed, 2009-08-12 at 17:46 +0100, Ed W wrote:
> My expectation then is that with local index and sql message storage
> that the performance should be very reasonable for a large class of
> users... (ok, other problems perhaps arise)
If messages are stored to SQL in dummy blobs then the performa
1) Since latency requirements are low, why did performance drop so
much previously when you implemented a very simple mysql storage
backend? I glanced at the code a few weeks ago and whilst it's
surprisingly complicated right now to implement a backend, I was also
surprised that a database
On Aug 12, 2009, at 2:21 AM, Steffen Kaiser > wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Tue, 11 Aug 2009, Eric Jon Rostetter wrote:
For a massively scaled system, there may be sufficient performance to
put the queues elsewhere.
Which also allows that the queue can easily have
On Aug 12, 2009, at 11:26 AM, Ed W wrote:
Hi
* Mail data on the other hand is just written once and usually read
maybe once or a couple of times. Caching mail data in memory probably
doesn't help all that much. Latency isn't such a horrible issue as
long
as multiple mails can be fetched at o
Hi
* Mail data on the other hand is just written once and usually read
maybe once or a couple of times. Caching mail data in memory probably
doesn't help all that much. Latency isn't such a horrible issue as long
as multiple mails can be fetched at once / in parallel, so there's only
a single l
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Tue, 11 Aug 2009, Eric Jon Rostetter wrote:
For a massively scaled system, there may be sufficient performance to
put the queues elsewhere.
Which also allows that the queue can easily have multiple machines pushing
& poping items.
But on a
Quoting Timo Sirainen :
It depends on the locking scheme used by the filesystem. Working queue
directories (the ones where stuff comes and goes rapidly) is best suited
for a local FS anyway.
And when a server and its disk dies, the emails get lost :(
It would appear he is not talking about a
On Tue, 2009-08-11 at 09:38 -0700, Seth Mattinen wrote:
> > Why don't queue directories and clusters mix well? Is this a performance
> > issue only, or something worse?
> >
>
> It depends on the locking scheme used by the filesystem. Working queue
> directories (the ones where stuff comes and go
Eric Jon Rostetter wrote:
> Quoting Seth Mattinen :
>
>> Queue directories and clusters don't
>> mix well, but a read-heavy maildir/dbox environment shouldn't suffer the
>> same problem.
>
> Why don't queue directories and clusters mix well? Is this a performance
> issue only, or something worse
On Aug 11, 2009, at 10:32 AM, Steffen Kaiser wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Mon, 10 Aug 2009, Timo Sirainen wrote:
4. Implement a multi-master filesystem backend for index files. The
idea
would be that all servers accessing the same mailbox must be
talking to
each
Quoting Seth Mattinen :
Queue directories and clusters don't
mix well, but a read-heavy maildir/dbox environment shouldn't suffer the
same problem.
Why don't queue directories and clusters mix well? Is this a performance
issue only, or something worse?
~Seth
--
Eric Rostetter
The Departme
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Mon, 10 Aug 2009, Timo Sirainen wrote:
4. Implement a multi-master filesystem backend for index files. The idea
would be that all servers accessing the same mailbox must be talking to
each others via network and every time something is changed, p
I was more thinking about thousands of servers, not clients. Each server should
contribute to the amount of storage you have. Buying huge storages is more
expensive. Also it would be nice if you could just keep plugging in more
servers to get more storage space, disk I/O and CPU and the system w
Timo Sirainen wrote:
> On Aug 11, 2009, at 2:16 AM, Seth Mattinen wrote:
>
>>> Show me a clustered filesystem that can guarantee that each file is
>>> stored in at least 3 different data centers and can scale linearly by
>>> simply adding more servers (let's say at least up to thousands).
>>
>> Ea
Robert Schetterer wrote:
> Timo Sirainen schrieb:
>> On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
>>
Nothing forces you to switch from maildir, if you're happy with it :)
But if you want to support millions of users, it's simpler to distribute
the storage and disk I/O evenly ac
Timo Sirainen schrieb:
> On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
>
>>> Nothing forces you to switch from maildir, if you're happy with it :)
>>> But if you want to support millions of users, it's simpler to distribute
>>> the storage and disk I/O evenly across hundreds of servers using
On Aug 11, 2009, at 2:16 AM, Seth Mattinen wrote:
Show me a clustered filesystem that can guarantee that each file is
stored in at least 3 different data centers and can scale linearly by
simply adding more servers (let's say at least up to thousands).
Easy, AFS. It is known to support tens of
Timo Sirainen wrote:
> On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
>
>>> Nothing forces you to switch from maildir, if you're happy with it :)
>>> But if you want to support millions of users, it's simpler to distribute
>>> the storage and disk I/O evenly across hundreds of servers using a
On Aug 11, 2009, at 12:41 AM, Seth Mattinen wrote:
Nothing forces you to switch from maildir, if you're happy with it :)
But if you want to support millions of users, it's simpler to
distribute
the storage and disk I/O evenly across hundreds of servers using a
database that was designed for i
Curtis Maloney wrote:
> Seth Mattinen wrote:
>> Ick, some people (myself included) hate the idea of storing mail in a
>> database versus simple and almost impossible to screw up plain text
>> files of maildir. Cyrus already does the whole mail-in-database thing.
>
> Why do you think 'maildir' isn'
Seth Mattinen wrote:
Ick, some people (myself included) hate the idea of storing mail in a
database versus simple and almost impossible to screw up plain text
files of maildir. Cyrus already does the whole mail-in-database thing.
Why do you think 'maildir' isn't a database?
Or to you does 'da
Timo Sirainen wrote:
> On Mon, 2009-08-10 at 14:33 -0700, Seth Mattinen wrote:
>> Timo Sirainen wrote:
>>> This is something I figured out a few months ago, mainly because this
>>> one guy at work (hi, Stu) kept telling me my multi-master replication
>>> plan sucked and we should use some existing
On Mon, 2009-08-10 at 14:33 -0700, Seth Mattinen wrote:
> Timo Sirainen wrote:
> > This is something I figured out a few months ago, mainly because this
> > one guy at work (hi, Stu) kept telling me my multi-master replication
> > plan sucked and we should use some existing scalable database. (I gu
Timo Sirainen wrote:
This is something I figured out a few months ago, mainly because this
one guy at work (hi, Stu) kept telling me my multi-master replication
plan sucked and we should use some existing scalable database. (I guess
it didn't go exactly like that, but that's the result anyway.)
This is something I figured out a few months ago, mainly because this
one guy at work (hi, Stu) kept telling me my multi-master replication
plan sucked and we should use some existing scalable database. (I guess
it didn't go exactly like that, but that's the result anyway.)
So, my current plan is
78 matches
Mail list logo
