Re: RFC: HTTP based storage API

[Nagy, Attila]

On 02/11/16 03:50, David Niklas wrote:

On 08 Feb 2016, at 11:01, Nagy, Attila  wrote:

Nearly every popular programming language has an LMTP/POP/IMAP
implementation, most of them suck in many different ways.
I don't know any server or library which provides a well-established,
compatible protocol frontend with an open backend API, which could be
used to easily make a custom storage backend for the LMTP/POP/IMAP
frontend in any language, in any programming paradigm, irregardless
of the frontend implementation.

Sorry to come in late, but as a curl lover I'd like to know what is wrong
with their implementation of pop3 and imap (project homepage
curl.haxx.se).

Well, most of the time it quickly turns out these are just partial 
addons, with so basic bugs, like #1456 was.
There is a reason why postfix, dovecot and others have so many man-hours 
in them.

BTW, this is about a server implementation...

Re: RFC: HTTP based storage API

[David Niklas]

On 08 Feb 2016, at 11:01, Nagy, Attila  wrote:  
> Nearly every popular programming language has an LMTP/POP/IMAP
> implementation, most of them suck in many different ways.
> I don't know any server or library which provides a well-established,
> compatible protocol frontend with an open backend API, which could be
> used to easily make a custom storage backend for the LMTP/POP/IMAP
> frontend in any language, in any programming paradigm, irregardless
> of the frontend implementation.

Sorry to come in late, but as a curl lover I'd like to know what is wrong
with their implementation of pop3 and imap (project homepage
curl.haxx.se).

Thanks, David

Re: RFC: HTTP based storage API

[lst_hoe02]

Zitat von Timo Sirainen :


On 08 Feb 2016, at 11:01, Nagy, Attila  wrote:


Hi,

Nearly every popular programming language has an LMTP/POP/IMAP  
implementation, most of them suck in many different ways.
I don't know any server or library which provides a  
well-established, compatible protocol frontend with an open backend  
API, which could be used to easily make a custom storage backend  
for the LMTP/POP/IMAP frontend in any language, in any programming  
paradigm, irregardless of the frontend implementation.


Dovecot is very close to this: it speaks IMAP to the backend, but  
you still have to make a nearly complete IMAP implementation, which  
is a PITA.


Therefore I am curious about your opinion: what about an HTTP-based  
open backend (lib-storage) API to Dovecot?


We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's  
close enough to how Dovecot works internally (although we'd still  
need the threading support) and also at least attempting to become a  
standard. We haven't really started it yet though, so now would be a  
good time to give any alternative suggestions or complaints about  
jmap :)


That would be a cool feature, Thanks.

Andreas




smime.p7s
Description: S/MIME Cryptographic Signature

Re: RFC: HTTP based storage API

[Timo Sirainen]

> On 08 Feb 2016, at 13:28, Nagy, Attila  wrote:
> 
> On 02/08/16 12:07, Timo Sirainen wrote:
>> On 08 Feb 2016, at 12:56, Nagy, Attila  wrote:
>>> On 02/08/16 11:16, Timo Sirainen wrote:
 Oh, you were thinking about ability to provide IMAP/etc support for other 
 random servers, and have Dovecot act as kind of a middleware and translate 
 the requests. Maybe the answer is still jmap though? It would require jmap 
 lib-storage backend similar to imapc, which would be doable, although not 
 really something we're right now planning to implement.
 
>>> Yeah, the opposite, in this case a jmap backend to Dovecot.
>>> BTW, I think jmap is too high level and implementing a jmap server is very 
>>> much like implementing an IMAP one.
>>> 
>>> I much more think of a pluggable, easy (remote) storage API, which has much 
>>> less to do with IMAP, but can offer capabilities, which can help Dovecot 
>>> (like the search and indexes).
>> What kind of use cases are you thinking for this remote storage API? What 
>> kind of remote storages would implement it, and what kind of installations 
>> would use them?
>> 
> Interfacing non-email systems to e-mail protocols and implementing other 
> storage options (I can't say a better analogue than what FUSE is).
> Using Dovecot as a protocol implementation, but not for the underlying 
> storage method, which may be custom to the given solution.
> A flexible way of implementing and adapting anything to IMAP.
> 
> For example if I want to store really old and archived e-mails in a costly 
> compression format somewhere in the cloud, but hot mails locally, based on my 
> custom policies.
> In an international company there may be some regulations about what should 
> go where, so for example if I can handle object placement myself, I can use 
> the same installation to store non-EU mails in non-EU countries and others in 
> US or EU clouds if law permits.

You could already implement these with mail-filter plugin. The locally stored 
mail would be just an object pointer, which the mail-filter plugin reads from 
the remote storage. Although I think mail-filter is lacking error handling 
right now.

> Intermixing several systems into one (like merging two IMAP accounts into 
> one) etc.

If I bothered to implement per-namespace imapc-settings, this could be done 
with virtual plugin.

> Some of these are very costly to develop in Dovecot, while just a few lines 
> in -say- Python or go.

I think it might not be that easy. Although I've made lib-storage backends 
easier and easier to implement, it's still not exactly trivial. Implementing it 
behind some API could simplify it somewhat, but it wouldn't really remove all 
the difficult work that needs to be done. For example some backends might want 
to be read-only, others read-write. Some might want Dovecot to store all the 
message flags and other metadata locally, while others might want to store it 
themselves (and that would require two-way syncing between them). Some backends 
would support searching, fetching some headers fast and in general support 
different kinds of optimizations, while others wouldn't. The imapc backend is 
of course already implementing a lot of this functionality, but it's also 
getting complex. The backend would also have to guarantee some things to be 
compatible with IMAP, mainly never modifying existing mails.

One alternative might be to add scripting support to Dovecot. I've been 
thinking about that several times over the years. So instead of implementing 
plugins with C, you could implement them with Python or some other language. 
Long time ago I tried to do this with SWIG, but it didn't work out. The main 
problem was function pointers in structs, but nowadays those aren't directly 
called and I think it could be implemented a bit differently.

RFC: HTTP based storage API

[Nagy, Attila]

Hi,

Nearly every popular programming language has an LMTP/POP/IMAP 
implementation, most of them suck in many different ways.
I don't know any server or library which provides a well-established, 
compatible protocol frontend with an open backend API, which could be 
used to easily make a custom storage backend for the LMTP/POP/IMAP 
frontend in any language, in any programming paradigm, irregardless of 
the frontend implementation.


Dovecot is very close to this: it speaks IMAP to the backend, but you 
still have to make a nearly complete IMAP implementation, which is a PITA.


Therefore I am curious about your opinion: what about an HTTP-based open 
backend (lib-storage) API to Dovecot?


Some thoughts on this:
1. it should be a simple REST API
2. it should support efficient operations on both sides (client, 
server), for example listing a multi-million folder shouldn't be a 
single JSON response, but a JSON stream, which could be produced and 
parsed individually or in smaller batches efficiently
3. it should support capabilities, somewhat like the current imapc 
backend, so if the backend service suppports search, it should announce 
this capability and dovecot could offload these requests to it, but if 
it doesn't, dovecot could build its indices.

4. only the needed amount of POP/IMAP legacy should appear in the protocol
5. the operations should be asynchronous on Dovecot's side (to be scalable)
6. maybe even Dovecot's index-queries could be offloaded to the backend 
service (this way the local file system wouldn't be used at all and all 
caches could be stored in a distributed dictionary), this could also be 
a capability

7. it could support some kind of push (for IDLE-ing and the like)
8. it should be stateless

If Dovecot could make this right, it could be the FUSE of the mail 
protocols, acting as a glue for experimental or production ready mail 
storage backends, without the need to write these backends in "Dovecot 
C" (I mean it's process and API structure).


Of course HTTP here is not a strict requirement. It could be anything, 
which can be easy to implement and understand and is able to fulfill the 
needs.

An example could be Dovecot's own dictionary protocol.

What do you think about this?

Re: RFC: HTTP based storage API

[Marcus Rueckert]

that sounds like s3 API based backend.

also note that dovecot comes with some "non local mail storage plugins"
in the commercial part:

```
In this release, we support Windows Azure, Amazon S3, Scality and
Dropbox systems.
```

so that should come close to what you want. for a local version you
could use e.g. radosdb/ceph or riak. both can provide the s3 API for
you.

hth

darix

-- 
   openSUSE - SUSE Linux is my linux
   openSUSE is good for you
   www.opensuse.org

Re: RFC: HTTP based storage API

[Nagy, Attila]

On 02/08/16 11:16, Timo Sirainen wrote:

On 08 Feb 2016, at 11:59, Timo Sirainen  wrote:

On 08 Feb 2016, at 11:01, Nagy, Attila  wrote:

Hi,

Nearly every popular programming language has an LMTP/POP/IMAP implementation, 
most of them suck in many different ways.
I don't know any server or library which provides a well-established, 
compatible protocol frontend with an open backend API, which could be used to 
easily make a custom storage backend for the LMTP/POP/IMAP frontend in any 
language, in any programming paradigm, irregardless of the frontend 
implementation.

Dovecot is very close to this: it speaks IMAP to the backend, but you still 
have to make a nearly complete IMAP implementation, which is a PITA.

Therefore I am curious about your opinion: what about an HTTP-based open 
backend (lib-storage) API to Dovecot?

We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close 
enough to how Dovecot works internally (although we'd still need the threading 
support) and also at least attempting to become a standard. We haven't really 
started it yet though, so now would be a good time to give any alternative 
suggestions or complaints about jmap :)

Oh, you were thinking about ability to provide IMAP/etc support for other 
random servers, and have Dovecot act as kind of a middleware and translate the 
requests. Maybe the answer is still jmap though? It would require jmap 
lib-storage backend similar to imapc, which would be doable, although not 
really something we're right now planning to implement.


Yeah, the opposite, in this case a jmap backend to Dovecot.
BTW, I think jmap is too high level and implementing a jmap server is 
very much like implementing an IMAP one.


I much more think of a pluggable, easy (remote) storage API, which has 
much less to do with IMAP, but can offer capabilities, which can help 
Dovecot (like the search and indexes).

Re: RFC: HTTP based storage API

[Marcus Rueckert]

On 2016-02-08 12:31:57 +0100, Nagy, Attila wrote:
> On 02/08/16 11:44, Marcus Rueckert wrote:
> >that sounds like s3 API based backend.
> >
> >also note that dovecot comes with some "non local mail storage plugins"
> >in the commercial part:
> >
> >```
> >In this release, we support Windows Azure, Amazon S3, Scality and
> >Dropbox systems.
> >```
> >
> >so that should come close to what you want. for a local version you
> >could use e.g. radosdb/ceph or riak. both can provide the s3 API for
> >you.
> >
> Close, but this may be too low level, for example it may hide whose e-mail
> do you store (I don't know the exact implementation, it's not open source
> AFAIK).

True ... but that might be the hook point you can use for your more
fancy implementation. and some of the mail backends are public. like
mdbox and so on. the api towards dovecot should be the same.

darix

-- 
   openSUSE - SUSE Linux is my linux
   openSUSE is good for you
   www.opensuse.org

Re: RFC: HTTP based storage API

[Nagy, Attila]

On 02/08/16 12:07, Timo Sirainen wrote:

On 08 Feb 2016, at 12:56, Nagy, Attila  wrote:

On 02/08/16 11:16, Timo Sirainen wrote:

Oh, you were thinking about ability to provide IMAP/etc support for other 
random servers, and have Dovecot act as kind of a middleware and translate the 
requests. Maybe the answer is still jmap though? It would require jmap 
lib-storage backend similar to imapc, which would be doable, although not 
really something we're right now planning to implement.


Yeah, the opposite, in this case a jmap backend to Dovecot.
BTW, I think jmap is too high level and implementing a jmap server is very much 
like implementing an IMAP one.

I much more think of a pluggable, easy (remote) storage API, which has much 
less to do with IMAP, but can offer capabilities, which can help Dovecot (like 
the search and indexes).

What kind of use cases are you thinking for this remote storage API? What kind 
of remote storages would implement it, and what kind of installations would use 
them?

Interfacing non-email systems to e-mail protocols and implementing other 
storage options (I can't say a better analogue than what FUSE is).
Using Dovecot as a protocol implementation, but not for the underlying 
storage method, which may be custom to the given solution.

A flexible way of implementing and adapting anything to IMAP.

For example if I want to store really old and archived e-mails in a 
costly compression format somewhere in the cloud, but hot mails locally, 
based on my custom policies.
In an international company there may be some regulations about what 
should go where, so for example if I can handle object placement myself, 
I can use the same installation to store non-EU mails in non-EU 
countries and others in US or EU clouds if law permits.


Intermixing several systems into one (like merging two IMAP accounts 
into one) etc.


Some of these are very costly to develop in Dovecot, while just a few 
lines in -say- Python or go.

Re: RFC: HTTP based storage API

[Nagy, Attila]

On 02/08/16 11:44, Marcus Rueckert wrote:

that sounds like s3 API based backend.

also note that dovecot comes with some "non local mail storage plugins"
in the commercial part:

```
In this release, we support Windows Azure, Amazon S3, Scality and
Dropbox systems.
```

so that should come close to what you want. for a local version you
could use e.g. radosdb/ceph or riak. both can provide the s3 API for
you.

Close, but this may be too low level, for example it may hide whose 
e-mail do you store (I don't know the exact implementation, it's not 
open source AFAIK).

Re: RFC: HTTP based storage API

[Timo Sirainen]

On 08 Feb 2016, at 11:01, Nagy, Attila  wrote:
> 
> Hi,
> 
> Nearly every popular programming language has an LMTP/POP/IMAP 
> implementation, most of them suck in many different ways.
> I don't know any server or library which provides a well-established, 
> compatible protocol frontend with an open backend API, which could be used to 
> easily make a custom storage backend for the LMTP/POP/IMAP frontend in any 
> language, in any programming paradigm, irregardless of the frontend 
> implementation.
> 
> Dovecot is very close to this: it speaks IMAP to the backend, but you still 
> have to make a nearly complete IMAP implementation, which is a PITA.
> 
> Therefore I am curious about your opinion: what about an HTTP-based open 
> backend (lib-storage) API to Dovecot?

We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close 
enough to how Dovecot works internally (although we'd still need the threading 
support) and also at least attempting to become a standard. We haven't really 
started it yet though, so now would be a good time to give any alternative 
suggestions or complaints about jmap :)

Re: RFC: HTTP based storage API

[Timo Sirainen]

> On 08 Feb 2016, at 11:59, Timo Sirainen  wrote:
> 
> On 08 Feb 2016, at 11:01, Nagy, Attila  wrote:
>> 
>> Hi,
>> 
>> Nearly every popular programming language has an LMTP/POP/IMAP 
>> implementation, most of them suck in many different ways.
>> I don't know any server or library which provides a well-established, 
>> compatible protocol frontend with an open backend API, which could be used 
>> to easily make a custom storage backend for the LMTP/POP/IMAP frontend in 
>> any language, in any programming paradigm, irregardless of the frontend 
>> implementation.
>> 
>> Dovecot is very close to this: it speaks IMAP to the backend, but you still 
>> have to make a nearly complete IMAP implementation, which is a PITA.
>> 
>> Therefore I am curious about your opinion: what about an HTTP-based open 
>> backend (lib-storage) API to Dovecot?
> 
> We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close 
> enough to how Dovecot works internally (although we'd still need the 
> threading support) and also at least attempting to become a standard. We 
> haven't really started it yet though, so now would be a good time to give any 
> alternative suggestions or complaints about jmap :)

Oh, you were thinking about ability to provide IMAP/etc support for other 
random servers, and have Dovecot act as kind of a middleware and translate the 
requests. Maybe the answer is still jmap though? It would require jmap 
lib-storage backend similar to imapc, which would be doable, although not 
really something we're right now planning to implement.

Re: RFC: HTTP based storage API

[Timo Sirainen]

On 08 Feb 2016, at 12:56, Nagy, Attila  wrote:
> 
> On 02/08/16 11:16, Timo Sirainen wrote:
>> Oh, you were thinking about ability to provide IMAP/etc support for other 
>> random servers, and have Dovecot act as kind of a middleware and translate 
>> the requests. Maybe the answer is still jmap though? It would require jmap 
>> lib-storage backend similar to imapc, which would be doable, although not 
>> really something we're right now planning to implement.
>> 
> Yeah, the opposite, in this case a jmap backend to Dovecot.
> BTW, I think jmap is too high level and implementing a jmap server is very 
> much like implementing an IMAP one.
> 
> I much more think of a pluggable, easy (remote) storage API, which has much 
> less to do with IMAP, but can offer capabilities, which can help Dovecot 
> (like the search and indexes).

What kind of use cases are you thinking for this remote storage API? What kind 
of remote storages would implement it, and what kind of installations would use 
them?