Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-06 Thread Barry Warsaw
On Jun 05, 2012, at 08:09 AM, Pierre-Yves Chibon wrote: With the help of wacky on #mailman this week-end we came up with this patch. Attached is both the diff and the file itself. This way we can start discussing :) Point of order: it's much easier to deal with branches and merge proposals

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-06 Thread Barry Warsaw
On Jun 03, 2012, at 11:21 PM, Patrick Ben Koetter wrote: Maybe - and to pick up an idea Barry had mentioned to me a long time ago about mailing list management directly from a mail client - we would gain the most if we implemented an LMTP client as archiver (better: archive transport). This

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-06 Thread Barry Warsaw
On Jun 03, 2012, at 02:49 PM, Pierre-Yves Chibon wrote: I can work on merging KittyStore and IMessageStore and providing the REST interface for it. Should I provide also an implementation ? Using PostgreSQL as backend ? It will be much easier to test with a SQLite backend, although as I'm

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-05 Thread Pierre-Yves Chibon
On Fri, 2012-06-01 at 21:33 -0400, Barry Warsaw wrote: * Look at the IMessageStore API. Is this complete? IOW, could you build a purely Python-level archiver like HyperKitty on top of this API? Here's where proper attachment handling would probably be necessary. With the help of wacky on

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-03 Thread Pierre-Yves Chibon
On Fri, 2012-06-01 at 21:33 -0400, Barry Warsaw wrote: * Look at the IMessageStore API. Is this complete? IOW, could you build a purely Python-level archiver like HyperKitty on top of this API? Here's where proper attachment handling would probably be necessary. I have defined a number of

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-03 Thread Pierre-Yves Chibon
On Sun, 2012-06-03 at 14:49 +0200, Pierre-Yves Chibon wrote: Main difference compare to what is in the IMessageStore at the moment is that I require the fq list name as first argument of each function (ie: I have one table for each list and need to know where you would like me to search)

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-03 Thread Barry Warsaw
On Jun 02, 2012, at 07:19 AM, Patrick Ben Koetter wrote: Maildir is robust but it doesn't scale under high load. You can add indexes, but they are limited sooner or later too. Concerning mailbox formats Timo Sirainens current approach to collect a limited number of messages in one file and then

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-03 Thread Patrick Ben Koetter
* Barry Warsaw ba...@list.org: On Jun 02, 2012, at 07:19 AM, Patrick Ben Koetter wrote: Maildir is robust but it doesn't scale under high load. You can add indexes, but they are limited sooner or later too. Concerning mailbox formats Timo Sirainens current approach to collect a limited

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-01 Thread Pierre-Yves Chibon
On Thu, 2012-05-31 at 10:28 -0700, Toshio Kuratomi wrote: The idea is that now, we can have different backend and each module needing access to the emails can use the API directly without having to bother about which storage system is behind. Wacky looked at this today and asked if we

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-01 Thread Barry Warsaw
Jeebus, finally catching up on this thread. On Apr 24, 2012, at 01:41 PM, Stephen J. Turnbull wrote: IMHO, premature optimization. Among other things, there isn't going to be a the archiver-core. Mailman should provide a archiver-core, and I think it should be based on maildir (which is

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-01 Thread Barry Warsaw
On Apr 24, 2012, at 11:12 AM, Toshio Kuratomi wrote: Ive been thinking about this and I'm in mild disagreement. I think that a mailing list system should give people an archive-store which is acessible behind a generalized API. I'm warming up to this. The IArchiver interface is generic enough

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-01 Thread Barry Warsaw
On Apr 26, 2012, at 06:36 PM, Pierre-Yves Chibon wrote: The current version of HK relies on mongodb for the storage, but I want to test HK with a traditionnal SQL backend. So I have started to work on this. The interface I defined is there:

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-01 Thread Barry Warsaw
On May 31, 2012, at 10:28 AM, Toshio Kuratomi wrote: Wacky looked at this today and asked if we should have the x-message-id-hash as another key value to look up an email. That seems proper to me. +1 Would we want a separate function or to overload get_email() so that it can either take

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-06-01 Thread Patrick Ben Koetter
* Barry Warsaw ba...@list.org: On Apr 24, 2012, at 01:41 PM, Stephen J. Turnbull wrote: IMHO, premature optimization. Among other things, there isn't going to be a the archiver-core. Mailman should provide a archiver-core, and I think it should be based on maildir (which is apparently

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-05-31 Thread Toshio Kuratomi
On Thu, Apr 26, 2012 at 06:36:02PM +0200, Pierre-Yves Chibon wrote: The current version of HK relies on mongodb for the storage, but I want to test HK with a traditionnal SQL backend. So I have started to work on this. The interface I defined is there:

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-05-31 Thread Terri Oda
On 05/31/2012 11:28 AM, Toshio Kuratomi wrote: Wacky looked at this today and asked if we should have the x-message-id-hash as another key value to look up an email. That seems proper to me. Would we want a separate function or to overload get_email() so that it can either take message_id or

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-04-26 Thread Pierre-Yves Chibon
On Tue, 2012-04-24 at 11:12 -0700, Toshio Kuratomi wrote: [mailman3 core] -- maintainance of list metadata, sending and receiving, provides a REST API [Web UIs] -- web ui to Core functions [Archive-stores] -- stores the messages sent to the mailing lists.

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-04-26 Thread Pierre-Yves Chibon
Resent as the first one doesn't seem to want to arrive. On Tue, 2012-04-24 at 11:12 -0700, Toshio Kuratomi wrote: [mailman3 core] -- maintainance of list metadata, sending and receiving, provides a REST API [Web UIs] -- web ui to Core functions [Archive-stores] --

Re: [Mailman-Developers] Speaking about kitties (or archivers)

2012-04-24 Thread Toshio Kuratomi
On Mon, Apr 23, 2012 at 06:20:18PM -0400, Barry Warsaw wrote: Thanks for posting this Pierre-Yves! On Apr 23, 2012, at 08:17 PM, Pierre-Yves Chibon wrote: archive-core (store the emails and expose them through an API) -- archivers/stats/NNTP The questions are then: - how do we store

[Mailman-Developers] Speaking about kitties (or archivers)

2012-04-23 Thread Pierre-Yves Chibon
Meeow miaou* We spoke on IRC about the archiver the other day and I said that I should present here my thoughts about it. So here they are (beware that might be long). First I think we should think about the structure/architecture of things. We have a number of component which need to be