Re: Advanced mail management

[Piers Cawley]

"Robert Shiels" <[EMAIL PROTECTED]> writes:
> From: "Leon Brocard" <[EMAIL PROTECTED]>
> 
> > Paul Makepeace sent the following bits through the ether:
> >
> > > The problem is when you have folders with tens of thousands of
> > > emails (London.pm is about to hit 20k here)
> >
> > I've moved my mail from laptop to laptop to new company computer over
> > the years and it's dawned on me that I never (that is NEVER) need
> > archives which are more than a week or so old.
> >
> > Agressive deletion is the only way.
> >
> 
> I disagree completely. I used to spend quite a lot of time deciding whether
> to keep mailing list emails or not, well, fractions of seconds per email,
> but on high volume lists this adds up :-) So I stopped deciding, and now
> keep everything, which is much faster. I have always kept every personal
> email, and HD size has grown much faster than the rate I receive/read email,
> so I have had no problems with storage. I am a natural hoarder though, and
> quite enjoy browsing old email folders to see what I was up to. I also have
> copies of course of every email I have ever sent.
> 
> I use outlook express, which has some searching facilities built in, which
> are good enough for me.
> 
> As to actually NEEDING any of this, I expect if i lost the complete email
> archive and all backups I'd survive, but I'd definitely be a bit upset.

I've done this, 5 years of email up in smoke. Note to self: Acquire a
real backup solution. Now. Really now. Soon. Well, when I get round to
it.

Hmm... anyone know of a tape drive with a firewire interface? For
reasonable money? Then I can get a firewire PC Card for this here
lapdog...

Anyway, this talk has finally led me to ditch procmail and I'm doing
all my mail splitting within gnus using nnmail-split-fancy or whatever
it's called, and it's very nice too. The splitting rules get built
automagically from group properties, which is nice.

For my next trick I'm going to write/find a thread based expiry
function, which will move a thread into an archive directory (probably
as a gzipped mbox) when its newest message is n days old, which should
keep my live message base reasonably small, but will provide me with
good historic stuff too.

And if I can get it to do sensible indexing at the same time. Hmm...
this could require some serious hacking...

-- 
Piers

Re: Advanced mail management

[Redvers Davies]

> message-id's -- anyone seen/done this?). Anyone played with/heard
> about putting mail itself in RDBMS? That way you could create virtual
> folders etc.

Pronto does this.  You can optionally put all emails and headers into
the database.  Mails can appear in multiple folders.

When you perform a search (freetext - cor) in Pronto it creates
a virtual folder which contains all your matching mails.

You have full perl regex to file/filter/trash mails.

You have full perl regex to score mails and deal with them as you wish.

> While I'm on this subject, how do people deal with searching email?
> Archival management? (And how that impacts searching?) I will write up a
> Web page on all this seeded on whatever comes out of this, if there
> isn't one already.

See above.  Archiving mail is as simple as mysqldump. 

Red

Re: Advanced mail management

[Robert Shiels]

From: "Leon Brocard" <[EMAIL PROTECTED]>

> Paul Makepeace sent the following bits through the ether:
>
> > The problem is when you have folders with tens of thousands of
> > emails (London.pm is about to hit 20k here)
>
> I've moved my mail from laptop to laptop to new company computer over
> the years and it's dawned on me that I never (that is NEVER) need
> archives which are more than a week or so old.
>
> Agressive deletion is the only way.
>

I disagree completely. I used to spend quite a lot of time deciding whether
to keep mailing list emails or not, well, fractions of seconds per email,
but on high volume lists this adds up :-) So I stopped deciding, and now
keep everything, which is much faster. I have always kept every personal
email, and HD size has grown much faster than the rate I receive/read email,
so I have had no problems with storage. I am a natural hoarder though, and
quite enjoy browsing old email folders to see what I was up to. I also have
copies of course of every email I have ever sent.

I use outlook express, which has some searching facilities built in, which
are good enough for me.

As to actually NEEDING any of this, I expect if i lost the complete email
archive and all backups I'd survive, but I'd definitely be a bit upset.

/Robert

Re: Advanced mail management

[Nick Cleaton]

On Tue, Oct 02, 2001 at 02:25:02PM +0100, Dominic Mitchell wrote:
> 
> I know that.
> 
> You know that.
> 
> Try explaining that to people who don't care.

I don't care.  Practice on me if you like.

Nick

--
SPAM while y#SPAM?\r\n# #d||s#.{24}#$PAM^=$&,''#e||s'$^'sPaM!SpaM!SPaM!SP@M
!SP@M!5PAM!SP@M!SPAM!SPAM!spam!sPaM!sPAM!SpaM!SpAM!SPAM?5P@M!5P@M?Spam?5P@m?sP@
M?SPam?5P@M!5P@M!Sp@M!5P@M?SP@M!5P@M?sPam?sPaM!SP@M?spAm?sPAM!SP@M!5P@M!SP@M?SP
@M!SP@M!SP@M!5PaM!SPAm!SpaM!SP@M!SPAM!SPAM!SP@M!SpaM!SPaM!SPAM!!';print"$PAM\n"

Re: Advanced mail management

[Dominic Mitchell]

On Tue, Oct 02, 2001 at 02:11:56PM +0100, Jon Nangle wrote:
> > "Dominic" == Dominic Mitchell <[EMAIL PROTECTED]> writes:
> 
> Dominic> [1] And emacs wasn't.  It bugs me to see people fire off yet
> Dominic> another emacs copy to make a single edit to a single file.
> Dominic> Swap, swap, swappity swap!
> 
> Well, that's why gnuserv/gnuclient exists.

I know that.

You know that.

Try explaining that to people who don't care.

-Dom

-- 
| Semantico: creators of major online resources  |
|   URL: http://www.semantico.com/   |
|   Tel: +44 (1273) 72   |
|   Address: 33 Bond St., Brighton, Sussex, BN1 1RD, UK. |

Re: Advanced mail management

[Jon Nangle]

> "Dominic" == Dominic Mitchell <[EMAIL PROTECTED]> writes:

Dominic> [1] And emacs wasn't.  It bugs me to see people fire off yet
Dominic> another emacs copy to make a single edit to a single file.
Dominic> Swap, swap, swappity swap!

Well, that's why gnuserv/gnuclient exists.

Jon

Re: Advanced mail management

[Dominic Mitchell]

On Tue, Oct 02, 2001 at 12:21:14PM +, Steve Mynott wrote:
> Dominic Mitchell <[EMAIL PROTECTED]> writes:
> > The trouble that I found with gnus the last time that I used it was that
> > when it wa doing something, it locked your emacs up.  Which is a pain if
> > you were trying to edit some perl code at the same time.
> 
> I am not sure what you mean by "lock up".
> 
> I have no problems in editing perl, IRC and reading mail/news together
> it works fine for me and all I have to do is switch between the
> buffers in the normal emacs way.  I have tried this on several
> platforms (solaris, linux and OpenBSD) and have never seen it lock up.

I'm talking more about accessing news rather mail, over an nntp
connection to a slow server.  My emacs process would stop responding and
painting the screen until it had gotten a response from the server.
This got to be enough of a pain that I no longer use gnus to read news.

Actually, I no longer read news, but that's a different story entirely.
:-)

> However I am unable to compose a mail in mutt (or rather the spawned
> editor) and at the same time view previous messages or browse the
> address book (as I can with gnus).
> 
> So for me it's mutt that seems to lock up.

Agreed.  But mutt was designed around multiple instances[1].

-Dom

[1] And emacs wasn't.  It bugs me to see people fire off yet another
emacs copy to make a single edit to a single file.  Swap, swap,
swappity swap!

-- 
| Semantico: creators of major online resources  |
|   URL: http://www.semantico.com/   |
|   Tel: +44 (1273) 72   |
|   Address: 33 Bond St., Brighton, Sussex, BN1 1RD, UK. |

Re: Advanced mail management

[Steve Mynott]

Dominic Mitchell <[EMAIL PROTECTED]> writes:

> The trouble that I found with gnus the last time that I used it was that
> when it wa doing something, it locked your emacs up.  Which is a pain if
> you were trying to edit some perl code at the same time.

I am not sure what you mean by "lock up".

I have no problems in editing perl, IRC and reading mail/news together
it works fine for me and all I have to do is switch between the
buffers in the normal emacs way.  I have tried this on several
platforms (solaris, linux and OpenBSD) and have never seen it lock up.

However I am unable to compose a mail in mutt (or rather the spawned
editor) and at the same time view previous messages or browse the
address book (as I can with gnus).

So for me it's mutt that seems to lock up.

-- 
1024/D9C69DF9 steve mynott [EMAIL PROTECTED]

socialism: an attempt to curb the destructive power of monopolies
by creating the biggest one of all.

Re: Advanced mail management

[Natalie Ford]

On Tue, Oct 02, 2001 at 09:41:44AM +0100, Leon Brocard wrote:
> I've moved my mail from laptop to laptop to new company computer over
> the years and it's dawned on me that I never (that is NEVER) need
> archives which are more than a week or so old.
> Agressive deletion is the only way.

I would have agreed a few years ago, but with the number of email
client changes, hard disk re-formats and operating system
re-installs I have made recently and the number of software
registrations and licence numbers I have lost in the process
(e.g. I am currently trying to track down my VmWare licence)
which cost money to re-licence, I now archive a lot of stuff...  :)

-- 
Natalie Ford .. [EMAIL PROTECTED]
Atomic Interactive Limited .. [EMAIL PROTECTED]
Yet Another Computer Solutions Company Limited ... [EMAIL PROTECTED]

Re: Advanced mail management

[Dominic Mitchell]

On Tue, Oct 02, 2001 at 11:29:12AM +, Steve Mynott wrote:
> Or use gnus to integrate your mailing list and news reading.
> 
> Gnus (by default) will delete a read email automatically unless you
> tag it in order to save it.
> 
> Because gnus is written in a high level language (lisp) it is
> _extremely_ configurable -- more so than even mutt.  The more recent
> versions even have icons (if you like that sort of thing) and will
> display images.
> 
> It even has internal mail filters so you can throw away procmail and
> do everything in one single program (which is also an advanced editing
> environment and an excellent IDE for perl).

The trouble that I found with gnus the last time that I used it was that
when it wa doing something, it locked your emacs up.  Which is a pain if
you were trying to edit some perl code at the same time.

Any volunteers for writing a multi-threaded emacs?  ;-)

-Dom

-- 
| Semantico: creators of major online resources  |
|   URL: http://www.semantico.com/   |
|   Tel: +44 (1273) 72   |
|   Address: 33 Bond St., Brighton, Sussex, BN1 1RD, UK. |

Re: Advanced mail management

[Steve Mynott]

"Dave Cross" <[EMAIL PROTECTED]> writes:

> From: Leon Brocard <[EMAIL PROTECTED]>
> Date: 10/2/01 9:41:44 AM
> 
> > I've moved my mail from laptop to laptop to new company 
> > computer over the years and it's dawned on me that I 
> > never (that is NEVER) need archives which are more than a 
> > week or so old.
> >
> > Agressive deletion is the only way.
> 
> This is _so_ true. And there's nothing more satisfying[1] than
> deleting a couple of thousand emails from a mailbox :)

Or use gnus to integrate your mailing list and news reading.

Gnus (by default) will delete a read email automatically unless you
tag it in order to save it.

Because gnus is written in a high level language (lisp) it is
_extremely_ configurable -- more so than even mutt.  The more recent
versions even have icons (if you like that sort of thing) and will
display images.

It even has internal mail filters so you can throw away procmail and
do everything in one single program (which is also an advanced editing
environment and an excellent IDE for perl).

-- 
1024/D9C69DF9 steve mynott [EMAIL PROTECTED]

just once, i wish we would encounter an alien menace that wasn't
immune to bullets -- the brigader, "dr. who"

Re: Advanced mail management

[Simon Wistow]

On Tue, Oct 02, 2001 at 12:12:05AM -0700, Paul Makepeace said:
> That seems like a good partial solution. The problem is when you have
> folders with tens of thousands of emails (London.pm is about to hit 20k
> here). The disadvantage to chopping these massive folders up is
> searching later[1].


Every week, last week's mail goes into london.pm--MM where appropriate.

To search through mails I just do

% grep 'search string' mail/london.pm*

or I can restrict it to year or month.

I could actually write a very simple app using Mail::MBox or whatever that
would actually allow me more powerful searching.


As for mutliple incoming folders there are couple of options ...

Do what Mr Clamp does which is label all incoming folders with something
unique ... say incoming_* and then have 

mailboxes `echo +inbox  ~/mail/incoming_*`

in your muttrc

or do soemthing similar and just have

mailboxes `echo ~/mail/* | perl -ne 'print unless /^sent/'`

you can probably do that in sed which would be better but you get the idea

I did think about doing stuff using RDMBS, PerlFS and some hackery but haven't
got round to it. Doing that you could just pretend all your mails were in mbox
(or maildir) folders even though they were in a DB and all (good) maill apps
would work with it automagically :) Plsu you could do neat tricks with
archiving and stuff.






-- 
: nature notes for the apocalypse

Re: Advanced mail management

[David Cantrell]

On Tue, Oct 02, 2001 at 02:16:08AM -0700, Dave Cross wrote:
> From: Leon Brocard <[EMAIL PROTECTED]>
> Date: 10/2/01 9:41:44 AM
> > I've moved my mail from laptop to laptop to new company 
> > computer over the years and it's dawned on me that I 
> > never (that is NEVER) need archives which are more than a 
> > week or so old.
> >
> > Agressive deletion is the only way.
> This is _so_ true. And there's nothing more satisfying[1] than
> deleting a couple of thousand emails from a mailbox :)

I tend not to filter everything into seperate folders, I just read my inbox
in chronological order.  Yes, I delete messages having read them, but I *do*
have a procmail filter which makes a copy of every single post, before any
other filters.  Every month or so I compress that archive folder and keep it.
There are a few lists which I filter into their own folders, but neither
london.pm nor (void) are one of them.

> And don't most mailing lists have searchable web-based archives
> anyway?

For some definitions of searchable.  Generally you can easily search by
subject and by date, but that doesn't help when you have obscene amounts
of thread drift and you can't remember that it was the February 2000
that someone posted the PROM monitor command you need.

In any case, it's not as if archives take that much space.

david@plough:~/archived$ ls -l inbox-2001*
-rw-rw-r--1 davidusers13531466 Feb  1  2001 inbox-2001-01.gz
-rw-rw-r--1 davidusers11796117 Mar  5  2001 inbox-2001-02.gz
-rw-rw-r--1 davidusers12473481 Apr  8 17:33 inbox-2001-03.gz
-rw-rw-r--1 davidusers13945763 May 16 22:28 inbox-2001-04.gz
-rw-rw-r--1 davidusers12550679 Jun  6 14:55 inbox-2001-05.gz
-rw-rw-r--1 davidusers15536947 Jul 11 12:28 inbox-2001-06.gz
-rw-rw-r--1 davidusers20028672 Aug  4 21:04 inbox-2001-07.gz
-rw-rw-r--1 davidusers19995496 Sep 10 14:00 inbox-2001-08.gz

-- 
David Cantrell | [EMAIL PROTECTED] | http://www.cantrell.org.uk/david

  Good advice is always certain to be ignored,
  but that's no reason not to give it-- Agatha Christie

Re: Advanced mail management

[Dave Cross]

From: Leon Brocard <[EMAIL PROTECTED]>
Date: 10/2/01 9:41:44 AM

> I've moved my mail from laptop to laptop to new company 
> computer over the years and it's dawned on me that I 
> never (that is NEVER) need archives which are more than a 
> week or so old.
>
> Agressive deletion is the only way.

This is _so_ true. And there's nothing more satisfying[1] than
deleting a couple of thousand emails from a mailbox :)

And don't most mailing lists have searchable web-based archives
anyway?

Dave...

[1] Ok, so I know this isn't strictly true.

-- 


"Let me see you make decisions, without your television"
   - Depeche Mode (Stripped)

Re: Advanced mail management

[Dominic Mitchell]

On Tue, Oct 02, 2001 at 09:41:44AM +0100, Leon Brocard wrote:
> Paul Makepeace sent the following bits through the ether:
> > The problem is when you have folders with tens of thousands of
> > emails (London.pm is about to hit 20k here)
> 
> I've moved my mail from laptop to laptop to new company computer over
> the years and it's dawned on me that I never (that is NEVER) need
> archives which are more than a week or so old.
> 
> Agressive deletion is the only way.

True for mailing lists, but other stuff like inboxes, company mail and
system mails (inn reports, bsd daily reports) are damned useful to refer
back to.  I also keep a folder full of bookmarked urls which is very
nice and easy to search through with mutt.

-Dom

-- 
| Semantico: creators of major online resources  |
|   URL: http://www.semantico.com/   |
|   Tel: +44 (1273) 72   |
|   Address: 33 Bond St., Brighton, Sussex, BN1 1RD, UK. |

Re: Advanced mail management

[Struan Donald]

* at 01/10 15:56 -0700 Paul Makepeace said:


 
> While I'm on this subject, how do people deal with searching email?
> Archival management? (And how that impacts searching?) I will write up a
> Web page on all this seeded on whatever comes out of this, if there
> isn't one already.

grepmail: http://grepmail.sourceforge.net/

deal with compressed mail folders and will recurse as well so at worst
you can do:

grepmail whatever ~mail/*

and then 

grepm: http://www.barsnick.net/sw/grepm.html

which opens the results in mutt (although there are equivalents for
other MUAs...)

and it's perl. (although not really looked at the code[0] so might be
perl with XS bits for speed...)

s

[0] I'm just too lazy to do that unless it doesn't work

Re: Advanced mail management

[Robin Houston]

I have this in my .bash_profile:

  export MAILCHECK=0
  MAILPATH=$MAIL
  for mbox in ~/Mail/*; do
if [ "$mbox" != ~/Mail/sent ]; then
  MAILPATH=$MAILPATH:$mbox\?"Mail in ${mbox#~/Mail/}"
fi
  done
  export MAILPATH
  
I know it only addresses a very small part of the problem.

 .robin.

Re: Advanced mail management

[Leon Brocard]

Paul Makepeace sent the following bits through the ether:

> The problem is when you have folders with tens of thousands of
> emails (London.pm is about to hit 20k here)

I've moved my mail from laptop to laptop to new company computer over
the years and it's dawned on me that I never (that is NEVER) need
archives which are more than a week or so old.

Agressive deletion is the only way.

Leon
-- 
Leon Brocard.http://www.astray.com/
Nanoware...http://www.nanoware.org/

... SYSTEM ERROR:  press F13 to continue...

Re: Advanced mail management

[Paul Makepeace]

On Tue, Oct 02, 2001 at 07:41:28AM +0100, Greg McCarroll wrote:
> Well my solution, is not that good, but it works for me, i simly use
> wmbiff, a little windowmaker docking bar app that lists the number of
> messages in each folder, unless there are new messages in which case
> it lists the number of new messages in a different colour.

That seems like a good partial solution. The problem is when you have
folders with tens of thousands of emails (London.pm is about to hit 20k
here). The disadvantage to chopping these massive folders up is
searching later[1].

This is what I use with exim:

$ alias mailwatch
alias mailwatch='tail -f /var/log/exim/mainlog | perl -lne '\''print "$2 $1 $3" if 
m~(\d+:\d+).*([-=]>) (?:$ENV{HOME}/mail/)?(\S+)~'\''&'
$

This has the negative effect of making me aware of *every single
message* that arrives, and the concommitment compulsion to read it is
often too much to resist.

I was going to write this as an IE CDF Channel a few years ago but
realized CDF is crap, little more than (AFAICT) a web page with a user
configurable refresh. It isn't really true push (again, AFAICT).

> of alias london="mutt -f ~/mail/london.pm", etc. That I can
> call from a shell when i want to read a folder. If i am

That's great when you don't have thousands of messages in each one :)

Paul

[1] which prompts the [insert adjective] reaction of putting your entire
mail collection at the mercy of google's spider and using
site:searchs..

Re: Advanced mail management

[Greg McCarroll]

* Greg McCarroll ([EMAIL PROTECTED]) wrote:
> 
> Well my solution, is not that good, but it works for me, i simly use
> wmbiff, a little windowmaker docking bar app that lists the number of
> messages in each folder, unless there are new messages in which case
> it lists the number of new messages in a different colour.

As shown in,

  http://217.34.97.146/~gem/misc/screenshot_scully_with_new_mail.jpg

for people with more bandwidth than sense.

Greg


-- 
Greg McCarroll http://217.34.97.146/~gem/

Re: Advanced mail management

[Greg McCarroll]

* Paul Johnson ([EMAIL PROTECTED]) wrote:
> On Mon, Oct 01, 2001 at 03:56:06PM -0700, Paul Makepeace wrote:
> > Like many people's, my incoming mail is routed off into various folders
> > by the MTA which I then read with mutt. The problem is that while mutt's
> > mail folder handling is pretty darn good it's still a pain for it to
> > scan the folders[1]. There are about a dozen I like to read throughout
> > the day.
> 
> This is what I do too, but I agree that there is still a problem waiting
> for a really good solution.
>

Well my solution, is not that good, but it works for me, i simly use
wmbiff, a little windowmaker docking bar app that lists the number of
messages in each folder, unless there are new messages in which case
it lists the number of new messages in a different colour.

To be perfectly honest I use 3 instances of it, giving me a total
of 15 folders listed.

You can see it in action, in this old screen shot ...

http://217.34.97.146/~gem/misc/screenshot_scully.jpg

Its the little grouping of three boxes in the middle of the
right hand edge. Then I have a pile of aliases along the lines
of alias london="mutt -f ~/mail/london.pm", etc. That I can
call from a shell when i want to read a folder. If i am
interested in several, i can do something like

london ; void ; squackers

I tried for a while at getting festival to let me know 
when particular people mailed me, such as my wife, accountant
etc. But I never really finished the larger project this
was part of ... Radio Greg (i mixing of mp3's and festival
generated infomercials), but thats another story.

> > about putting mail itself in RDBMS? That way you could create virtual
> > folders etc.
> 
> I've heard talk of this, but never seen a good implementation.

I think if this sort of thing was ever implemented properly, I'd
want to use it for more than just mail, i.e. project files as well
etc.

I'm getting closer and closer to having a static organised
filesystem on my home network, when this happens i might look
at extending my TODO list to have a little pop up virtual
folder containing links to related files. A sort of half
way house towards the sort of system your talking about (albeit
for more than just mail)

Greg

-- 
Greg McCarroll http://217.34.97.146/~gem/

Re: Advanced mail management

[Paul Johnson]

On Mon, Oct 01, 2001 at 03:56:06PM -0700, Paul Makepeace wrote:
> Like many people's, my incoming mail is routed off into various folders
> by the MTA which I then read with mutt. The problem is that while mutt's
> mail folder handling is pretty darn good it's still a pain for it to
> scan the folders[1]. There are about a dozen I like to read throughout
> the day.

This is what I do too, but I agree that there is still a problem waiting
for a really good solution.

> How do people deal with this appalling modern dilemma? One hack I've
> considered is saving duplicates of my interesting folders into one
> folder which I set to threaded display & read as a stream as it comes

I tried this ...

> in. This leaves the hassle of having lots of other folders with unread
> messages (so perhaps a script that marks those as read based on
> message-id's -- anyone seen/done this?). Anyone played with/heard

... but for me the problems outweighed he benefits.

> about putting mail itself in RDBMS? That way you could create virtual
> folders etc.

I've heard talk of this, but never seen a good implementation.

> While I'm on this subject, how do people deal with searching email?
> Archival management? (And how that impacts searching?) I will write up a
> Web page on all this seeded on whatever comes out of this, if there
> isn't one already.

I have a script that I occasionally run which turns london.pm into
london.pm.20011002.bz2 for example.  I use it on log files too.

My high tech archival searching goes something like

  bz2cat london.pm.20011002.bz2 | less
  /interesting_stuff
  nnn

or

  bz2cat london.pm.20011002.bz2 | gvim -

gives the syntax colouring.  Occasionally I'll just decompress the
file and load it into mutt, if I want to follow the thread for example.

No, it's not nice.

-- 
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net