Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Hi Jon, Many years ago I added the hooks interface to nmh. Hasn't been as controversial as my attachment handling stuff, which makes me think that it isn't widely used. I might vaguely recall it being mentioned. I found http://git.savannah.gnu.org/cgit/nmh.git/tree/docs/README-HOOKS. Perhaps the lack of man-page coverage has hindered its uptake? Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
I'm not sure I follow. From where I'm sitting MH mailboxes are only useful with traditional Unix tools true if the vast majority of email you have in your mailbox is US-ASCII text/plain only. Can't speak for anyone else, but that's slowly becoming less and less true for me. Since MH tools only pay attention to files matching /^\d+/, it seems like breaking messages out into \d+.attachments would make things easier to grep, etc. I suggested this before, but having recently learned about hooks, this should be easy for me to mock-up sometime this week https://github.com/mcr/nmh/blob/master/docs/README-HOOKS inc (add hook) will mhstore to \d.attachment refile will rename and move as necessary rmm will remove everything else is transparent, although aliases could be made for mhlist/mhshow, which use the pre-dumled attachments. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Ken Hornstein k...@pobox.com writes: See, this is where we run into two competing ideas as to what MH's original innovation is. Do people think it's: 1) The message store (1 file per message), instead of one file for the whole store. 2) Individual commands to perform message operations, as opposed to traditional monolithic MUAs. Both. -- Bill Wohler woh...@newt.com aka bill.woh...@nasa.gov http://www.newt.com/wohler/ GnuPG ID:610BD9AD ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
if mail commands can't trivially be part of shell scripts then having one message in a file is a lot less interesting I'm not sure I follow. From where I'm sitting MH mailboxes are only useful with traditional Unix tools true if the vast majority of email you have in your mailbox is US-ASCII text/plain only. Can't speak for anyone else, but that's slowly becoming less and less true for me. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Ken Hornstein wrote: if mail commands can't trivially be part of shell scripts then having one message in a file is a lot less interesting I'm not sure I follow. From where I'm sitting MH mailboxes are only useful with traditional Unix tools true if the vast majority of email you have in your mailbox is US-ASCII text/plain only. Can't speak for anyone else, but that's slowly becoming less and less true for me. given the number of messages in my mail repo whose bodies are in quoted-printable or base64, or which have attachments that are binary, i've only been able to access my stored mail, or indeed inbound mail containing commands of some kind, with perl's MIME modules for some time now. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
On Feb 22, 2014, at 21:20, Paul Vixie p...@redbarn.org wrote: given the number of messages in my mail repo whose bodies are in quoted-printable or base64, or which have attachments that are binary, i've only been able to access my stored mail, or indeed inbound mail containing commands of some kind, with perl's MIME modules for some time now. There is nothing that says we cannot undo transport encodings for text/* parts when we write store a message into a folder. And despite the prevalence of QP and base64, I still find grep works quite well for me. Anything text/* that is all base64 is almost always spam, so the base64 encoding is actually a 'feature' for me, ensuring the message doesn't generate a false positive match for what I am really looking for. If we undid the transport encodings for text/* and converted everything to utf8, you could then, e.g., alias mhgrep='env LC_ALL=en_CA.UTF-8 grep'. --lyndon ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Lyndon Nerenberg wrote: If we undid the transport encodings for text/* and converted everything to utf8, you could then, e.g., alias mhgrep='env LC_ALL=en_CA.UTF-8 grep'. all true. but there's a strong feeling among MH zealots, of whom i am one and which feeling i do share, that what gets written to $HOME/Mail/inbox/42 ought to be the raw BODY that came in over SMTP. i'm beyond the point of wanting 'anno' dead, since it perturbs this. i'm reaching the point where the \r\n terminators present in the SMTP session seem to me like they ought to be kept, but i know the transport (postfix) strips them before handing off to the local delivery agent. so while i also want to break down every message into its body and parts, and build a directory tree for those, and convert all of the text parts to native filesystem text format... i don't want to do away with or in any way modify the message per file that MH gives me now. do both is not too expensive in this day and age. vixie ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
And despite the prevalence of QP and base64, I still find grep works quite well for me. Anything text/* that is all base64 is almost always spam, so the base64 encoding is actually a 'feature' for me, ensuring the message doesn't generate a false positive match for what I am really looking for. Sigh. I wish that was true for me, but it isn't. Various mobile devices used by co-workers default to base64 for text/plain. Also, if you send something with the high bit set to this mailing list, it helpfully converts it to base64 for you. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
On Feb 22, 2014, at 21:43, Paul Vixie p...@redbarn.org wrote: all true. but there's a strong feeling among MH zealots, of whom i am one and which feeling i do share, that what gets written to $HOME/Mail/inbox/42 ought to be the raw BODY that came in over SMTP. But which SMTP? The MTAs are already free to modify the transport encoding on the fly. E.g., I submit an 8bit message to my local MTA. It then forwards it to your local MTA, which only supports 7 bit ASCII, requiring my MTA to apply a transport encoding to the message before sending it to you. Which of these two SMTP hops is the authoritative one you want reflected in the on-disk representation of the message? --lyndon ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Lyndon wrote: There is nothing that says we cannot undo transport encodings for text/* parts when we write store a message into a folder. mhfixmsg -decodetext 8bit And despite the prevalence of QP and base64, I still find grep works quite well for me. Agreed. If we undid the transport encodings for text/* and converted everything to utf8, you could then, e.g., alias mhgrep='env LC_ALL=en_CA.UTF-8 grep'. mhfixmsg -textcharset UTF-8, if built with iconv. Paul V. wrote: all true. but there's a strong feeling among MH zealots, of whom i am one and which feeling i do share, that what gets written to $HOME/Mail/inbox/42 ought to be the raw BODY that came in over SMTP. mhfixmsg moves the original message the same way as rmm, if you'd like to keep it. It will create a , file (by default) or call your rmmproc. David ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Hi Norm, Here, for example, is one stab at how messages would be stored. Each message would be a directory. Each component would be a file or directory, whose name was the component name, Each MIME part would be a directory. Plan 9's filesystem that presents an email as a directory is worth a look. It includes raw* files that give the original email. http://man.cat-v.org/plan_9/4/upasfs Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Hi Jeff, Just because you can use a lot of disk space doesn't mean you should. If you use a database backend for indexing the messages you detect and avoid duplicate copies of attachments. The actual pieces of the message could still exist in directories with duplicate copies linked in. Attachments, and copies of attachments, are the main source of disk usage in my mail folders. I've been idly thinking about this quite a bit, and not just for MH's ~/mail. I think deduplication is better done by the filesystem than every application. If the filesystem doesn't provide it then you get bloat, but we already have that. Hard links act coarsely on whole files and have only one inode for metadata. btrfs, a Linux filesystem, has IIRC a filesystem format that lets a sequence of stored bytes be referred to by other stored data, perhaps with a compression method involved. The ioctl(2) interface only works for runs of data that are a multiple of block size but the on-disk format is more flexible and perhaps the code will catch up. btrfs does some deduplication as it receives data but the rest is done as a background task, and userspace gets involved AIUI, so it isn't putting lots of application logic in a filesystem. btrfs isn't alone in supporting deduplication beyond hard links. For a filesystem in archive mode one can imagine an email with a base64-encoded tar file sitting in ~/mail, foo.tar being extracted and including foo/bar.ps. Over time, this might become a compressed bar.ps stored on disk with foo/bar.ps referring to it, foo.tar referring to foo/bar.ps along with other files, and *part* of ~/mail/inbox/42 being stored using compression method base64 on foo.tar. Yes, overhead in CPU and access-time but then Facebook have developed a Blue-ray disk-changer for archive storage. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Hi Ken, I dunno; to me, databases are fine for specialized things, but just add a level of complexity that I don't necessarily think is useful. Also, I'm not sure of the savings; maybe it's just me, but I rarely get identical copies of the same message or attachment. I think it's just you. Those in non-computing offices send me lots of emails, each with a moderately-sized image or two for the footer, quite often in a poor choice of image format. Then a long thread just tacks on a few lines at the top and another attachment or two at the bottom, leaving all the others present. Sometimes I get all the attachments duplicated, ABCABC, sometimes thrice. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Hi Ken, 1) The message store (1 file per message), instead of one file for the whole store. 2) Individual commands to perform message operations, as opposed to traditional monolithic MUAs. I've stuck with MH because it's both. Either on its own may not have kept me over the decades. But I believe the answer is 2). To me the power of MH is using the individual commands, which lets you do scriptable operations from the shell, and also enables the front-ends that have cropped up. What you seem to be saying is that instead of an MUA where you are in its own shell, e.g. mail(1), you have an MUA comprised of many commands, scriptable with a bit of shell control-flow, but no commands outside the MUA's set should touch the storage. I want to find emails partially by email metadata but also by using Unix's toolset on them, e.g. I'm sure there was a email from Tom, copied to Dick, with a tar file containing a PDF produced by his new typesetter; pdfinfo(1)'s output would come into play. pick(1) or its replacement isn't enough alone, nor should it insist I stay within its confines by adding find-like -execs. If your model is you want people to have to look directly in the store to copy out PDFs that people send to you ... well, that just seems like it sucks to me from a UI perspective. For common operations, MH can offer convience in accessing the store, but it shouldn't be off limits and the current format is inadequate in the MIME world. I guess I'd want to know what people want to happen when they show a message with some images or PDFs attached to it. Let's figure out what UI people have in their minds and work from there. MH is two things as you pointed out. Yes, figure that out but the UI is just one half. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Hi Ken, Maybe a set of primitives, like mhcat that would let you output a part of a message on stdout Aside, I have $ readlink ~/bin/mhcat /usr/bin/mh/mhstore $ Don't recall where I learnt to create it, mhstore(1) here doesn't mention `mhcat', but I use it quite a bit. The `storing message 42 to stdout' on stderr is annoying but I've never looked into if it can easily be shut-up. It clearly should be off by default unless running on VMS. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Hi Paul, i would. first, i'd adopt the messages-as-directories model norm gave, since it supports MIME, where our current model does not. by support i mean i want to use normal UNIX file access to work with my message store. Yes. The original email should also be available, like upas's raw* files, but I'd like to see decoded UTF-8, with Unicode canonical equivalence taken into account, for the headers and body parts, and headers unwrapped so each is on a single line. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
What you seem to be saying is that instead of an MUA where you are in its own shell, e.g. mail(1), you have an MUA comprised of many commands, scriptable with a bit of shell control-flow, but no commands outside the MUA's set should touch the storage. Well, not exactly. I guess my feeling is that the store isn't off-limits, but we don't make any guarantees. We should PROVIDE the tools you need without having to resort to mucking around in the store. But this exposes some tension here. You say that the current store is inadequate. Okay, I can't really argue that. But ... we've got a problem here. We have a number of front-ends that have grown up assuming that the store isn't going to change. If we went to a directory-per-message, that would bust every front-end out there. I mean, we already broke things for MH-E when we went to comp(1) supporting mh-format(5), because MH-E assumed that it could read components directly. We can argue about whether or not that was a valid assumption, but it hadn't changed for over 30 years so I can't really blame them on that one. Sure, we can develop a migration plan, some settings to decide if a folder should be nmhv2 format, etc etc ... but that just makes it more complicated. I guess my core thought here is that if people want to change the store, THEY should work on that. I am personally neutral; I can see the advantages, but I'd rather work on improving the UI side of things. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
$ readlink ~/bin/mhcat /usr/bin/mh/mhstore $ Don't recall where I learnt to create it, mhstore(1) here doesn't mention `mhcat', but I use it quite a bit. Are you sure you don't have something in .mh_profile that makes that work? I don't see anything in the code that would make that work (and I just tried that myself and it didn't send the content to stdout). --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Hi Ken, $ readlink ~/bin/mhcat /usr/bin/mh/mhstore $ Don't recall where I learnt to create it, mhstore(1) here doesn't mention `mhcat', but I use it quite a bit. Are you sure you don't have something in .mh_profile that makes that work? I don't see anything in the code that would make that work (and I just tried that myself and it didn't send the content to stdout). Ah, I bet I do. $ g mhcat ~/.mh_profile mhcat-store-application: - mhcat-store-text: - $ Yep. :-) Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Hi Ken, You say that the current store is inadequate. Okay, I can't really argue that. But ... we've got a problem here. We have a number of front-ends that have grown up assuming that the store isn't going to change. No, this isn't nmh. This is Norm's MH-2014. :-) The other projects can either stick or twist. Cheers, Ralph. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
n...@dad.org writes: I am suggesting this exercise only for recreational purposes. Design MH, from scratch, without the constraints affecting the original MH design. Suppose, you weren't designing a system to run on a time shared PDF 1145, but on a single user, multi-core system. Suppose that you had multi gigabyte disks available. But also suppose you had to worry about distributed data and processing. But, mostly importantly, suppose that you didn't have to worry about being too radical for acceptance. What, then. would MH look like? Here, for example, is one stab at how messages would be stored. Each message would be a directory. Each component would be a file or directory, whose name was the component name, Each MIME part would be a directory. The design of my future mail reader will be as follows: mail is stored in Maildir (read-only support for Mailboxes is available, too). Maildir support allows me to use offlineimap to sync against an IMAP store. However, for reading the *results of query* against the store (for now, I use mairix for this), such as all new/unread mail, all mail matching $words etc. will be rendered into trees like this: idx cur - 1/ 1/raw - /home/chris/Mail/INBOX/cur/1343755814_1.9959.juno,U=623,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S 1/text 2/raw - /home/chris/Mail/INBOX/cur/1343755654_3.9959.juno,U=544,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S 2/text 3/raw - /home/chris/Mail/INBOX/cur/1343755143_0.9959.juno,U=131,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,S 3/text 4/0 4/0/0 4/0/1 4/0/2 4/1 4/musicextras-gui.po - 0/2 4/musicextras.po - 0/1 4/raw - /home/chris/Mail/INBOX/cur/1343755102_3.9959.juno,U=89,FMD5=7e33429f656f1e6e9d79b29c3f8 The file idx contains the sender and subject data, easily sortable by various keys (date, from, subject, threaded), as well as the imap flags. (i.e. like a cache of MH scan(1)). The text contains a rendered version (roughly like mhshow) of the mail kept in raw (for mbox, this is generated, for Maildir, it's just a symlink). HTML is rendered with w3m or lynx, for zip files there is a table of contents etc. MIME attachments are readily expanded into files (e.g. message 4), so you can apply patches directly or look at attached pictures. MH-style tools operate on these trees. next/prev update the cur symlink to keep track. Updating message flags like read/replied/trash directly operate on the Maildir, but also update the rendered tree (yet tbd). I'm not yet sure about other sequences (they could be realized as lists of Maildir files, but then they need to be updated on flag changes...) These trees are ephemeral, when I'm done reading them they will be deleted. Rendering them is plenty fast, reading is instant (I keep them in an tmpfs, even). You can have several trees in different directories (probably an environment variable will choose the current one, or it will check if $PWD is one). You can also render them from procmail or cron if you really want instant mail reading. The system is about 2/3 done so far and essentially consists of a 1KLOC Ruby script (with no foreign dependencies) and a bunch of shell scripts and aliases. The mechanic is really simple. Currently, I'm considering (ab)using vi(m) as an interface for reading many mails quickly. vim macros are easy to bind to keys, can run external commands and show their output, and provide split windows. I have been thinking about this for years now and tried various other ideas already. Please ask if anything is unclear. -- Christian Neukirchen chneukirc...@gmail.com http://chneukirchen.org ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
So again, I think that the UI issues and message store issues are independent. I'm personally much more interested in the UI issues. As I've said before, the ability to scan/next/prev/show attachments would do it for me. This wouldn't be too hard to add to the current UI. Oh, and the ability to show an attachment to stdout. I'd work on it if I had time. I don't want to mess with the message store because of the number of tools that people have written that count on that format. And, I don't think that it needs to be changed. Many years ago I added the hooks interface to nmh. Hasn't been as controversial as my attachment handling stuff, which makes me think that it isn't widely used. The hooks interface allows you to specify programs in your profile that get invoked when messages are added/deleted/refiled. I use these hooks to build a separate searchable database. I have a number of additional command line utilities that operate on this database. Any of you could use these hooks to maintain a separate directory tree that stores mail in a format that you prefer, such as one directory per message. And, I suppose, you could construct scripts that would pluck attachments from that tree. You could use the hooks to prototype your ideas for other storage formats. Probably less than an hour of work. I would point out that a minor problem with this approach, which would also exist if nmh were to change storage systems, is making sure that the subsidiary files are synchronized with the originals. I have (and nmh would need) a fsck-like utility that checks and fixes this stuff. Problems don't often arise, but a disk-full or system crash at the wrong time can mess things up. Jon ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
I think the whole exercise is too limited. If we're going to really blue sky an idea, lets be revolutionary. First of all, the whole concept of email is kind of obsolete. There should be a universal messaging protocol that allows, for example, me to send you a message via email, and you to receive it as a text on your phone, or however you choose to receive it. In other words, people should be able to use any messaging system they choose to send, and to receive, without regard to intermediate formats. Conversion, if necessary, would be completely invisible to the users. This universal messaging protocol should use something like XML or ASN.1 or some other more universal format. MIME is pretty much a kludge, and a pretty inefficient one at that. Binary data has to be converted via Base64 encoding or something which, if memory serves, expands it by 20% or more. (Of course, XML has this problem too, but maybe just using URIs and having the data accessed on demand would solve this and also reduce message size.) I note that GMail is now pre-fetching remote content, such as embedded images, and showing you copies on their servers when you read the message, rather than accessing it from its source each time. This is more efficient, and protects your identity from potential hostiles, so that seems like a pretty good model. When I was working on my own browser-based email client, the working title was AFMP, for Archive/Folder/Message/Part, the four levels of navigation which are visible to the user. In hindsight, even that seems pretty nerdy, since most users don't think of messages as having parts anyway. They think of heterogeneous documents with text, voice, pictures, animation, etc. Of course, I also believe the Web should do everything that Facebook does, but without the proprietary stuff, ads and privacy concerns. So take what I say with a grain of salt. -pd -- Peter Davis The Tech Curmudgeon www.techcurmudgeon.com ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
maybe just using URIs and having the data accessed on demand would solve this and also re duce message size. MIME already has this. It's nice in theory, but it doesn't get used. Among other things, it requires people to treat URIs as they're meant to (permanent identifiers), and also to maintain a publicly accessible resource server. Not to mention issues of the sender redacting a resource they sent after the fact. GMail's image cacheing is to protect users from tracking by spammers/phishers, while probably allowing them to do a bit more of their own info collection X-P As for your Universal Messaging Protocol, that is way beyon the scope of this list I think, although Jabber's probably a good place to start with it. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Jon Steinhart j...@fourwinds.com wrote: I use these hooks to build a separate searchable database. I have a number of additional command line utilities that operate on this database. I'm interested in trying this. Any of you could use these hooks to maintain a separate directory tree that stores mail in a format that you prefer, such as one directory per message. I would also run this if it existed. I would point out that a minor problem with this approach, which would also exist if nmh were to change storage systems, is making sure that the subsidiary files are synchronized with the originals. I have (and nmh would need) a fsck-like utility that checks and fixes this stuff. Problems don't often arise, but a disk-full or system crash at the wrong time can mess things up. True. I think the right approach is that we need to figure out what are the missing APIs that would make mh-e and exmh happy, and add them before we think about any changes to the store. I move my older emails from my desktop to another machine, and they become essentially read-only, but I'd sure like to have a cached read-only copy on my laptop. I keep thinking that disconnected AFS is really the right answer, but to date, I have never run AFS... -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works| network architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
[Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
I am suggesting this exercise only for recreational purposes. Design MH, from scratch, without the constraints affecting the original MH design. Suppose, you weren't designing a system to run on a time shared PDF 1145, but on a single user, multi-core system. Suppose that you had multi gigabyte disks available. But also suppose you had to worry about distributed data and processing. But, mostly importantly, suppose that you didn't have to worry about being too radical for acceptance. What, then. would MH look like? Here, for example, is one stab at how messages would be stored. Each message would be a directory. Each component would be a file or directory, whose name was the component name, Each MIME part would be a directory. There would be program, call it mhpath, that would give the URLs of selected parts of selected messages. Norman Shapiro ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Just because you can use a lot of disk space doesn't mean you should. If you use a database backend for indexing the messages you detect and avoid duplicate copies of attachments. The actual pieces of the message could still exist in directories with duplicate copies linked in. Attachments, and copies of attachments, are the main source of disk usage in my mail folders. You could go even further and use FUSE to present a legacy interface. While maintaining a backend in another format, even on an IMAP server. Thanks Jeff -- Jeffrey C. Honig j...@honig.net http://www.honig.net/jch GnuPG ID:14E29E13 http://www.honig.net/jch/key.shtml On Wed, Feb 19, 2014 at 11:22 AM, n...@dad.org wrote: I am suggesting this exercise only for recreational purposes. Design MH, from scratch, without the constraints affecting the original MH design. Suppose, you weren't designing a system to run on a time shared PDF 1145, but on a single user, multi-core system. Suppose that you had multi gigabyte disks available. But also suppose you had to worry about distributed data and processing. But, mostly importantly, suppose that you didn't have to worry about being too radical for acceptance. What, then. would MH look like? Here, for example, is one stab at how messages would be stored. Each message would be a directory. Each component would be a file or directory, whose name was the component name, Each MIME part would be a directory. There would be program, call it mhpath, that would give the URLs of selected parts of selected messages. Norman Shapiro ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Just because you can use a lot of disk space doesn't mean you should. If you use a database backend for indexing the messages you detect and avoid duplicate copies of attachments. The actual pieces of the message could still exist in directories with duplicate copies linked in. Attachments, and copies of attachments, are the main source of disk usage in my mail folders. I dunno; to me, databases are fine for specialized things, but just add a level of complexity that I don't necessarily think is useful. Also, I'm not sure of the savings; maybe it's just me, but I rarely get identical copies of the same message or attachment. You could go even further and use FUSE to present a legacy interface. While maintaining a backend in another format, even on an IMAP server. Ow, FUSE? Say goodbye to portability. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
On Feb 19, 2014, at 8:22 AM, n...@dad.org wrote: Suppose, you weren't designing a system to run on a time shared PDF 1145, but on a single user, multi-core system. Suppose that you had multi gigabyte disks available. MH is not resource constrained by CPU, or memory for that matter, so I would change nothing. As for disk space, how much you use is a function of the types of email you receive and how much of a pack rat you are. This is not a function of your MUA. (Well, it might be, indirectly. If you are forced to use an MUA that really really sucks, odds are you won't be saving many messages.) But also suppose you had to worry about distributed data and processing. This I don't understand. How would reading/storing/replying/searching e-mail require distributed processing? And what do you mean by distributed data? You can already store your Mail hierarchy on (say) NFS mounted filesystems. Are you talking about supporting multiple storage back-ends (i.e. IMAP)? I'm not convinced this would work. You are always going to run into back ends that don't support some part of the required semantics (think message annotations with IMAP), so ultimately you end up crippling MH's functionality. If I was going to do it all over again, most of the changes I would make would be to streamline the commands and their interfaces to make them a bit more amenable to scripted processing of the messages. It would also have the hindsight of MIME not being an add-on. Perhaps the most intrusive change I would inflict would be to make the whole environment UTF-8 from end-to-end. --lyndon signature.asc Description: Message signed with OpenPGP using GPGMail ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Lyndon Nerenberg wrote: On Feb 19, 2014, at 8:22 AM, n...@dad.org wrote: Suppose, you weren't designing a system to run on a time shared PDF 1145, but on a single user, multi-core system. Suppose that you had multi gigabyte disks available. MH is not resource constrained by CPU, or memory for that matter, so I would change nothing. i would. first, i'd adopt the messages-as-directories model norm gave, since it supports MIME, where our current model does not. by support i mean i want to use normal UNIX file access to work with my message store. there is no file-level access to attachments right now, which means i have to run mhstore to turn an opaque black-box attachment into a UNIX file i can access. mhstore is highly impure by MH's overall design principles, even though i'm grateful to have it under the circumstances. second, i'd put all of MH's folder and message and reception and transmission logic into a library. this library would be fast and well documented even though it might make huge use of the heap which would have made it impractical in the PDP 11/45 era. all MH command line utilities would be UI wrappers around this fast portable well documented library. we would encourage languages beyond C to skin our API so that it could be directly called by perl, python, php, e-lisp, and so on. the definition of MH aware would add uses MH's library to the other elements already present (can access things using normal UNIX file methods, and knows how to execute MH commands and parse their output. mh-e would become a library client that would almost never have to call popen. third and finally, i'd fully support the IMAP UID psychosis, by offering persistent-for-the-life-of-the-message numeric identifiers, and i'd add code to the library and UI elements to the command set to access things that way. an IMAP server should be able to link to the MH library and everything should just work. note well: this is practical today, as long as we can still access old style folders as well, and as long as we offer some kind of FUSE interface (for systems who have FUSE or similar) to allow new-style folders to be accessed with old-style tools. vixie ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
On Wed, Feb 19, 2014 at 1:33 PM, Lyndon Nerenberg wrote: As for disk space, how much you use is a function of the types of email you receive and how much of a pack rat you are. Yep. Ages ago, if one had a ton of email, you may have issues of running out of inodes, but I do not think that is much of a problem anymore. --ewh ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
On Wed, Feb 19, 2014 at 1:44 PM, Paul Vixie wrote: i would. first, i'd adopt the messages-as-directories model norm gave, since it supports MIME, where our current model does not. by support i mean i want to use normal UNIX file access to work with my message store. there is no file-level access to attachments right now, which means i have to run mhstore to turn an opaque black-box attachment into a UNIX file i can access. mhstore is highly impure by MH's overall design principles, even though i'm grateful to have it under the circumstances. I have mixed feelings about this. Personally, I want the original, raw email message left alone and always available. Extraction of attachments could still be done, but leave the original. This allows preexisting email to be reparsed to leverage newer capabilities of the MUA, or by other external systems (e.g. digital signature verification). A problem that exists is the definition of attachment; it is not clearly defined in email. Content-Disposition is supposed to indicate if something is an attachment, but attachment could be inferred if the MUA is unable to rendering the entity directly (at a given moment of time). So, would it be required to extract each MIME entity to a separate file, regardless of disposition since one cannot predict how the user may want to access them at the file-level? It would cause a lot of duplication of data. --ewh ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
On Feb 19, 2014, at 11:44 AM, Paul Vixie p...@redbarn.org wrote: third and finally, i'd fully support the IMAP UID psychosis, by offering persistent-for-the-life-of-the-message numeric identifiers, For-the-life-of-the-message where? Within a folder? Within ~/Mail? If you want a truly unique identifier for a message, use a cryptographic hash of the message file. (The original message, before MH starts mucking it up with things like annotations.) This even works with Norm's hypothetical distributed data model. This is one of the reasons why I would stick with the 'one message per file' paradigm, rather than the 'message per directory' scheme you outlined. It's also one of the benefits of UTF-8 everywhere. Remove all the MIME encodings and convert to a common character set and you have a single common representation of the message that's independent of the originator's composition environment and the intervening message transports. --lyndon signature.asc Description: Message signed with OpenPGP using GPGMail ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
i would. first, i'd adopt the messages-as-directories model norm gave, since it supports MIME, where our current model does not. by support i mean i want to use normal UNIX file access to work with my message store. there is no file-level access to attachments right now, which means i have to run mhstore to turn an opaque black-box attachment into a UNIX file i can access. mhstore is highly impure by MH's overall design principles, even though i'm grateful to have it under the circumstances. See, this is where we run into two competing ideas as to what MH's original innovation is. Do people think it's: 1) The message store (1 file per message), instead of one file for the whole store. 2) Individual commands to perform message operations, as opposed to traditional monolithic MUAs. If you think the answer is 1), then splitting all of your attachments into separate files makes perfect sense. But I believe the answer is 2). To me the power of MH is using the individual commands, which lets you do scriptable operations from the shell, and also enables the front-ends that have cropped up. It's obvious to me that the MH message store was developed because it would be near impossible to have a single-file store if every command had to rewrite it every time (well, you could probably make it work, but the performance would suck). So, you think mhstore is highly impure by the original MH design standards. I can't really argue that, but I'd point out that the original design standards had no idea of MIME; messages were text, and that was that. That assumption is all throughout the code, and MIME was grafted on later. Alright, so you want to change that. I don't see how changing the message store helps here. If your model is you want people to have to look directly in the store to copy out PDFs that people send to you ... well, that just seems like it sucks to me from a UI perspective. I guess I'd want to know what people want to happen when they show a message with some images or PDFs attached to it. Let's figure out what UI people have in their minds and work from there. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Ken Hornstein writes: i would. first, i'd adopt the messages-as-directories model norm gave, since it supports MIME, where our current model does not. by support i mean i want to use normal UNIX file access to work with my message store. there is no file-level access to attachments right now, which means i have to run mhstore to turn an opaque black-box attachment into a UNIX file i can access. mhstore is highly impure by MH's overall design principles, even though i'm grateful to have it under the circumstances. See, this is where we run into two competing ideas as to what MH's original innovation is. Do people think it's: 1) The message store (1 file per message), instead of one file for the whole store. 2) Individual commands to perform message operations, as opposed to traditional monolithic MUAs. If you think the answer is 1), then splitting all of your attachments into separate files makes perfect sense. But I believe the answer is 2). To me the power of MH is using the individual commands, which lets you do scriptable operations from the shell, and also enables the front-ends that have cropped up. It's obvious to me that the MH message store was developed because it would be near impossible to have a single-file store if every command had to rewrite it every time (well, you could probably make it work, but the performance would suck). So, you think mhstore is highly impure by the original MH design standards. I can't really argue that, but I'd point out that the original design standards had no idea of MIME; messages were text, and that was that. That assumption is all throughout the code, and MIME was grafted on later. Alright, so you want to change that. I don't see how changing the message store helps here. If your model is you want people to have to look directly in the store to copy out PDFs that people send to you ... well, that just seems like it sucks to me from a UI perspective. I guess I'd want to know what people want to happen when they show a message with some images or PDFs attached to it. Let's figure out what UI people have in their minds and work from there. --Ken I've been trying to ignore this but you got me. I'm at 1.5 on the Ken scale. #2 is a big win. But so is #1 because it allows new commands to be easily constructed using other elements of the *NIX toolkit. Just #2 means making new tools that don't work well with others. I believe that the majority of what MH needs is improvements to the UI to support better handling of MIME. The rest of it is good enough for me. Jon ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Earl Hood wrote: On Wed, Feb 19, 2014 at 1:44 PM, Paul Vixie wrote: i would. first, i'd adopt the messages-as-directories model norm gave, since it supports MIME, where our current model does not. by support i mean i want to use normal UNIX file access to work with my message store. there is no file-level access to attachments right now, which means i have to run mhstore to turn an opaque black-box attachment into a UNIX file i can access. mhstore is highly impure by MH's overall design principles, even though i'm grateful to have it under the circumstances. I have mixed feelings about this. Personally, I want the original, raw email message left alone and always available. Extraction of attachments could still be done, but leave the original. This allows preexisting email to be reparsed to leverage newer capabilities of the MUA, or by other external systems (e.g. digital signature verification). i can absolutely live with this. i love original messages. in fact i hate the 'anno' command. in today's world there is no reason not to keep both the full original message and also a directory containing that message and its attachments, with all the quoted-printable crud gone. and lest i be misunderstood -- i love mhstore, i just don't like mhstore very much. vixie ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
I feel about that like I do about Paul's rearchitecturing of the nmh library: I think that's a great idea, my only concern is who, exactly, is going to write that code? I don't have enough time on my plate to finish the relatively straightforward things like RFC 2231 support; that's a huge rearchitecturing. I don't want to discourage anyone from tackling a tough problem; please, if you've got the time, go for it! Might one use mailutils as a starting point? Didn't it come up in a recent discussion that it was effectively a MH library with interface utilities? Not necessarily the same ones we're looking for, but the core is possibly a better starting point than scratch if one has no moral objections to the GPL. Purely speculation on my part... Although it looks like they've got an incomplete project to rearchitect things themselves http://mailutils.org/wiki/Mailutils-3 (The download server actually has 2.99.98 from March 2013, vs. the 2011 date of the wiki, and git shows recent activity by Sergey) ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Might one use mailutils as a starting point? Didn't it come up in a recent discussion that it was effectively a MH library with interface utilities? Not necessarily the same ones we're looking for, but the core is possibly a better starting point than scratch if one has no moral objections to the GPL. You (or anyone else) are free to do that. Me? I lack the energy (I have looked at it). --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Might one use mailutils as a starting point? Didn't it come up in a recent discussion that it was effectively a MH library with interface utilities? Not necessarily the same ones we're looking for, but the core is possibly a better starting point than scratch if one has no moral objections to the GPL. You (or anyone else) are free to do that. Me? I lack the energy (I have looked at it). I understand, was just tossing the idea out there for someone who was considering pitching in, to make the project seem more achievable. ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
Though that could be replaced by a mhgrep that extracts messages from whatever form they're in and feeds them to grep (and caches due to likely locality of reference). 'pick' exists, although it doesn't support everything that grep does, but it's pretty close. Analogous to what less(1) does with compressed files: it just works. With that, I could live with an IMAP, database, filesystem or other reasonable interface to whatever is used for actual storage. It'd be nice if nmh supported any such interface if provided with a suitable adapter. Could we get there from here? Probably but it would be painful, due to not being designed to support that kind of extension as well as having to support a storied legacy. I'd advocate starting over from scratch, which I think gets back to Norm's point. You know, I've looked at the current API with an eye toward multiple backends. It's actually not terrible. Here's a typical sequence of things a program does (I'm using show(1) as an example) and omitting a few steps)): maildir = m_maildir(folder) chdir(maildir) folder_read(folder) m_convert(mp, msgnumbers) iterate over selected messages in mp, by using m_name() on each one seq_setunseen(mp) context_save() The big things that stand out here are chdir() and directly opening each message (you basically do an open() on the output of m_name()). Other than those two things, you have a reasonable start at an abstract API. Define a syntax where folder_read() (or a replacement) takes something that refers to an IMAP mailbox, and you're most of the way there. That's just the user program API, of course ... the backends would require a lot of work. --Ken ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
Re: [Nmh-workers] A useless but interesting exercise: Design MH from scratch in the 2014 context
On Feb 19, 2014, at 9:37 PM, Paul Vixie p...@redbarn.org wrote: i have a moral objection to the GPL. I have a double-barreled shotgun for the GPL. No, wait, against the GPL! signature.asc Description: Message signed with OpenPGP using GPGMail ___ Nmh-workers mailing list Nmh-workers@nongnu.org https://lists.nongnu.org/mailman/listinfo/nmh-workers
