ANNOUNCE: muchsync 7 released
I've just released version 7 of muchsync, a tool for pairwise synchronization of maildirs and notmuch tags across machines. There aren't any new features. However, I discovered that some of the sqlite queries were generating full table scans rather than simple lookups. The new version should be an order of magnitude faster when deleting large numbers of mail messages or reorganizing your mail directory structure. You can download muchsync from the usual place: https://www.muchsync.org/ Enjoy, David ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
What do people use for calendar invites?
Calendar invites and the text/calendar mime type seem to be getting increasingly important. I asked about this five years ago and didn't get a good response, so I apologize for the repeat question, but I'm wondering if anything has changed since then. If you have found a good solution for integrating your calendar with notmuchmail, would you mind sharing what you are doing? Some specific questions: * What's a good way to generate text/calendar attachments from within notmuch (especially the emacs interface, which I use)? * What's a good way to apply text/calendar attachments to a calendar from within notmuch? * For those of us who hate gmail and love notmuch, but are still stuck using and hating google calendar, what alternative solutions should we be considering? Thanks, David ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Problems with unicode characters under emacs and Xorg
dm-list-email-notm...@scs.stanford.edu writes: > I just installed the ttf-symbola package from AUR and ran fc-cache (not > sure if necessary). Now the problem is completely gone. Not only that, > but I even get the little memo symbol instead of a box with the hex code > point number. > > Thank you so much! This was driving me nuts for months. Sadly, I spoke too soon. This does fix the particular problem that I posted, and makes the situation better, but I'm still getting occasional lockups, presumably because symbola does not cover every possible symbol. For example, I had another email containing U+8BDD (CJK UNIFIED IDEOGRAPH-8BDD), and this one still caused my emacs to spin for many minutes, before displaying a box with 8BDD in it. So unfortunately the problem seems to be that any character not supported by an installed font takes about 2-5 minutes of CPU time to resolve. And of course most characters that I'd use are in installed fonts, except that I can control what's in the emails I receive, so this makes notmuch very painful to use. But if no one else is running into the problems, then I may be able to get around it by installing whatever fonts other people have installed. Thanks, David ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Problems with unicode characters under emacs and Xorg
I usually use notmuch in emacs under X windows on arch linux. Recently, I've had a problem where some screens in notmuch take several minutes of 100% CPU time to load. For example, I'll just open a search, and emacs will completely lock up (even Ctrl-G doesn't do anything) for 3 minutes while my fan spins and my laptop battery drains significantly. This appears to be related to the display of certain unicode characters in email--particularly if they are in the email subject, because then the whole search screen will freeze. So far, the only workaround I've found is to kill -15 emacs, start it again in an xterm or urxvt with "emacs -nw", delete or archive the offending message, and then restart the Xorg emacs. This is quite painful particularly since it's not always obvious which email message is causing the problem. Has anyone else experienced this problem? Is there any way to workaround the problem by, for instance, defaulting to unibyte mode for notmuch buffers? I do use unicode for other languages, but I guess wouldn't mind having to type "M-x toggle-enable-multibyte-characters" to get them if as a result my emacs never locked up. It's likely that this is an emacs-wide problem, but since whatever these characters are only show up in email, I'm hoping there are people on this list who know how to solve the problem or have better workarounds. Thanks, David ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: How do you synchronize your notmuch tags across multiple machines?
Jeff Templon writes: > quote: "mbsync maintains a mapping of remote (col 1) to local (col 2) uids. > when you migrate, you can just duplicate the columns." > > He's talking about migrating a mail store from offlineimap to mbsync, I > guess the issues would be the same. I wonder if the thing to do would be to write an mbsync-specific utility to convert IMAP uids to notmuch properties and back. Then maybe I could extend muchsync to sync properties (which it currently ignores because I wrote muchsync before properties existed, and still don't really know what properties are good for). A problem with syncing properties, though, is that I don't know how to do conflict resolution correctly 100% of the time. One of the key insights behind muchsync is that with file names and binary flags, I think I figured out a sane way of resolving all conflicts without involving the user (namely take the max number of links in each directory with special treatment for new/ and cur/, then logically and the muchsync.and_tags, and logically or the other tags). With properties, it's not clear how to sync them generically. Moreover, some properties like session-key probably should not be synced, since you probably don't want to upload the equivalent of plaintext main messages to your mail server. David ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
Dan Čermák writes: >> The ideal solution would be to implement an imap server on top of >> libnotmuch. If we had that, then you could just use offlineimap and >> isync through the imap (as opposed to file system) interface, and >> everything will just work. > > Stupid question: how would the tags be synced with your local machines > in this case? Via muchsync or would one keep the tags on the > notmuch-imap server? However you want. If you run a local imap server and a remote imap server, then you can just sync between the two imap servers with any imap synching tool. But if the two hosts happen to be running imap servers implemented on top of libnotmuch, then you could use muchsync which would be faster. > How about a hacky and not ideal solution: "teach" notmuch to not only > synchronize the read and deleted tags with the maildir, but all tags? Unfortunately, there's no standard for how to encode these. Also, there's a pretty fundamental design decision in notmuch that it doesn't edit the mail messages. Now some imap servers, like dovecot, define other flags on a per-maildir basis, but it only works up to a certain number of flags, and you'd have to parse an extra file that tells you what the mappings are. David ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: How do you synchronize your notmuch tags across multiple machines?
Dan Čermák writes: > Could you maybe elaborate in more detail, as I don't know how it would > help me exactly? Are you running a server for syncing your tags? > > My problem is the following: I have 3 machines E, B and C of which none > is always up and running. I would like to be able to sync my email on > any of the machines with offlineimap (but always only with one syncing) > and to transfer what's missing to the others, once I switch the > currently used machine. The problem is that muchsync doesn't know about imap or offlineimap. Muchsync will happily ensure that all of your notmuch maildirs and tags are synchronized. But if you run offlineimap on E and then sync E to B, B will already have a copy of all new messages. If you run offlineimap on B then, it will download additional copies of the messages already synchronized over muchsync. One solution would be to archive all the messages downloaded with offlineimap as you download them. But then you have to remember to run muchsync to upload them to other machines, or you risk losing track of messages if you download them to a machine that you then immediately turn off. I'm open to other ideas if they require some kind of new feature from muchsync, but I'm worried that this would require delving into the guts of offlineimap (and then wouldn't work with other solutions like isync). The ideal solution would be to implement an imap server on top of libnotmuch. If we had that, then you could just use offlineimap and isync through the imap (as opposed to file system) interface, and everything will just work. That would also have the benefit of making notmuch work really well with phones--you could use the gmail app on your phone and the emacs/vim front-end on your desktop. It's even conceivable that such an imap server could use notmuch's indexing to support the gmail imap extensions: https://developers.google.com/gmail/imap/imap-extensions Unfortunately, implementing an imap server is a bit beyond the scope of muchsync and not something I have time for right now. David ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
ANNOUNCE: muchsync 3 released
Muchsync 3 is now available from the usual place: http://www.muchsync.org/ Muchsync synchronizes mail and tags in your notmuch database across machines. Not much has changed since the last release, because the tool already worked well. However, there are two changes people requested: * Muchsync only buffers 128MiB of data, rather than an unbounded amount. People synchronizing large databases on machines without enough swap space were running out of memory. A 128MiB buffer should still be plenty to saturate the network. * There's a new option --newid to change the local replica id. This potentially allows a faster way of initializing muchsync: Copying your entire mail database and .notmuch-config file from the original machine, then running "muchsync --newid" on the destination machine to give the new replica a unique identity. Enjoy. David ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Breaking a really long thread
Mark Walters writes: > By default the reference header is hidden. It is controlled by > message-hidden-headers which you can customize. (Note notmuch adds > user-agent to this list via notmuch-mua-hidden-header.) Thanks for explaining this! So I posted about one problem, and instead got a solution to a problem I didn't even realize I had. Adding: (setq message-hidden-headers (delete "^References:" message-hidden-headers)) to my eval-after-loaded notmuch-config.el file solved this problem cold. No more unintentional References: headers for me. Arguably, I would say either both the In-Reply-To and the References header should be hidden or neither. Otherwise, what was happening is that I was deleting the In-Reply-To header as it was the only one I saw, and figuring that maybe References was adjusted after the fact based on In-Reply-To. After all, the message buffer doesn't keep track of the parent message. Unless there's a reason that someone would want to alter In-Reply-To without altering References, it doesn't make sense to show one without the other. David ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Breaking a really long thread
David Bremner writes: > David Mazieres writes: > >> Is there any way to break an existing thread (so as to start over with a >> smaller thread), or otherwise to tweak the threading rules so that a >> particular References header gets ignored. > > Currently there is no way to do this, as threads are "stateless" > i.e. created on the fly by _notmuch_create_thread based only on > immutable mail data. Thanks. >> It's annoyingly slow to open >> a thread with 10,000 messages just to read one SMS. I'm almost tempted >> to mangle the messages on delivery and remove the References header >> before notmuch sees them, but it would be nice to have a cleaner >> solution, as there are other situations in which one might want to >> "reset" a really long thread. > > Like this thread ;). Oops, sorry for the irrelevant thread inclusion. I guess emacs adds the References header after a message is sent is sent? In my setup, the easiest way to post to a mailing list is to reply to an existing message (since I subscribe to each list under a different email address). I tried to start a new thread by deleting the In-Reply-To and header which was all I saw, but I guess the References header got inserted later... David ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Breaking a really long thread
For some reason, Google voice always includes a header of the form "References: <+1...@txt.google.com>" (where the 5s are my actual phone number) when I get a text. The result is that every SMS I've ever received is in a single thread containing many thousands of messages. And of course huge threads are annoyingly slow to open, particularly to read a single 140-character message. Is there any way to break an existing thread (so as to start over with a smaller thread), or otherwise to tweak the threading rules so that a particular References header gets ignored. It's annoyingly slow to open a thread with 10,000 messages just to read one SMS. I'm almost tempted to mangle the messages on delivery and remove the References header before notmuch sees them, but it would be nice to have a cleaner solution, as there are other situations in which one might want to "reset" a really long thread. Thanks, David ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: muchsync files renames
Amadeusz Żołnowski writes: > Not necessarily. The recommended setup of notmuch for afew is that > "notmuch new" tags messages with "new" tag only. Then afew processes all > messages with "new" tag. So if it is a spam, then it gets "new" removed > and "spam" added. A spam message at any time doesn't have "unread" tag > assigned which should explain this behaviour. So the problem is to be > fixed on the afew side. Let's just make sure I understand: Your mail starts out like this: Path: spam/new/nnn.MnnnPnnnQnRn.machine Tags: new Then you run afew, and afew runs notmuch tag -new +spam You are saying that that even though maildir.synchronize_tags is true, you end up with: Path: spam/new/nnn.MnnnPnnnQnRn.machine Tags: spam That's a little surprising, because the next time you run "notmuch new," I would have expected it to add the unread flag based on the pathname. But, I suppose it might make sense for notmuch to special-case that flag. In other words, if notmuch new finds a file called: spam/new/nnn.MnnnPnnnQnRn.machine:2, Then it will add the unread tag to the Xapian database. But maybe if it finds a file in the new folder it doesn't add the unread flag. But why does notmuch_message_tags_to_maildir_flag() then feel the need to rename the file when muchsync calls it. Muchsync should ideally behave exactly the same as the notmuch tag command. Specifically, when muchsync receives a new file from the server, it does the following: 1. create file in same directory as the server (presumably spam/new) 2. Call the following functions on this file: notmuch_database_add_message() notmuch_message_freeze() notmuch_message_remove_all_tags() notmuch_message_add_tag() for each tag in new.tags if (synchronize_tags) notmuch_message_tags_to_maildir_flag() notmuch_message_thaw() 3. get the current tags of the message from the server (presumably just spam) 4. Call the following functions on the Message-ID: notmuch_message_freeze() notmuch_message_remove_all_tags() notmuch_message_add_tag() for each tag sent *by the server* if (synchronize_tags) notmuch_message_tags_to_maildir_flag() notmuch_message_thaw() So what I'm wondering is how this is any different from what is already happening on the server. "notmuch new" should be doing what muchsync does in step 2, and afew (via "notmuch tag") should be doing what muchsync does in step 4. Yet somehow you are saying that on the server the file stays in spam/new/, while on the client muchsync's actions cause the file to move to spam/cur/? So that means there's still something I don't really understand--possibly the series of notmuch library calls happening server side (which I should then maybe emulate client side). None of this is super serious, beyond a one-time extra cost, but I like to understand things thoroughly, particularly when writing software that manipulates critical data like mail... It there any possibility that new.tags has a different setting on your client and server machines? Thanks, David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: muchsync files renames
So just to follow up a bit. I looked into things a bit further, and here is what I found with maildir.synchronize_flags set to true. Initially, when you run "muchsync --init", it copies all the files to your maildir, and for each file invokes notmuch_message_tags_to_maildir_flag. That changes the name of the file, with the result that the sql database and the actual mail directory end up out of sync. That on it's own is not a big deal, but it means that the next time muchsync, muchsync will have to rescan all of the files, as their names are no longer correct. That shouldn't cause any extra traffic between the two machines, but it will require time on the client. That is likely the source of the delay you were seeing. However, if you C-c the client during this process, I still don't see any problems arising that cause more links to be transferred between machines. So I'm kind of stumped about that part. Maybe muchsync should create the original file name based on tags, so as to avoid having to rescan in the common case. I wish there were some way of getting the renamed file after notmuch_message_tags_to_maildir_flags. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: muchsync files renames
Amadeusz Żołnowski writes: > Hi David, > > Fist of all thank you for such elaborate answer. > > I have missed the paragraph about maildir.synchronize_flags somehow. I > have it enabled. So this must be source of a problem (?). I've only ever tested with mailder.synchronize_flags = true, because I'm worried that if I disable it I will have a hard time re-enabling it. I do think it is a source of inefficiency, but muchsync should still be usable. > Here follows steps I followed: > > 1. I initialized server locally with muchsync -vv. My mail is stored in > ~/Mail on the server. > 2. I run muchsync --init ~/mail SERVER. (Directory names do not need to > be the same, do they?) Confirmed that directory names do not need to be the same. > 3. I run muchsync SERVER. > 4. When it lasted much longer then initialization I canceled it by > single SIGINT (^c). Interesting. I wish I knew why this was taking much longer than running it on the server, and whether the delay was caused by client activity or server activity. I don't suppose you'd be willing to make a copy of your mail database to repeat the experiment without any risk of messing up your real maildir? Basically what would be interesting to see is (assuming .notmuch-copy on server is configured for this disposable copy): # Log everything for later analysis script # Use new config file location for disposable copy export NOTMUCH_CONFIG=$HOME/.notmuch-copy # Set up a new initial database muchsync - --init ~/test-copy SERVER - --config=.notmuch-copy # Now initial copy is made, see if client is slow # Is notmuch new itself slow? notmuch new # Is resynchronizing muchsync with notmuch slow? muchsync - # Now see if something weird happened on server to make notmuch new slow ssh SERVER notmuch --config=.notmuch-copy new # Now see if something weird happened on server to make muchsync slow ssh SERVER muchsync - --config=.notmuch-copy # Now finally try resynchronizing to see if this is slow muchsync - SERVER - --config=.notmuch-copy # Save script file exit Does something along these lines make sense? If you can figure out which of these is slow (other than --init, which always will be), and what the verbose chatter is, that would help me narrow down and identify any problem. > 5. I rerun muchsync SERVER and then it notified me that notmuch > identified files names changes - more than 1000. Were the link changes on the client (sent) or the server (received) side? > 6. I waited a bit and then I canceled it by SIGINT. > 7. I run muchsync --noup SERVER. This took only seconds to finish. So this suggests the issue is on the client side. It sounds like a bug. I wonder if I we can just reproduce this using a public email corpus, so we don't have to worry about people's private email. > I suspected that muchsync at step 3 and 5 tried to push files renames > back to server. But now I am not sure what was going on. Have I > desynchronized file mail flags? It's hard to say if anything has broken > for me, but I am a bit worried anyway. > > If I just disable maildir.synchronize_flags and rerun muchsync, will > everything get synchronized properly? I don't think that will change things. maildir.synchronized_flags will make things slower, but it shouldn't affect the SUMMARY numbers you see at the end of muchsync, other than maybe files moving from .../new to .../cur. But presumably most of your mail files were already in cur directories. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Resending the same email
Sebastian Fischmeister writes: > Hi, > > My previous mail editor had a useful feature to resend already sent > emails. It's essentially opening an already sent email and have the > senders, subject, and body pre-filled as well as all attachments > attached. > > Is this easy to achieve in notmuch? The attachments seem a bit tricky. I often pipe the message to sendmail, using emacs's notmuch-show-pipe-message command (bound to '|' by default). However I agree it would be much nicer to be able to edit the headers slightly, so as to add headers like Resent-To. On a related note, another feature I really miss is the ability to edit and re-send a bounced message. (I realize this could be problematic for notmuch, in that you really want to re-send it with the same message-ID.) David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: muchsync files renames
Amadeusz Żołnowski writes: > Hi, > > I am testing muchsync-2 and it looks to me that files names across > machines are different. Moreover when syncing again after > initialization it seems muchsync is working on something. I have > canceled this and rerun muchsync. notmuch reported lots of files > renames on server. What and why it happens? What muchsync specifically synchronizes for messages in the mapping: (directory, SHA-1-hash, link-count) So if a directory contains two copies of a file on one machine, it will end up with two copies on the other machine. However, the file names themselves are not the same, but rather are created in accordance with the maildir spec. (Note SHA-1 wouldn't be my first choice of hash function, but notmuch already uses this for messages with long message IDs, so I figured I'd just be consistent with existing practice.) In terms of what muchsync is working on, you can run it with "-" on both sides to get an idea, as in "muchsync - server -". Better yet, you can just run it on one side with "muchsync -". You'll get a lot of output, so maybe run it inside the script command to save the output.maybe run it inside the script command to save the output. If you have enabled maildir.synchronize_flags, it could be that notmuch is initially renaming all of your files, in which case muchsync needs to re-hash them to make sure they haven't changed. How did you cancel muchsync? If you send it a single SIGINT or SIGTERM, it attempts to clean up after itself. However, upon multiple signals or other signals, it immediately exits. Muchsync is conservative about updating the database, to avoid missing tags or files that have been changed. It always updates the notmuch database first, then its own sqlite database with a version number. That means if you kill muchsync, some number of files may get picked up as changed again even though really they were just copied from a peer. To mitigate this problem, the muchsync client syncs the database every 10 seconds, so that in theory you should only get 10 seconds of extra work from killing the client. However, the server does not sync periodically, on the assumption that it is more likely to read an EOF than get killed, although currently it doesn't appear to commit any pending transactions to the sqlite database upon EOF, which may be an oversight. So to summarize: * File names are not the same across machine, only file contents and directory structure. * Give muchsync lots of "-v" options to see what it is doing. * Try to avoid killing muchsync. Doing so is safe, but likely to generate extra work in the form of phantom renames or tag changes that get synchronized even though they don't need to be. * Possibly the server should handle EOF more gracefully and commit any pending transactions, or the client should periodically send a commit command to the server. If you think something is wrong, I can help you figure it out, but I need to know what maildir.synchronize_flags is set to on each replica, what you mean by "canceled", and roughly what was happening when you canceled (uploading or downloading). David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
ANNOUNCE: muchsync 2 released
I'm pleased to announce the release of muchsync version 2. Muchsync is a tool for replicating and synchronizing your notmuch databases across machines. The new version is reported to build out of the box on Mac OS X. There's one new feature, which is a new notmuch config option, "muchsync.and_tags". When a message experiences an update conflict, muchsync deletes any tag in muchsync.and_tags unless that tag is present in both replicas (a logical and). For all other tags, muchsync adds the tag if it exists on either replica (a logical or). The default for muchsync.and_tags is to use the value of new.tags, as in previous versions of muchsync. As usual, the distribution is available from the muchsync home page: http://www.muchsync.org/ Enjoy. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Enabling and disabling maildir.synchronize_flags
David Bremner writes: > dm-list-email-notm...@scs.stanford.edu writes: > >> It seems that disabling it should simply be safe. But re-enabling, one >> risks losing tags, as the next notmuch new will cause old maildir flags >> to override the xapian database. So that suggests something like: >> >>notmuch dump > backup >>notmuch config set maildir.synchronize_flags false >># Do I need to run notmuch new here? >>notmuch restore < backup > > Hi David; > > Sorry about the long delay. I'm not sure I follow the connection between > your paragraph above and the the example. The example seems safe, but as > you say, disabling synching should bs safe anyway. It's not an example, it's kind of a worst case scenario if there's no easy and safe way way to enable synchronize_flags. I want to try disabling flags, but if I change my mind and the only way to get it back is to do a full notmuch dump/restore, that's a pretty hefty penalty. Also, even if I do a full notmuch dump / restore, I'm not sure if the notmuch new is necessary in the middle. So my question remains, what's the easiest safe way to re-enable synchronize_flags after disabling it? (Safe meaning it won't change any tags.) It could be that there's a very simple answer, in which case sticking it in the man page might be nice. Thanks, David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Enabling and disabling maildir.synchronize_flags
David Bremner writes: > dm-list-email-notmuch at scs.stanford.edu writes: > >> It seems that disabling it should simply be safe. But re-enabling, one >> risks losing tags, as the next notmuch new will cause old maildir flags >> to override the xapian database. So that suggests something like: >> >>notmuch dump > backup >>notmuch config set maildir.synchronize_flags false >># Do I need to run notmuch new here? >>notmuch restore < backup > > Hi David; > > Sorry about the long delay. I'm not sure I follow the connection between > your paragraph above and the the example. The example seems safe, but as > you say, disabling synching should bs safe anyway. It's not an example, it's kind of a worst case scenario if there's no easy and safe way way to enable synchronize_flags. I want to try disabling flags, but if I change my mind and the only way to get it back is to do a full notmuch dump/restore, that's a pretty hefty penalty. Also, even if I do a full notmuch dump / restore, I'm not sure if the notmuch new is necessary in the middle. So my question remains, what's the easiest safe way to re-enable synchronize_flags after disabling it? (Safe meaning it won't change any tags.) It could be that there's a very simple answer, in which case sticking it in the man page might be nice. Thanks, David
Re: notmuch-lazysync -- synchronizing tags using dropbox
Daniel Schoepe writes: > The way tag changes are logged is a bit of a hack, but it could be > improved in the future by adding a post-tag hook to notmuch. One thing to look into, if you are thinking of a better logging mechanism, is that Xapian itself has a change logging mechanism for replicating databases (http://xapian.org/docs/replication.html). I do think it would be cleaner to do this in a way that is integrated with notmuch, but I think the best way to do this is to integrate a "modtime" value into the Xapian database. Having a modtime for each record would not only allow incremental transfers (just record the highest timestamp sent to each replica), it would also solve this terrible problem that in emacs you can end up tagging messages you don't see (because you apply a tag to the query result, when new mail has come in--which would be solved by tagging only through the higest modtime actually displayed). When you have one mechanism (modtime) that solves multiple problems, it is likely the right thing to use... David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Modify message after send...?
mailingli...@nawaz.org writes: > Hi, > > I use notmuch via Emacs. > > Here's what I want: > > When I hit C-c C-c to send a message, I'd like it to be passed to a > script (likely a Python one, although I may consider an Elisp function > if an external script is not possible) for modification of headers, > before it is sent to the MTA (Postfix in my case). A bonus would be to > have the modified message stored in the FCC location, instead of the > original one. > > Is this possible? An alternative may be to modify the message /before/ > it goes to message-send-and-exit. I'm inexperienced in Elisp - would > this be via what's called "advising"? > > BTW, all I really want to do is modify the From: field based on the > recipients (for every email, with no default From). I'll welcome > suggestions for existing ways to do that. I Googled a little, but didn't > find a clear good solution. Furthermore, I expect over time the rules by > which I pick the From: field will get more complex than my knowledge of > Elisp. For modifying the From field, I recommend doing it before you send the message, as this gives you an opportunity to see what you are about to send and possibly edit it by hand. I do something like this using defadvice around notmuch-mua-mail, do adjust things about my messages. I also put a defadvice around notmuch-call-notmuch-sexp to filter the To and Cc headers I get from "notmuch reply" (because notmuch won't let me put wildcards in my .notmuch-config file). Finally, and possibly most relevant to you, I use a message-send-hook to insert the Return-Path address and make a few more modifications before sending a message. Note that I only do this if there isn't already a Return-Path header. That way, if I type C-c C-c and don't like what I see, I edit the result and send again, leaving the generated Return-Path header, and this time it goes through unmodified. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Modify message after send...?
mailinglists at nawaz.org writes: > Hi, > > I use notmuch via Emacs. > > Here's what I want: > > When I hit C-c C-c to send a message, I'd like it to be passed to a > script (likely a Python one, although I may consider an Elisp function > if an external script is not possible) for modification of headers, > before it is sent to the MTA (Postfix in my case). A bonus would be to > have the modified message stored in the FCC location, instead of the > original one. > > Is this possible? An alternative may be to modify the message /before/ > it goes to message-send-and-exit. I'm inexperienced in Elisp - would > this be via what's called "advising"? > > BTW, all I really want to do is modify the From: field based on the > recipients (for every email, with no default From). I'll welcome > suggestions for existing ways to do that. I Googled a little, but didn't > find a clear good solution. Furthermore, I expect over time the rules by > which I pick the From: field will get more complex than my knowledge of > Elisp. For modifying the From field, I recommend doing it before you send the message, as this gives you an opportunity to see what you are about to send and possibly edit it by hand. I do something like this using defadvice around notmuch-mua-mail, do adjust things about my messages. I also put a defadvice around notmuch-call-notmuch-sexp to filter the To and Cc headers I get from "notmuch reply" (because notmuch won't let me put wildcards in my .notmuch-config file). Finally, and possibly most relevant to you, I use a message-send-hook to insert the Return-Path address and make a few more modifications before sending a message. Note that I only do this if there isn't already a Return-Path header. That way, if I type C-c C-c and don't like what I see, I edit the result and send again, leaving the generated Return-Path header, and this time it goes through unmodified. David
notmuch-lazysync -- synchronizing tags using dropbox
Daniel Schoepe writes: > The way tag changes are logged is a bit of a hack, but it could be > improved in the future by adding a post-tag hook to notmuch. One thing to look into, if you are thinking of a better logging mechanism, is that Xapian itself has a change logging mechanism for replicating databases (http://xapian.org/docs/replication.html). I do think it would be cleaner to do this in a way that is integrated with notmuch, but I think the best way to do this is to integrate a "modtime" value into the Xapian database. Having a modtime for each record would not only allow incremental transfers (just record the highest timestamp sent to each replica), it would also solve this terrible problem that in emacs you can end up tagging messages you don't see (because you apply a tag to the query result, when new mail has come in--which would be solved by tagging only through the higest modtime actually displayed). When you have one mechanism (modtime) that solves multiple problems, it is likely the right thing to use... David
Re: ANNOUNCE: muchsync 1 - share notmuch DB across machines
Suvayu Ali writes: > I noticed that the configure script didn't notice notmuch-devel was not > installed on my system. I noticed it when the compilation failed. Bug? What OS and distribution are you using? notmuch-devel sounds like something a linux distribution might do, rather than anything that's part of notmuch itself. On arch linux, for example, there's just a notmuch package that includes notmuch. Obviously there's no way to compile muchsync without notmuch installed. But if you elaborate on what it is you saw and what you would like to see I might be able to address it in the next release. > Besides that, I had a question. I would like to synchronize just the > tags, not the maildirs, I want to use OfflineIMAP for that. Is that > possible? Not really, no. You could run offlineimap followed by muchsync, but you run the risk that mail will be delivered in the meantime. You could also run offlineimap to fetch your mail in the first place, and then muchsync to sync it between your devices. Can I ask why you would want to do this, anyway? Muchsync should be faster than notmuch, particularly if you have a lot of mail directories and/or are going to pay the cost of tag synchronization anyway. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
ANNOUNCE: muchsync 1 - share notmuch DB across machines
Suvayu Ali writes: > I noticed that the configure script didn't notice notmuch-devel was not > installed on my system. I noticed it when the compilation failed. Bug? What OS and distribution are you using? notmuch-devel sounds like something a linux distribution might do, rather than anything that's part of notmuch itself. On arch linux, for example, there's just a notmuch package that includes notmuch. Obviously there's no way to compile muchsync without notmuch installed. But if you elaborate on what it is you saw and what you would like to see I might be able to address it in the next release. > Besides that, I had a question. I would like to synchronize just the > tags, not the maildirs, I want to use OfflineIMAP for that. Is that > possible? Not really, no. You could run offlineimap followed by muchsync, but you run the risk that mail will be delivered in the meantime. You could also run offlineimap to fetch your mail in the first place, and then muchsync to sync it between your devices. Can I ask why you would want to do this, anyway? Muchsync should be faster than notmuch, particularly if you have a lot of mail directories and/or are going to pay the cost of tag synchronization anyway. David
ANNOUNCE: muchsync 1 - share notmuch DB across machines
I've just released a new version of muchsync, the tool for replicating your notmuch database on multiple machines. The source release is available at: http://www.muchsync.org/src/muchsync-1.tar.gz The new version fixes a bug that incorrectly synchronized tags containing '%' characters. It also fixes several build problems people reported. Hence, the new version should work out of the box on more operating systems. More information about muchsync is available on the project home page: http://www.muchsync.org/ Enjoy, David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
ANNOUNCE: muchsync 1 - share notmuch DB across machines
I've just released a new version of muchsync, the tool for replicating your notmuch database on multiple machines. The source release is available at: http://www.muchsync.org/src/muchsync-1.tar.gz The new version fixes a bug that incorrectly synchronized tags containing '%' characters. It also fixes several build problems people reported. Hence, the new version should work out of the box on more operating systems. More information about muchsync is available on the project home page: http://www.muchsync.org/ Enjoy, David
Re: How do I prevent notmuch-show in Emacs from automatically following links in emails?
Philip writes: > Hi, > > When I open emails in the notmuch-show mode on Emacs, it automatically > follows links which are present in the email. For instance, it > connects to various tracking objects included in the email. How can I > prevent it from doing this automatically all the time? Is there a > setting by which notmuch-show will not follow links in emails? If you just want something quick and dirty, put the following in your .emacs file: (setq gnus-inhibit-images t) I do that, and then in the very rare occasions where I want to see all the images, I just open the message in my browser by typing ". v" on the html part button. I'm sure the patch Tomi linked to is much better, but this works with unmodified notmuch. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
How do I prevent notmuch-show in Emacs from automatically following links in emails?
Philip writes: > Hi, > > When I open emails in the notmuch-show mode on Emacs, it automatically > follows links which are present in the email. For instance, it > connects to various tracking objects included in the email. How can I > prevent it from doing this automatically all the time? Is there a > setting by which notmuch-show will not follow links in emails? If you just want something quick and dirty, put the following in your .emacs file: (setq gnus-inhibit-images t) I do that, and then in the very rare occasions where I want to see all the images, I just open the message in my browser by typing ". v" on the html part button. I'm sure the patch Tomi linked to is much better, but this works with unmodified notmuch. David
ANNOUNCE: muchsync 0 - share notmuch DB across machines
I'm pleased to announce the release of muchsync: http://www.muchsync.org/ Muchsync is a notmuch mail synchronizer. It lets you keep a full copy of your notmuch database on every one of your computers. Muchsync properly synchronizes maildir contents and notmuch tags, obeying the no lost updates rule and sensibly but conservatively resolving update conflicts with no manual intervention. After initialization (which is slow because it has to download and re-index all of your mail), muchsync is bandwidth efficient and works well over high-latency networks. It is much faster than alternatives such as offlineimap, particularly when you have lots of different maildirs. I originally had plans to rename the utility syncmuch and add tons of features. However, as soon as I ran the first tests of the system, I found that I could not go back to any other way of reading email and my test became production use. Just this base functionality has been enough to solve all of my email problems for the last 13 months. I'm therefore releasing it in the hopes that others will similarly find it useful. (And if muchsync ever takes off, there are small changes to the notmuch API we can discuss that would make it even more efficient...) David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
ANNOUNCE: muchsync 0 - share notmuch DB across machines
I'm pleased to announce the release of muchsync: http://www.muchsync.org/ Muchsync is a notmuch mail synchronizer. It lets you keep a full copy of your notmuch database on every one of your computers. Muchsync properly synchronizes maildir contents and notmuch tags, obeying the no lost updates rule and sensibly but conservatively resolving update conflicts with no manual intervention. After initialization (which is slow because it has to download and re-index all of your mail), muchsync is bandwidth efficient and works well over high-latency networks. It is much faster than alternatives such as offlineimap, particularly when you have lots of different maildirs. I originally had plans to rename the utility syncmuch and add tons of features. However, as soon as I ran the first tests of the system, I found that I could not go back to any other way of reading email and my test became production use. Just this base functionality has been enough to solve all of my email problems for the last 13 months. I'm therefore releasing it in the hopes that others will similarly find it useful. (And if muchsync ever takes off, there are small changes to the notmuch API we can discuss that would make it even more efficient...) David
Re: folder and path completely broken in HEAD?
Mark Walters writes: > Hi > > Before checking other things: have you run notmuch new? That's needed to > update the database. It is an irreversible update so notmuch-0.17 will > not work with the updated database. No, I haven't. Okay, so that must be it. Sorry for bothering people. The discussion of the NEWS was a bit confusing, so I wanted to check it out. I knew something had to be very wrong. Weill the new primitives still allow me to group my mail hierarchically in searches? The new man page is not totally clear on what is being matched. Thanks, David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
folder and path completely broken in HEAD?
Mark Walters writes: > Hi > > Before checking other things: have you run notmuch new? That's needed to > update the database. It is an irreversible update so notmuch-0.17 will > not work with the updated database. No, I haven't. Okay, so that must be it. Sorry for bothering people. The discussion of the NEWS was a bit confusing, so I wanted to check it out. I knew something had to be very wrong. Weill the new primitives still allow me to group my mail hierarchically in searches? The new man page is not totally clear on what is being matched. Thanks, David
Re: github mirror
Sam Halliday writes: > David Mazieres writes: >> The problem is that different imap servers store tags in different >> ways. Since notmuch does not use imap, it would be hard for notmuch to >> synchronize the tags, other than the standard ones (for which notmuch >> already has support). >> >> One thing you could do is build an external tool that synchronizes >> notmuch tags and spawns an imap server in preauth mode to sync the tags. >> (That would be yet another use for the ctime values we have discussed on >> this list.) > > The improvements to offlineimap to use the mail header hack might work > well for both of us. Currently the only way to add/remove "labels" (a > gmail concept) is to copy/move mail between folders. And this is how > notmuch "tags" are synced. But with outstanding pull request, this can > all be managed via email headers and that means you *only* need to > synchronise your "All Mail" folder. > > So, I'd be interested to see what your code could do in that world :-) My code assumes shell access to your mail server, so it doesn't do squat in the gmail world. I suppose you could use gmail just as your SMTP server, then download everything to your own server with offlineimap and manage it with notmuch there, in which case my code gives you the notmuch experience on all your devices. However, there's a much better SMTP server out than what google is running (www.mailavenger.org), so such a setups is backwards... The right thing to do is to receive everything on a mail server that you control, then push to gmail only what you want to read on your phone and/or disclose to the NSA (which in my case is only a tiny fraction of my email). David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: github mirror
Gaute Hope writes: >> Really what you want is an imap server built on top of the notmuch >> library. That way you could use notmuch from your desktop and then use >> imap from your phone, and everything would stay perfectly in sync. >> Implementing such a server wouldn't be that hard, but it would help if >> notmuch made the _notmuch_message_get_doc_id and >> _notmuch_directory_get_document_id functions semi-public. Then the imap >> server could just use docids as uids. (Plus then muchsync wouldn't have >> to go through gross contortions to get docids information...) > > That would be nice, but a solution where the user does not need to run > his own server is in my opinion pretty essential. You don't have to "run" an imap server, you can use it in preauth mode. Having a notmuch-based imap server would let you synchronize gmail with your laptop and then read email offline, for instance. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
github mirror
Gaute Hope writes: >> Really what you want is an imap server built on top of the notmuch >> library. That way you could use notmuch from your desktop and then use >> imap from your phone, and everything would stay perfectly in sync. >> Implementing such a server wouldn't be that hard, but it would help if >> notmuch made the _notmuch_message_get_doc_id and >> _notmuch_directory_get_document_id functions semi-public. Then the imap >> server could just use docids as uids. (Plus then muchsync wouldn't have >> to go through gross contortions to get docids information...) > > That would be nice, but a solution where the user does not need to run > his own server is in my opinion pretty essential. You don't have to "run" an imap server, you can use it in preauth mode. Having a notmuch-based imap server would let you synchronize gmail with your laptop and then read email offline, for instance. David
Re: github mirror
Austin Clements writes: > As for storing this information directly in messages, in general, the > notmuch community is opposed to modifying messages. This causes many > problems, and immutable messages are more robust and simplify so many > things. IMAP assumes messages are immutable. Maildir assumes > messages are immutable. Notmuch new would get dramatically slower if > it had to check for messages modifications. What do you do if you > change a tag and there are multiple copies of a message? What do you > do if there are multiple copies and they disagree about the tags? How > do you atomically update the tags stored in a message? From an > engineering standpoint, it's much better to avoid mutable messages. The speed penalty would be very minor in the common case. Muchsync scans directories (since it has to scan file contents) and the cost to compute SHA-1 hashes of modified files is under 50 msec or something in the common case. Extracting tags would be even cheaper. The reason is that A) you only need to scan modified directories, and B) you don't need to open the file unless the inode, mtime, or size has changed. Originally I was going to implement an optimization to detect renamed files and avoid computing SHA-1 again (for the case where maildir flags have changed), but in the end this wasn't even worth it because the cost is so small. That said, I agree that the complexity of altering files is not worth it. Especially since most imap servers will not know about this. Also, the question of what do you do with duplicate message IDs (which is effectively what you have when the tags disagree) is a more general problem still needing a solution, and would be exacerbated by embedding important information like tags in the message. Really what you want is an imap server built on top of the notmuch library. That way you could use notmuch from your desktop and then use imap from your phone, and everything would stay perfectly in sync. Implementing such a server wouldn't be that hard, but it would help if notmuch made the _notmuch_message_get_doc_id and _notmuch_directory_get_document_id functions semi-public. Then the imap server could just use docids as uids. (Plus then muchsync wouldn't have to go through gross contortions to get docids information...) David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
github mirror
Austin Clements writes: > As for storing this information directly in messages, in general, the > notmuch community is opposed to modifying messages. This causes many > problems, and immutable messages are more robust and simplify so many > things. IMAP assumes messages are immutable. Maildir assumes > messages are immutable. Notmuch new would get dramatically slower if > it had to check for messages modifications. What do you do if you > change a tag and there are multiple copies of a message? What do you > do if there are multiple copies and they disagree about the tags? How > do you atomically update the tags stored in a message? From an > engineering standpoint, it's much better to avoid mutable messages. The speed penalty would be very minor in the common case. Muchsync scans directories (since it has to scan file contents) and the cost to compute SHA-1 hashes of modified files is under 50 msec or something in the common case. Extracting tags would be even cheaper. The reason is that A) you only need to scan modified directories, and B) you don't need to open the file unless the inode, mtime, or size has changed. Originally I was going to implement an optimization to detect renamed files and avoid computing SHA-1 again (for the case where maildir flags have changed), but in the end this wasn't even worth it because the cost is so small. That said, I agree that the complexity of altering files is not worth it. Especially since most imap servers will not know about this. Also, the question of what do you do with duplicate message IDs (which is effectively what you have when the tags disagree) is a more general problem still needing a solution, and would be exacerbated by embedding important information like tags in the message. Really what you want is an imap server built on top of the notmuch library. That way you could use notmuch from your desktop and then use imap from your phone, and everything would stay perfectly in sync. Implementing such a server wouldn't be that hard, but it would help if notmuch made the _notmuch_message_get_doc_id and _notmuch_directory_get_document_id functions semi-public. Then the imap server could just use docids as uids. (Plus then muchsync wouldn't have to go through gross contortions to get docids information...) David
github mirror
Sam Halliday writes: > David Mazieres writes: >> The problem is that different imap servers store tags in different >> ways. Since notmuch does not use imap, it would be hard for notmuch to >> synchronize the tags, other than the standard ones (for which notmuch >> already has support). >> >> One thing you could do is build an external tool that synchronizes >> notmuch tags and spawns an imap server in preauth mode to sync the tags. >> (That would be yet another use for the ctime values we have discussed on >> this list.) > > The improvements to offlineimap to use the mail header hack might work > well for both of us. Currently the only way to add/remove "labels" (a > gmail concept) is to copy/move mail between folders. And this is how > notmuch "tags" are synced. But with outstanding pull request, this can > all be managed via email headers and that means you *only* need to > synchronise your "All Mail" folder. > > So, I'd be interested to see what your code could do in that world :-) My code assumes shell access to your mail server, so it doesn't do squat in the gmail world. I suppose you could use gmail just as your SMTP server, then download everything to your own server with offlineimap and manage it with notmuch there, in which case my code gives you the notmuch experience on all your devices. However, there's a much better SMTP server out than what google is running (www.mailavenger.org), so such a setups is backwards... The right thing to do is to receive everything on a mail server that you control, then push to gmail only what you want to read on your phone and/or disclose to the NSA (which in my case is only a tiny fraction of my email). David
Re: github mirror
Sam Halliday writes: > Dear NotMuch, > > But in any case, my RFE/question was this: how hard would it be to have > an optional mode of behaviour where tags are stored in the message > itself, so that syncing with an IMAP server (e.g. via offlineimap) > would make the tags available on all devices. This would negate the need > for workarounds, such as shared notmuch databases, when users have > multiple machines. The problem is that different imap servers store tags in different ways. Since notmuch does not use imap, it would be hard for notmuch to synchronize the tags, other than the standard ones (for which notmuch already has support). One thing you could do is build an external tool that synchronizes notmuch tags and spawns an imap server in preauth mode to sync the tags. (That would be yet another use for the ctime values we have discussed on this list.) > It would also allow applications like offlineimap to introduce a gmail > plugin that would copy the message into a folder according to its tags, > so gmail labels and notmuch tags would be in sync. Unfortunately, offlineimap is dog slow, especially when you have a ton of maildirs. It just doesn't seem to pipeline very well at all. Even when I run 5 or 10 threads, it still seems to take a whole bunch of round trip times. I used to want what you want. But then it got to the point that offlineimap was taking 30 seconds to sync my 60 maildirs over 3G, even when there was no new mail. Worse, one thread seems to busy-wait for the others to finish, so it often pegs the CPU and drains the battery on my laptop. As a result, I built a tool that synchronizes notmuch tags directly. Now my problem is waiting for "notmuch new" to run, but at least the synchronization part is really fast and pleasant. I'll be releasing my tool in the next couple of months once I've kicked the tires a bit more and fixed up a few things. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
github mirror
Sam Halliday writes: > Dear NotMuch, > > But in any case, my RFE/question was this: how hard would it be to have > an optional mode of behaviour where tags are stored in the message > itself, so that syncing with an IMAP server (e.g. via offlineimap) > would make the tags available on all devices. This would negate the need > for workarounds, such as shared notmuch databases, when users have > multiple machines. The problem is that different imap servers store tags in different ways. Since notmuch does not use imap, it would be hard for notmuch to synchronize the tags, other than the standard ones (for which notmuch already has support). One thing you could do is build an external tool that synchronizes notmuch tags and spawns an imap server in preauth mode to sync the tags. (That would be yet another use for the ctime values we have discussed on this list.) > It would also allow applications like offlineimap to introduce a gmail > plugin that would copy the message into a folder according to its tags, > so gmail labels and notmuch tags would be in sync. Unfortunately, offlineimap is dog slow, especially when you have a ton of maildirs. It just doesn't seem to pipeline very well at all. Even when I run 5 or 10 threads, it still seems to take a whole bunch of round trip times. I used to want what you want. But then it got to the point that offlineimap was taking 30 seconds to sync my 60 maildirs over 3G, even when there was no new mail. Worse, one thread seems to busy-wait for the others to finish, so it often pegs the CPU and drains the battery on my laptop. As a result, I built a tool that synchronizes notmuch tags directly. Now my problem is waiting for "notmuch new" to run, but at least the synchronization part is really fast and pleasant. I'll be releasing my tool in the next couple of months once I've kicked the tires a bit more and fixed up a few things. David
Re: [PATCH] Add configurable changed tag to messages that have been changed on disk
Austin Clements writes: > I'd like to have efficient change detection, too. In my case, I'd > like to use it to support efficient live search and show updates. The > design I'd sketched out for that used a log rather than ctimes, and > I'm curious if you have thoughts on the relative merits and > suitability for tag sync of these approaches. Both logging ctime are very general mechanisms than can solve many problems. For example, is there still an issue that pressing "*" in emacs notmuch-search mode can affect messages that are not visible if someone ran notmuch new in a different window? ctimes would be one way to solve this. (Conservatively exempt any messages that have changed since the displayed search was run.) > I'd been leaning toward logging because it can capture changes to > things that aren't represented as documents in the database, such as > thread membership. This probably doesn't matter for synchronization, > but it makes it much easier to figure out which threads in thread > search results need to be refreshed. A log can also capture message > deletion easily, while ctimes would require tombstones (which may be a > good idea for other reasons [1]). > > On the other hand, logging is obviously more mechanism than ctimes, > and probably requires some garbage collection story. The advantage of ctime over logging is that the interface is super simple and intuitive. How would the interface to the log work? In terms of implementation, have you thought about leveraging the XAPIAN_MAX_CHANGESETS mechanism to produce your logs? David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] Add configurable changed tag to messages that have been changed on disk
Austin Clements writes: > I'd like to have efficient change detection, too. In my case, I'd > like to use it to support efficient live search and show updates. The > design I'd sketched out for that used a log rather than ctimes, and > I'm curious if you have thoughts on the relative merits and > suitability for tag sync of these approaches. Both logging ctime are very general mechanisms than can solve many problems. For example, is there still an issue that pressing "*" in emacs notmuch-search mode can affect messages that are not visible if someone ran notmuch new in a different window? ctimes would be one way to solve this. (Conservatively exempt any messages that have changed since the displayed search was run.) > I'd been leaning toward logging because it can capture changes to > things that aren't represented as documents in the database, such as > thread membership. This probably doesn't matter for synchronization, > but it makes it much easier to figure out which threads in thread > search results need to be refreshed. A log can also capture message > deletion easily, while ctimes would require tombstones (which may be a > good idea for other reasons [1]). > > On the other hand, logging is obviously more mechanism than ctimes, > and probably requires some garbage collection story. The advantage of ctime over logging is that the interface is super simple and intuitive. How would the interface to the log work? In terms of implementation, have you thought about leveraging the XAPIAN_MAX_CHANGESETS mechanism to produce your logs? David
Re: [PATCH] Add configurable changed tag to messages that have been changed on disk
Gaute Hope writes: > A db-tick or a _good_ ctime solution can as far as I can see solve both > David M's (correct me if I am wrong) and my purposes, as well as > probably have more use cases in the future. It would even be an > interesting direct search: show me everything that changed lately, > sorted. I could live with a db-tick scheme. I would prefer a ctime scheme, since then I can answer questions such as "what has changed in the last five minutes"? I mean all kinds of other stuff starts to break if your clock goes backwards on a mail server machine, not the least of which is that incremental backups will fail silently, so you risk losing your mail. A middle ground might be to use the maximum of two values: 1) the time-of-day at which notmuch started executing, and 2) the highest ctime in the database plus 100 microseconds (leaving plenty of slop to store timestamps as IEEE doubles with 52 significant bits). Since the values will be Btree-indexed, computing the max plus one will be cheap. Incidentally, if you are really this paranoid about time stamps, it should bother you that notmuch's directory timestamps only have one second granularity. It's not that hard to get a new message delivered in the same second that notmuch new finished running. In my synchronizer, I convert st_mtim (a struct timespec) into a double and keep that plus size in the database to decide if I need to re-hash files. But for directories, I'm stuck with NOTMUCH_VALUE_TIMESTAMP, which are quantized to the second. (Ironically, I think Xapian::sortable_serialize converts time_ts to doubles anyway, so avoiding st_mtim is not really helping performance.) David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] Add configurable changed tag to messages that have been changed on disk
Gaute Hope writes: > A db-tick or a _good_ ctime solution can as far as I can see solve both > David M's (correct me if I am wrong) and my purposes, as well as > probably have more use cases in the future. It would even be an > interesting direct search: show me everything that changed lately, > sorted. I could live with a db-tick scheme. I would prefer a ctime scheme, since then I can answer questions such as "what has changed in the last five minutes"? I mean all kinds of other stuff starts to break if your clock goes backwards on a mail server machine, not the least of which is that incremental backups will fail silently, so you risk losing your mail. A middle ground might be to use the maximum of two values: 1) the time-of-day at which notmuch started executing, and 2) the highest ctime in the database plus 100 microseconds (leaving plenty of slop to store timestamps as IEEE doubles with 52 significant bits). Since the values will be Btree-indexed, computing the max plus one will be cheap. Incidentally, if you are really this paranoid about time stamps, it should bother you that notmuch's directory timestamps only have one second granularity. It's not that hard to get a new message delivered in the same second that notmuch new finished running. In my synchronizer, I convert st_mtim (a struct timespec) into a double and keep that plus size in the database to decide if I need to re-hash files. But for directories, I'm stuck with NOTMUCH_VALUE_TIMESTAMP, which are quantized to the second. (Ironically, I think Xapian::sortable_serialize converts time_ts to doubles anyway, so avoiding st_mtim is not really helping performance.) David
Re: notmuch, OpenBSD issues
David Bremner writes: > Allan Streib writes: > >> I've been using notmuch on OpenBSD for a while, and I really appreciate >> the project. Since there is a thread on this... I'm using notmuch 0.17 on openbsd (from the ports tree). My problem is that notmuch new is just unbearably slow. I don't know if it's because I'm running the 32-bit (i386) mode or what, but it takes over one second per mail message. E.g., this is typical of what I see when checking for new mail: Processed 18 total files in 20s (0 files/sec.). Added 5 new messages to the database. Linux is 10 times faster. Have you seen any similar performance issues? David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
notmuch, OpenBSD issues
David Bremner writes: > Allan Streib writes: > >> I've been using notmuch on OpenBSD for a while, and I really appreciate >> the project. Since there is a thread on this... I'm using notmuch 0.17 on openbsd (from the ports tree). My problem is that notmuch new is just unbearably slow. I don't know if it's because I'm running the 32-bit (i386) mode or what, but it takes over one second per mail message. E.g., this is typical of what I see when checking for new mail: Processed 18 total files in 20s (0 files/sec.). Added 5 new messages to the database. Linux is 10 times faster. Have you seen any similar performance issues? David
Re: Synchronization success stories?
Tilmann Singer writes: > The steps performed on a sync run are roughly like this: > > - local: notmuch new > - local: notmuch search --output=messages .. > - remote: notmuch new > - remote: notmuch search --output=messages .. > - compare search results > - run rsync for mails that only exist locally > (using notmuch search --output=files to get the filenames) > - run rsync for mails that only exist remotely > (using notmuch search --output=files to get the filenames) What happens if you get a message that's been stuck in a queue for a few days and has an old Date: header? Or if you get new messages that have the same Message-ID as old ones? > Synchronization of the notmuch tags database is only necessary when I > switch between different client computers, which happens less > frequently. Do you use a laptop everywhere? I've found that for switching between my desktop machine at home, my laptop on the train, and my desktop at work (which amounts to five switches a day), the notmuch dump time is painfully slow--like well over 10 seconds for 100,000 messages. Hook that into notmuch-poll and you have a recipe for hanging emacs every time you type "G". Of course, I'm also experiencing the problem of "notmuch new" itself being painfully slow, but at least that's now my bottleneck in switching machines. I suspect the source of the notmuch new problem is that I have some huge, huge mailboxes. Some of my maildir/cur directories are multiple megabytes on a BSD FFS file system (no hashing, so linear filename lookups in something that doesn't fit in the dcache). On linux ext4 things are much faster. I intend to reorganize my maildir so that there is a top-level directory with the year and hence no single directory ever contains mail from more than one year. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Synchronization success stories?
Tilmann Singer writes: > The steps performed on a sync run are roughly like this: > > - local: notmuch new > - local: notmuch search --output=messages .. > - remote: notmuch new > - remote: notmuch search --output=messages .. > - compare search results > - run rsync for mails that only exist locally > (using notmuch search --output=files to get the filenames) > - run rsync for mails that only exist remotely > (using notmuch search --output=files to get the filenames) What happens if you get a message that's been stuck in a queue for a few days and has an old Date: header? Or if you get new messages that have the same Message-ID as old ones? > Synchronization of the notmuch tags database is only necessary when I > switch between different client computers, which happens less > frequently. Do you use a laptop everywhere? I've found that for switching between my desktop machine at home, my laptop on the train, and my desktop at work (which amounts to five switches a day), the notmuch dump time is painfully slow--like well over 10 seconds for 100,000 messages. Hook that into notmuch-poll and you have a recipe for hanging emacs every time you type "G". Of course, I'm also experiencing the problem of "notmuch new" itself being painfully slow, but at least that's now my bottleneck in switching machines. I suspect the source of the notmuch new problem is that I have some huge, huge mailboxes. Some of my maildir/cur directories are multiple megabytes on a BSD FFS file system (no hashing, so linear filename lookups in something that doesn't fit in the dcache). On linux ext4 things are much faster. I intend to reorganize my maildir so that there is a top-level directory with the year and hence no single directory ever contains mail from more than one year. David
Re: Synchronization success stories?
David Bremner writes: > Brian Sniffen writes: > >> I'm thrilled by using notmuch to manage my mail. Low-latency search is >> very important to me. But I use computers in a couple of >> places---several of which are laptops. Has anyone stories to share of >> successful multi-computer notmuch sync, for a corpus of a >> quarter-million messages or so? > > I use syncmaildir to sync the actual messages, and a copy of the output > of "notmuch dump" in git to sync the metadata. > > It works OK. A bit slow; depends how often you need to fetch new mail. If you want to see my solution, it is here: http://www.scs.stanford.edu/~dm/muchsync-0.tar.gz I'm a little embarrassed by this code, as I just started to test it a week ago then instantly became completely dependent on it. I will probably change the name (from muchsync to syncmuch) and the database format before releasing. But if you feel like beta-testing and giving me feedback, have a look. Beware that if you have been using notmuch dump, you may become instantly hooked on my solution... David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Synchronization success stories?
David Bremner writes: > Brian Sniffen writes: > >> I'm thrilled by using notmuch to manage my mail. Low-latency search is >> very important to me. But I use computers in a couple of >> places---several of which are laptops. Has anyone stories to share of >> successful multi-computer notmuch sync, for a corpus of a >> quarter-million messages or so? > > I use syncmaildir to sync the actual messages, and a copy of the output > of "notmuch dump" in git to sync the metadata. > > It works OK. A bit slow; depends how often you need to fetch new mail. If you want to see my solution, it is here: http://www.scs.stanford.edu/~dm/muchsync-0.tar.gz I'm a little embarrassed by this code, as I just started to test it a week ago then instantly became completely dependent on it. I will probably change the name (from muchsync to syncmuch) and the database format before releasing. But if you feel like beta-testing and giving me feedback, have a look. Beware that if you have been using notmuch dump, you may become instantly hooked on my solution... David
gnus-alias integration bug
Ian Kelling writes:
> gnus-alias allows you to set your identity based on the headers in the
> message you are replying. This is an essential feature in a mail client
> for me, I want to automatically reply as the person an email was sent to
> for certain addresses. This does not work in notmuch.
I need this feature, too. There is some support for the same
functionality built into notmuch. The big problem for me is that
.notmuch-config does not support wildcards (e.g., *@mydomain.com or
user-*@mydomain.com). Compounding this issue is that, other than
notmuch reply's internals, notmuch pretty much suppresses the
Delivered-To header, which is crucial for knowing in a non-heuristic way
where an email was actually delivered to. I want to compute my From:
header based on the Delivered-To, so I created a
mail-delivered-to-reply-from alist that maps one to the other.
It's pretty horrifying what I had to do to get this to work, but it
looks something like the following. (Apologies for any mistakes, I had
to strip out additional complexity I have for other things in there, so
it's not verbatim the code that I'm using, but it's pretty close.)
If someone knows a better way of doing this with stock notmuch, please
let me know. I know it's simpler to fix notmuch-reply, but I really
don't want to start depending on a custom-forked version on the notmuch
binary, as I read my mail from multiple operating systems and I'm sure
I'd forget to update one of them.
David
;;; Regular expression matching all of my own email addresses. Note
;;; this is also used by mail-dont-reply-to (which ships with emacs).
(setq mail-dont-reply-to-names
(concat "REXEXP MATCHING ALL YOUR EMAIL ALIASES"))
;;; Alist mapping Delivered-To addresses to the address that should be
;;; in the From header of replies.
(setq mail-delivered-to-reply-from
'(("delivered-to-\\(.*\\)@address.com"
. "Name ")
;; ...
))
(defun compute-from (email)
(let ((case-fold-search t)
(subst (assoc-default email mail-delivered-to-reply-from
#'string-match)))
(and subst
(concat (notmuch-user-name) " <"
(replace-match subst t nil email) ">"
(defun get-delivered-to-from-path (path)
(with-temp-buffer
; It's weird to use sed, but there's also no reason to read a huge
; file into a buffer right here when we reply to messags with
; attachments. (notmuch show doesn't support --body=false in
; --format=raw mode, and the other modes all get rid of Delivered-To.)
(call-process "sed" nil t t "-ne"
"/^Delivered-To: */{s///p;q;}; /^$/q" path)
(goto-char (point-max))
(or (bobp) (delete-char -1))
(and (not (bobp)) (buffer-string
(defun get-msg-pathnames (query)
(notmuch-call-notmuch-sexp
"search" "--format=sexp" "--format-version=1"
"--output=files" "--" query))
(defun compute-reply-from (pl)
(let* ((orig (plist-get pl :original))
(path (plist-get orig :filename))
(orig-from (plist-get (plist-get orig :headers) :From))
(addr
(cond
((get-delivered-to-from-path path))
;; in case of maildir.synchronize_flags=true, the pathname
;; might already be incorrect, so try getting it again
((and (setq path (car (get-msg-pathnames query-string)))
(get-delivered-to-from-path path)))
((string-match mail-dont-reply-to-names
(setq orig-from (car (rfc822-addresses orig-from
orig-from)
)))
(and addr (compute-from addr
;;; Somehow no plist-del function in emacs
(defun my-plist-del (plist0 prop)
(if (eq prop (car plist0))
(cddr plist0)
(let ((p (cdr plist0)))
(while (and p (not (eq prop (cadr p
(setq p (cddr p)))
(if p (setcdr p (cdddr p)))
plist0)))
(defun fix-reply-sexp (pl)
(let* ((rh (plist-get pl :reply-headers))
(from (compute-reply-from pl))
h)
(if from (setq rh (plist-put rh :From from)))
(and (setq h (plist-get rh :To))
(setq h (mail-dont-reply-to h))
(not (equal h ""))
(setq rh (plist-put rh :To h)))
(and (setq h (plist-get rh :Cc))
(setq h (mail-dont-reply-to h))
(if (equal h "")
(setq rh (my-plist-del rh :Cc))
(setq rh (plist-put rh :Cc h
(plist-put pl :reply-headers rh)))
(defadvice notmuch-call-notmuch-sexp (after fix-reply activate)
"Fix From/To/CC headers in reply"
(if (equal (ad-get-arg 0) "reply")
(setq ad-return-value (fix-reply-sexp ad-return-value
Re: gnus-alias integration bug
Ian Kelling writes:
> gnus-alias allows you to set your identity based on the headers in the
> message you are replying. This is an essential feature in a mail client
> for me, I want to automatically reply as the person an email was sent to
> for certain addresses. This does not work in notmuch.
I need this feature, too. There is some support for the same
functionality built into notmuch. The big problem for me is that
.notmuch-config does not support wildcards (e.g., *@mydomain.com or
user-*@mydomain.com). Compounding this issue is that, other than
notmuch reply's internals, notmuch pretty much suppresses the
Delivered-To header, which is crucial for knowing in a non-heuristic way
where an email was actually delivered to. I want to compute my From:
header based on the Delivered-To, so I created a
mail-delivered-to-reply-from alist that maps one to the other.
It's pretty horrifying what I had to do to get this to work, but it
looks something like the following. (Apologies for any mistakes, I had
to strip out additional complexity I have for other things in there, so
it's not verbatim the code that I'm using, but it's pretty close.)
If someone knows a better way of doing this with stock notmuch, please
let me know. I know it's simpler to fix notmuch-reply, but I really
don't want to start depending on a custom-forked version on the notmuch
binary, as I read my mail from multiple operating systems and I'm sure
I'd forget to update one of them.
David
;;; Regular expression matching all of my own email addresses. Note
;;; this is also used by mail-dont-reply-to (which ships with emacs).
(setq mail-dont-reply-to-names
(concat "REXEXP MATCHING ALL YOUR EMAIL ALIASES"))
;;; Alist mapping Delivered-To addresses to the address that should be
;;; in the From header of replies.
(setq mail-delivered-to-reply-from
'(("delivered-to-\\(.*\\)@address.com"
. "Name ")
;; ...
))
(defun compute-from (email)
(let ((case-fold-search t)
(subst (assoc-default email mail-delivered-to-reply-from
#'string-match)))
(and subst
(concat (notmuch-user-name) " <"
(replace-match subst t nil email) ">"
(defun get-delivered-to-from-path (path)
(with-temp-buffer
; It's weird to use sed, but there's also no reason to read a huge
; file into a buffer right here when we reply to messags with
; attachments. (notmuch show doesn't support --body=false in
; --format=raw mode, and the other modes all get rid of Delivered-To.)
(call-process "sed" nil t t "-ne"
"/^Delivered-To: */{s///p;q;}; /^$/q" path)
(goto-char (point-max))
(or (bobp) (delete-char -1))
(and (not (bobp)) (buffer-string
(defun get-msg-pathnames (query)
(notmuch-call-notmuch-sexp
"search" "--format=sexp" "--format-version=1"
"--output=files" "--" query))
(defun compute-reply-from (pl)
(let* ((orig (plist-get pl :original))
(path (plist-get orig :filename))
(orig-from (plist-get (plist-get orig :headers) :From))
(addr
(cond
((get-delivered-to-from-path path))
;; in case of maildir.synchronize_flags=true, the pathname
;; might already be incorrect, so try getting it again
((and (setq path (car (get-msg-pathnames query-string)))
(get-delivered-to-from-path path)))
((string-match mail-dont-reply-to-names
(setq orig-from (car (rfc822-addresses orig-from
orig-from)
)))
(and addr (compute-from addr
;;; Somehow no plist-del function in emacs
(defun my-plist-del (plist0 prop)
(if (eq prop (car plist0))
(cddr plist0)
(let ((p (cdr plist0)))
(while (and p (not (eq prop (cadr p
(setq p (cddr p)))
(if p (setcdr p (cdddr p)))
plist0)))
(defun fix-reply-sexp (pl)
(let* ((rh (plist-get pl :reply-headers))
(from (compute-reply-from pl))
h)
(if from (setq rh (plist-put rh :From from)))
(and (setq h (plist-get rh :To))
(setq h (mail-dont-reply-to h))
(not (equal h ""))
(setq rh (plist-put rh :To h)))
(and (setq h (plist-get rh :Cc))
(setq h (mail-dont-reply-to h))
(if (equal h "")
(setq rh (my-plist-del rh :Cc))
(setq rh (plist-put rh :Cc h
(plist-put pl :reply-headers rh)))
(defadvice notmuch-call-notmuch-sexp (after fix-reply activate)
"Fix From/To/CC headers in reply"
(if (equal (ad-get-arg 0) "reply")
(setq ad-return-value (fix-reply-sexp ad-return-value
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] Add configurable changed tag to messages that have been changed on disk
Gaute Hope writes: > When one of the source files for a message is changed on disk, renamed, > deleted or a new source file is added. A configurable changed tag is > is added. The tag can be configured under the option 'changed_tags' in > the [new] section, the default is none. Tests have been updated to > accept the new config option. > > notmuch-setup now asks for a changed tag after the new tags question. > > This could be useful for for example 'afew' to detect remote changes in > IMAP folders and update the FolderNameFilter to also add tags or remove > tags when a _existing_ message has been added to or removed from a > maildir. I think this is the wrong way to achieve such functionality, because then the change tag A) is expensive to remove, B) is easy to misuse (remember to call fsync everywhere before deleting the change tag), and C) can be used by only one application. A better approach would be to add a new "modtime" xapian value that is updated whenever the tags or any other terms (such as XFDIRENTRY) are added to or deleted from a docid. If it's a Xapian value, rather than a term, then modtime will be queriable just like date, allowing multiple applications to query all docids modified since the last time they ran. I currently have multiple applications that could significantly benefit from such a modtime. An obvious one is proper incremental backups with notmuch-dump. Another example is a tool I have that synchromizes maildirs and notmuch tags across machines. With the current interface, there is no way to do this without scanning the entire database, because any message, even a very old one, may have changed tags or links. Moreover, something like notmuch-dump is way, way too slow to run every time you want to check for new mail. notmuch-dump costs 5-10 seconds on my 110,000-message maildir! In fact, any approach the gathers tags associated with each individual docid is a complete non-starter, forcing me to violate abstraction and examine the postlists associated with each tag and XFDIRENTRY term. Even my highly optimized implementation takes about 250 msec (1400 msec on a 32-bit machine), which adds perceptible latency to synchronizing my clients' notmuch maildirs with my server's when I poll for new mail. Yet another application is something like nottoomuch-addresses, which currently uses an occasionally incorrect heuristic to detect new messages based on the Date header. Let me make a stronger statement, which is that not only are modification times an incredibly useful and general primitive, but lack of modification times is the single thing that kept me away from notmuch despite years of wanting to switch. In the end, I invested months developing a highly-optimized change detector that efficiently diffs Xapian's Btrees against a mysql database with a snapshot of the same information. My solution works, and I now enjoy a replicated notmuch setup synchronized across three machines, including offline access on my laptop. But my 4,000-line C++ program might have been a 400-line shell script if only notmuch supported docid mod times. Also, to put this in perspective, how long does it take to remove the changed tags from a bunch of messages? If it's longer than 300 msec on a 64-bit machine, then even with a single application you'd be better off using my crazy on-the-side mysql version vector scheme. David
Re: [PATCH] Add configurable changed tag to messages that have been changed on disk
Gaute Hope writes: > When one of the source files for a message is changed on disk, renamed, > deleted or a new source file is added. A configurable changed tag is > is added. The tag can be configured under the option 'changed_tags' in > the [new] section, the default is none. Tests have been updated to > accept the new config option. > > notmuch-setup now asks for a changed tag after the new tags question. > > This could be useful for for example 'afew' to detect remote changes in > IMAP folders and update the FolderNameFilter to also add tags or remove > tags when a _existing_ message has been added to or removed from a > maildir. I think this is the wrong way to achieve such functionality, because then the change tag A) is expensive to remove, B) is easy to misuse (remember to call fsync everywhere before deleting the change tag), and C) can be used by only one application. A better approach would be to add a new "modtime" xapian value that is updated whenever the tags or any other terms (such as XFDIRENTRY) are added to or deleted from a docid. If it's a Xapian value, rather than a term, then modtime will be queriable just like date, allowing multiple applications to query all docids modified since the last time they ran. I currently have multiple applications that could significantly benefit from such a modtime. An obvious one is proper incremental backups with notmuch-dump. Another example is a tool I have that synchromizes maildirs and notmuch tags across machines. With the current interface, there is no way to do this without scanning the entire database, because any message, even a very old one, may have changed tags or links. Moreover, something like notmuch-dump is way, way too slow to run every time you want to check for new mail. notmuch-dump costs 5-10 seconds on my 110,000-message maildir! In fact, any approach the gathers tags associated with each individual docid is a complete non-starter, forcing me to violate abstraction and examine the postlists associated with each tag and XFDIRENTRY term. Even my highly optimized implementation takes about 250 msec (1400 msec on a 32-bit machine), which adds perceptible latency to synchronizing my clients' notmuch maildirs with my server's when I poll for new mail. Yet another application is something like nottoomuch-addresses, which currently uses an occasionally incorrect heuristic to detect new messages based on the Date header. Let me make a stronger statement, which is that not only are modification times an incredibly useful and general primitive, but lack of modification times is the single thing that kept me away from notmuch despite years of wanting to switch. In the end, I invested months developing a highly-optimized change detector that efficiently diffs Xapian's Btrees against a mysql database with a snapshot of the same information. My solution works, and I now enjoy a replicated notmuch setup synchronized across three machines, including offline access on my laptop. But my 4,000-line C++ program might have been a 400-line shell script if only notmuch supported docid mod times. Also, to put this in perspective, how long does it take to remove the changed tags from a bunch of messages? If it's longer than 300 msec on a 64-bit machine, then even with a single application you'd be better off using my crazy on-the-side mysql version vector scheme. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
