Re: [PATCH] emacs: wash: make word-wrap bound message width
On Sat, 20 Aug 2016, Tomi Ollila wrote: > On Wed, Jul 06 2016, Mark Walters wrote: > >> Previously if notmuch-wash-wrap-lines-length was set then all messages >> would be wrapped at this value (or window-width if that is >> smaller). This was done regardless of the message's depth in a thread: >> for example, if the n.w.w.l.l is 80 and the messages depth is 20 >> (so indented 20 by default) the messages text only got 60 characters >> of space. >> >> This commit changes that so a message always gets the full n.w.w.l.l >> of width regardless of its indentation (unless that goes over >> window-width of course). >> --- >> >> This is what I would like -- I don't know if anyone would like to keep >> the previous behaviour as an option. The code-part for that is easy, but >> getting >> the docstrings and and defcustoms right is not clear. > > The change looks good to me -- but I just don't understand why someone > would set notmuch-wash-wrap-lines-length to something else than nil > -- and if it is set to some number what the behaviour should be ? I like to run notmuch in a very wide window as that make search view work better. But I don't like to read very wide text in show view. Emails sent from notmuch are fine as they hard-wrapped at something like 80 characters (usually), but some clients seem to make each paragraph one line and leave the recipient to wrap the text. notmuch-word-wrap-long-lines just controls where to wrap this text. The problem with the current version is that it ignores the indentation due to being deep in a thread, so if I set it to 80, then deep messages (even ones hard wrapped to 80) get wrapped at say 60 characters. Mine wraps at indentation+80 (if you set nw.w.l.l to 80) which means no hard-wrapped message get further wrapped however deep in a thread it is. Best wishes Mark >> emacs/notmuch-wash.el | 6 +++--- >> 1 file changed, 3 insertions(+), 3 deletions(-) >> >> diff --git a/emacs/notmuch-wash.el b/emacs/notmuch-wash.el >> index 57e6dfa..e16b8cc 100644 >> --- a/emacs/notmuch-wash.el >> +++ b/emacs/notmuch-wash.el >> @@ -121,8 +121,8 @@ collapse the remaining lines into a button." >> >> If this is nil, lines in messages will be wrapped to fit in the >> current window. If this is a number, lines will be wrapped after >> -this many characters or at the window width (whichever one is >> -lower)." >> +this many characters (ignoring indentation due to thread depth) >> +or at the window width (whichever one is lower)." >>:type '(choice (const :tag "window width" nil) >> (integer :tag "number of characters")) >>:group 'notmuch-wash) >> @@ -336,7 +336,7 @@ the wrapped text are maintained." >> >>(let* ((coolj-wrap-follows-window-size nil) >> (limit (if (numberp notmuch-wash-wrap-lines-length) >> -(min notmuch-wash-wrap-lines-length >> +(min (+ notmuch-wash-wrap-lines-length depth) >> (window-width)) >>(window-width))) >> (fill-column (- limit >> -- >> 2.1.4 >> >> ___ >> notmuch mailing list >> notmuch@notmuchmail.org >> https://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Mail archives in Git using ssoma
"W. Trevor King" wrote: > On Fri, Nov 07, 2014 at 11:03:21AM -0800, W. Trevor King wrote: > > Eric Wong has been working on some tools to store email in a Git > > repository, and his client-side code is ssoma [1]. I wanted a bit > > more metadata than the stock ssoma-mda [2], and ended up just > > writing a ssoma-mda in Python [3]… Btw, for public-inbox, I'm using git-fast-import now, so imports are a bit faster and $GIT_DIR/ssoma.index is no longer used. This was crucial for getting git@vger archives imported in a reasonable time. public-inbox-* still keeps ssoma.index up-to-date for backwards compatibility with ssoma, and will probably do so until 2020 or later (there'll be a few years of deprecation notices) So I or someone else needs to update Perl ssoma to use fast-import at some point, too; and I suggest your python version do the same. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Mail archives in Git using ssoma (Docker image)
"W. Trevor King" wrote: > On Sun, Aug 21, 2016 at 12:08:52PM +, Eric Wong wrote: > > "W. Trevor King" wrote: > > > This is the ssoma archive (with the data in it). I just set up a > > > basic HTTP archive (following [1]) based on a Docker image [2] (Gentoo > > > doesn't package all the Perl dependencies public-inbox needs). > > > > Ugh, that sucks (sorry, not a fan of Docker). > > > > What's missing from Gentoo? > > Gentoo doesn't package (or I couldn't find the package for) > Encode::MIME::Header or Mail::Thread. I tried installing things from > CPAN, but ran into a compile-time error from the ‘cpan’ invocationand > gave up ;). I can try and reproduce the error if you're curious, but > I don't have it handy at the moment. Encode::MIME::Header is distributed with perl itself on Debian and also the stock upstream install. Not sure if there's an option you missed or disabled. Which perl version do you use? perl on 5.14 Debian wheezy even seems to have it. I actually still want everything to work on 5.8, since that seems to be the de-facto baseline in the wild. Mail::Thread is one .pm, and I'll probably replace it with something (same algorithm) which can use half the memory by avoiding wrapper object abstractions (it's probably the biggest memory hog at the moment). lib/PublicInbox/Thread.pm already has 3 monkey patches to workaround upstream bugs in Mail::Thread. It's dead upstream, and not available on FreeBSD, either. > > > $ git config -f srv/notmuch.git/config publicinbox.http > > > http://tremily.us > > > $ git config -f srv/notmuch.git/config publicinbox.email > > > notmuch@notmuchmail.org > > > > That should probably be: > > > > ; based on your [3] > > git config -f srv/notmuch.git/config \ > > publicinbox.notmuch.url http://tremily.us/notmuch > > > > git config -f srv/notmuch.git/config \ > > publicinbox.notmuch.address notmuch@notmuchmail.org > > > > ; this is crucial for all the public-inbox-* tools > > git config -f srv/notmuch.git/config \ > > publicinbox.notmuch.mainrepo /path/to/notmuch.git > > I was using these in the Dockerfile's CMD: > > (cd /srv; >for NAME in *; >do > CONF="/srv/${NAME}/config"; > public-inbox-init "${NAME}" "/srv/${NAME}" $(git config -f "${CONF}" > publicinbox.http) $(git config -f "${CONF}" publicinbox.email); >done) && … > > Are you saying that I can skip the ~/.public-inbox/config entries > setup by public-inbox-init if I set publicinbox.{name}.* in the ssoma > repository's config? That would be nice. Erm, sorry, no, I mean ~/.public-inbox/config as the "git config -f" arg in the above commands. Your original config was meaningless in the context of public-inbox itself; I don't recall public-inbox relies on $GIT_DIR/config much (if at all) outside of standard git things. Using ~/.public-inbox/config is required for multi-inbox lookups (since you normally run MDA w/o args) You can also override ~/.public-inbox/config by setting the PI_CONFIG env (like GIT_CONFIG). > I don't see a point to having {name} in ssoma-config settings though, > since you're already in a single bucket by that point (using > publicinbox.{name}.* makes sense in the multi-bucket > ~/.public-inbox/config). > > > > It's not updating automatically yet, but that will probably look > > > like: > > > > > > 1. Pull new mbox [4]. > > > 2. Import into notmuch-arcives [5]. > > > 3. Re-run public-inbox-index (this could probably be via ‘docker exec …’. > > > > > > But I'll have to test that to confirm. And ideally we'd be using > > > ssoma-mda or similar directly, instead of going through mbox, but I'd > > > rather get the official headers on the stored mail than be efficient > > > ;). > > > > For mirroring existing lists, I started using public-inbox-watch > > which currently watches Maildirs. > > If I had a Maildir locally, I'd just use procmail and push new > messages into ssoma-mda. I'm using the import script because my local > mail has “how we delivered this to Trevor” headers (which I don't want > to add) but the downloaded mbox has “how we delivered this to > notmuch@notmuchmail.org” (which seems like a better fit for a shared > ssoma repo). I don't mind extra/different headers. The majority of messages in public-inbox.org/git/ has messages that were delivered to gmane; recent ones are delivered to me, and some holes were filled in by Jeff King's archives. All of our mail systems add different headers. > > I recommend public-inbox-watch for mirroring existing lists (such as > > what I did with git@vger) but public-inbox-mda for self-hosted lists > > (such as m...@public-inbox.org). > > Why is that? Procmail + public-inbox-mda (or my Python ssoma-mda fork > [1,2]) seems simpler and equally effective if you want to insert a > message that your mail system is delivering locally. -watch is usable for importing big archives or bursts of traffic since it doesn't have to r
Re: Mail archives in Git using ssoma
On Sun, Aug 21, 2016 at 06:37:04PM +, Eric Wong wrote: > "W. Trevor King" wrote: > > On Fri, Nov 07, 2014 at 11:03:21AM -0800, W. Trevor King wrote: > > > Eric Wong has been working on some tools to store email in a Git > > > repository, and his client-side code is ssoma [1]. I wanted a bit > > > more metadata than the stock ssoma-mda [2], and ended up just > > > writing a ssoma-mda in Python [3]… > > Btw, for public-inbox, I'm using git-fast-import now, so imports are > a bit faster and $GIT_DIR/ssoma.index is no longer used. This was > crucial for getting git@vger archives imported in a reasonable time. > > public-inbox-* still keeps ssoma.index up-to-date for backwards > compatibility with ssoma, and will probably do so until 2020 or > later (there'll be a few years of deprecation notices) > > So I or someone else needs to update Perl ssoma to use fast-import > at some point, too; and I suggest your python version do the same. ssoma-mda imports 22k notmuch messages in around 15 minutes (with profiling enabled), and: $ python -m cProfile -o profile import.py notmuch.mbox $ python -c "import pstats; p=pstats.Stats('profile'); p.sort_stats('cumulative').print_stats(10)" Sun Aug 21 12:56:49 2016profile 101823722 function calls (99078415 primitive calls) in 885.069 seconds Ordered by: cumulative time List reduced from 1145 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 70/10.0020.000 885.069 885.069 {built-in method exec} 10.1110.111 885.069 885.069 /home/wking/src/notmuch/notmuch-archives.git/import.py:9() 10.4000.400 884.915 884.915 /home/wking/src/notmuch/notmuch-archives.git/import.py:17(import_mbox) 228750.6010.000 863.3710.038 /home/wking/src/notmuch/notmuch-archives.git/ssoma_mda.py:362(deliver) 228758.9430.000 810.4590.035 /home/wking/src/notmuch/notmuch-archives.git/ssoma_mda.py:207(append) 228750.4180.000 308.3530.013 /home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:146(write_tree) 22875 307.8550.013 307.8550.013 {built-in method git_index_write_tree} 228740.5750.000 279.2930.012 /home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:238(diff_to_tree) 22874 278.5010.012 278.5010.012 {built-in method git_diff_tree_to_index} 228750.0880.000 80.4130.004 /home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:99(read) 38 ms per ssoma delivery is probably fast enough, especially if you are invoking ssoma-mda once per message, since process setup will take a similar amount of time: $ time python -c 'print("hello")' hello real0m0.016s user0m0.013s sys 0m0.003s It's possible that fast-import would shave a few ms off the pygit2 addition (I'm not sure, and maybe pygit2 is faster than fast-import). But I doubt it matters enough either way to be worth changing unless you are dealing with a really large corpus. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Mail archives in Git using ssoma (Docker image)
On Sun, Aug 21, 2016 at 12:08:52PM +, Eric Wong wrote: > "W. Trevor King" wrote: > > This is the ssoma archive (with the data in it). I just set up a > > basic HTTP archive (following [1]) based on a Docker image [2] (Gentoo > > doesn't package all the Perl dependencies public-inbox needs). > > Ugh, that sucks (sorry, not a fan of Docker). > > What's missing from Gentoo? Gentoo doesn't package (or I couldn't find the package for) Encode::MIME::Header or Mail::Thread. I tried installing things from CPAN, but ran into a compile-time error from the ‘cpan’ invocationand gave up ;). I can try and reproduce the error if you're curious, but I don't have it handy at the moment. > > $ git config -f srv/notmuch.git/config publicinbox.http http://tremily.us > > $ git config -f srv/notmuch.git/config publicinbox.email > > notmuch@notmuchmail.org > > That should probably be: > > ; based on your [3] > git config -f srv/notmuch.git/config \ > publicinbox.notmuch.url http://tremily.us/notmuch > > git config -f srv/notmuch.git/config \ > publicinbox.notmuch.address notmuch@notmuchmail.org > > ; this is crucial for all the public-inbox-* tools > git config -f srv/notmuch.git/config \ > publicinbox.notmuch.mainrepo /path/to/notmuch.git I was using these in the Dockerfile's CMD: (cd /srv; for NAME in *; do CONF="/srv/${NAME}/config"; public-inbox-init "${NAME}" "/srv/${NAME}" $(git config -f "${CONF}" publicinbox.http) $(git config -f "${CONF}" publicinbox.email); done) && … Are you saying that I can skip the ~/.public-inbox/config entries setup by public-inbox-init if I set publicinbox.{name}.* in the ssoma repository's config? That would be nice. I don't see a point to having {name} in ssoma-config settings though, since you're already in a single bucket by that point (using publicinbox.{name}.* makes sense in the multi-bucket ~/.public-inbox/config). > > It's not updating automatically yet, but that will probably look > > like: > > > > 1. Pull new mbox [4]. > > 2. Import into notmuch-archives [5]. > > 3. Re-run public-inbox-index (this could probably be via ‘docker exec …’. > > > > But I'll have to test that to confirm. And ideally we'd be using > > ssoma-mda or similar directly, instead of going through mbox, but I'd > > rather get the official headers on the stored mail than be efficient > > ;). > > For mirroring existing lists, I started using public-inbox-watch > which currently watches Maildirs. If I had a Maildir locally, I'd just use procmail and push new messages into ssoma-mda. I'm using the import script because my local mail has “how we delivered this to Trevor” headers (which I don't want to add) but the downloaded mbox has “how we delivered this to notmuch@notmuchmail.org” (which seems like a better fit for a shared ssoma repo). > I recommend public-inbox-watch for mirroring existing lists (such as > what I did with git@vger) but public-inbox-mda for self-hosted lists > (such as m...@public-inbox.org). Why is that? Procmail + public-inbox-mda (or my Python ssoma-mda fork [1,2]) seems simpler and equally effective if you want to insert a message that your mail system is delivering locally. > > One shift from Gmane's mid.gmane.org/… is that the public-inbox UI > > Message-ID lookup is per-bucket, and public-inbox seems to be > > encouraging per-list buckets. > > The public-inbox-nntpd interface supports mid lookups across all > inboxes in that instance; so it should be doable in the WWW > interface, too. Either way, I think it has to be O(n) where (n) is > the number of Xapian DBs, though. I'm more concerned about the interface, and less about the implementation (which can be improved later). The (n) lookups are trivially parallelizable, and you can always add a Message-ID → buckets lookup table if (n) lookups turns out to be too slow. Cheers, Trevor [1]: id:20141107190321.gl23...@odin.tremily.us [2]: id:af679af8257e250ac606e35a1307ad02907b8426.1413663212.git.wk...@tremily.us http://public-inbox.org/meta/af679af8257e250ac606e35a1307ad02907b8426.1413663212.git.wk...@tremily.us/t/#u -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Mail archives in Git using ssoma (Docker image)
Eric Wong writes: > For mirroring existing lists, I started using public-inbox-watch > which currently watches Maildirs. The config knobs are sorta > documented from my announcement to git@vger: > > https://public-inbox.org/git/20160710004813.ga20...@dcvr.yhbt.net/ > http://hjrcffqmbrq6wope.onion/git/20160710004813.ga20...@dcvr.yhbt.net/ > > Initial import (w/o spamassassin) was done with > scripts/import_vger_from_mbox in the source: > > torsocks git clone http://hjrcffqmbrq6wope.onion/public-inbox > git clone https://public-inbox.org/ public-inbox > git clone git://repo.or.cz/public-inbox > FWIW, I already have a Maildir with a complete (and updated) archive of the list (and only that) for use of nmbug. So at the risk of putting all eggs in one basket, perhaps public-inbox-watch could watch that maildir. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Mail archives in Git using ssoma (Docker image)
On Sat, Aug 20, 2016 at 09:36:31PM -0700, W. Trevor King wrote: > [2]: git://tremily.us/notmuch-archives.git This is the ssoma archive (with the data in it). I just set up a basic HTTP archive (following [1]) based on a Docker image [2] (Gentoo doesn't package all the Perl dependencies public-inbox needs). Dockerfile for rebuilding the image is in [2]. I'm currently hosting the archives (HTTP only) at [3]. Spinning that up from the Docker image looks like: $ mkdir srv $ git clone --bare git://tremily.us/notmuch-archives.git srv/notmuch $ echo 'Notmuch -- Just an email system' >srv/notmuch.git/description $ git config -f srv/notmuch.git/config publicinbox.http http://tremily.us $ git config -f srv/notmuch.git/config publicinbox.email notmuch@notmuchmail.org $ docker run --name notmuch-archives -d -p 80:8080 -v ${PWD}/srv/:/srv/ wking/public-inbox (although I'm using -p ###:8080 and have an Nginx reverse-proxy in front). It's not updating automatically yet, but that will probably look like: 1. Pull new mbox [4]. 2. Import into notmuch-archives [5]. 3. Re-run public-inbox-index (this could probably be via ‘docker exec …’. But I'll have to test that to confirm. And ideally we'd be using ssoma-mda or similar directly, instead of going through mbox, but I'd rather get the official headers on the stored mail than be efficient ;). One shift from Gmane's mid.gmane.org/… is that the public-inbox UI Message-ID lookup is per-bucket, and public-inbox seems to be encouraging per-list buckets. And while I feel like I had a good grasp of the ssoma format two years ago, I know very little about Perl and public-inbox. I'm sure you could setup a public-inbox host that is more efficient than what's currently in my Docker image. Cheers, Trevor [1]: http://public-inbox.org/INSTALL [2]: https://hub.docker.com/r/wking/public-inbox/ [3]: http://tremily.us/notmuch/ [4]: https://notmuchmail.org/archives/notmuch.mbox [5]: id:20160821043631.ga2...@odin.tremily.us -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch