[PATCH 2/2] NEWS: News for big endian sha1 bug fix.
Quoth david at tethera.net on Nov 24 at 5:29 pm: > From: David Bremner > > We could give more details about how to migrate tags, but I'm not sure > that it's a practical problem, or just a theoretical one. > --- > NEWS | 14 +- > 1 file changed, 13 insertions(+), 1 deletion(-) > > diff --git a/NEWS b/NEWS > index 3383ecf..31c6284 100644 > --- a/NEWS > +++ b/NEWS > @@ -1,6 +1,18 @@ > -Notmuch 0.17~rc1 (2013-11-20) > +Notmuch 0.17~rc2 (2013-xx-yy) > = > > +Incompatible change in SHA1 computation > +--- > + > +Previously on big endian architectures like sparc and powerpc the > +computation of SHA1 hashes was incorrect. This meant that messages > +with overlong or missing message-ids were given different computed > +message-ids than on more common little endian architectures like i386 > +and amd64. If you use notmuch on a big endian architecture, you are > +strongly advised to make a backup of your tags using `notmuch dump` > +before this upgrade. It should be possible to migrate the tags using a > +script. > + Should this mention how to find such messages? Something like notmuch dump | awk '/^notmuch-sha1-[0-9a-f]{40} / {system("notmuch search id:" $1)}'
[PATCH] new: Detect dirent.d_type support at configure time
On Sun, Nov 24 2013, Austin Clements wrote: > Support for dirent.d_type is OS-specific. Previously, we used > _DIRENT_HAVE_D_TYPE to detect support for this, but this is apparently > a glic-ism (FreeBSD, for example, supports d_type, but does not define > this). Since there's no cross-platform way to detect support for > dirent.d_type, detect it using a test compile at configure time. > --- LGTM. Tomi > compat/have_d_type.c | 10 ++ > configure| 16 > notmuch-new.c| 2 +- > 3 files changed, 27 insertions(+), 1 deletion(-) > create mode 100644 compat/have_d_type.c > > diff --git a/compat/have_d_type.c b/compat/have_d_type.c > new file mode 100644 > index 000..9ca6c6e > --- /dev/null > +++ b/compat/have_d_type.c > @@ -0,0 +1,10 @@ > +#include > + > +int main() > +{ > +struct dirent ent; > + > +(void) ent.d_type; > + > +return 0; > +} > diff --git a/configure b/configure > index 1a8e939..d2d193c 100755 > --- a/configure > +++ b/configure > @@ -557,6 +557,17 @@ else > fi > rm -f compat/have_timegm > > +printf "Checking for dirent.d_type... " > +if ${CC} -o compat/have_d_type "$srcdir"/compat/have_d_type.c > /dev/null > 2>&1 > +then > +printf "Yes.\n" > +have_d_type="1" > +else > +printf "No (will use stat instead).\n" > +have_d_type="0" > +fi > +rm -f compat/have_d_type > + > printf "Checking for standard version of getpwuid_r... " > if ${CC} -o compat/check_getpwuid "$srcdir"/compat/check_getpwuid.c > > /dev/null 2>&1 > then > @@ -745,6 +756,9 @@ HAVE_STRCASESTR = ${have_strcasestr} > # build its own version) > HAVE_STRSEP = ${have_strsep} > > +# Whether struct dirent has d_type (if not, then notmuch will use stat) > +HAVE_D_TYPE = ${have_d_type} > + > # Whether the Xapian version in use supports compaction > HAVE_XAPIAN_COMPACT = ${have_xapian_compact} > > @@ -805,6 +819,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) > \$(GMIME_CFLAGS) \\ > \$(VALGRIND_CFLAGS) \\ > -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\ > -DHAVE_STRSEP=\$(HAVE_STRSEP) \\ > +-DHAVE_D_TYPE=\$(HAVE_D_TYPE) \\ > -DSTD_GETPWUID=\$(STD_GETPWUID) \\ > -DSTD_ASCTIME=\$(STD_ASCTIME) \\ > -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT) > @@ -813,6 +828,7 @@ CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) > \$(GMIME_CFLAGS)\\ >\$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\ >-DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\ >-DHAVE_STRSEP=\$(HAVE_STRSEP) \\ > + -DHAVE_D_TYPE=\$(HAVE_D_TYPE) \\ >-DSTD_GETPWUID=\$(STD_GETPWUID) \\ >-DSTD_ASCTIME=\$(STD_ASCTIME) \\ >-DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT) > diff --git a/notmuch-new.c b/notmuch-new.c > index ba05cb4..423e188 100644 > --- a/notmuch-new.c > +++ b/notmuch-new.c > @@ -167,7 +167,7 @@ dirent_type (const char *path, const struct dirent *entry) > char *abspath; > int err, saved_errno; > > -#ifdef _DIRENT_HAVE_D_TYPE > +#if HAVE_D_TYPE > /* Mapping from d_type to stat mode_t. We omit DT_LNK so that > * we'll fall through to stat and get the real file type. */ > static const mode_t modes[] = { > -- > 1.8.4.rc3 > > ___ > notmuch mailing list > notmuch at notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch
notmuch sha1 implementation broken on (some) big-endian architectures
On Sun, Nov 24 2013, David Bremner wrote: > The following code, when linked with libnotmuch.a and libutil.a does a > passable imitation of sha1sum on amd64 (and I guess also i386) but > computes a different digest on powerpc and probably sparc and s390x. > > In the long run we should maybe outsource hash computations to > e.g. librhash, but I'd like a simpler fix for 0.17, if possible > > P.S. I blame Austin for adding the "missing-headers" test which found > this bug ;). This is interesting problem, I would have guessed that this would fails on LITTLE_ENDIAN machines easier, if ever... ... especially as there is line lib/libsha1.c:52:#if (PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN) ... but... I could not find any (other) matches for PLATFORM_BYTE_ORDER nor IS_LITTLE_ENDIAN in source code or in /usr/include/**/*.h or in /usr/lib/gcc/**/*.h I did some testing and it seems that #if (F == BBBAAARRR) #endif will have in output file in case neither of the above are defined... :/ So, this could work: #if // for BYTE_ORDER && LITTLE_ENDIAN and then #if (BYTE_ORDER == LITTLE_ENDIAN) ... to replace lib/libsha1.c:53 (53 now after endian.h added) Please test on BIG_ENDIAN machine... In case this works, then we'd need to inform users that their long/missing Message ID:s are now coded differently in their databases... Tomi
[PATCH 2/2] NEWS: News for big endian sha1 bug fix.
From: David BremnerWe could give more details about how to migrate tags, but I'm not sure that it's a practical problem, or just a theoretical one. --- NEWS | 14 +- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/NEWS b/NEWS index 3383ecf..31c6284 100644 --- a/NEWS +++ b/NEWS @@ -1,6 +1,18 @@ -Notmuch 0.17~rc1 (2013-11-20) +Notmuch 0.17~rc2 (2013-xx-yy) = +Incompatible change in SHA1 computation +--- + +Previously on big endian architectures like sparc and powerpc the +computation of SHA1 hashes was incorrect. This meant that messages +with overlong or missing message-ids were given different computed +message-ids than on more common little endian architectures like i386 +and amd64. If you use notmuch on a big endian architecture, you are +strongly advised to make a backup of your tags using `notmuch dump` +before this upgrade. It should be possible to migrate the tags using a +script. + Command-Line Interface -- -- 1.8.4.2
[PATCH 1/2] lib: fix byte order test in libsha1.c
From: David BremnerPreviously PLATFORM_BYTE_ORDER and IS_LITTLE_ENDIAN were not defined, so the little endian code was always compiled in. This will have the effect that the "SHA1s" on big endian architectures will change (i.e. become actual sha1s). So someone re-indexing their database could conceivable lose tags on messages without a message-id header. --- lib/libsha1.c | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/lib/libsha1.c b/lib/libsha1.c index 5d16f6a..794854b 100644 --- a/lib/libsha1.c +++ b/lib/libsha1.c @@ -49,11 +49,17 @@ extern "C" #define bswap_32(x) ((rotr32((x), 24) & 0x00ff00ff) | (rotr32((x), 8) & 0xff00ff00)) -#if (PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN) -#define bsw_32(p,n) \ -{ int _i = (n); while(_i--) ((uint32_t*)p)[_i] = bswap_32(((uint32_t*)p)[_i]); } +#ifdef __BYTE_ORDER__ +# if (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) +#define bsw_32(p,n) \ + { int _i = (n); while(_i--) ((uint32_t*)p)[_i] = bswap_32(((uint32_t*)p)[_i]); } +# elif (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) +#define bsw_32(p,n) +# else +#error "unknown byte order" +# endif #else -#define bsw_32(p,n) +#error "macro __BYTE_ORDER__ is not defined" #endif #define SHA1_MASK (SHA1_BLOCK_SIZE - 1) -- 1.8.4.2
[PATCH] Add NEWS entry for notmuch-compact
Ben Gamari writes: > Signed-off-by: Ben Gamari > --- > NEWS | 8 > 1 file changed, 8 insertions(+) pushed. d
crypto test failures on debian/kfreebsd.
0.17~rc1 is failing several crypto tests on debian/kfreebsd [1] FAIL signature verification --- crypto.2.expected 2013-11-24 04:17:13.0 + +++ crypto.2.output 2013-11-24 04:17:13.0 + @@ -11,7 +11,7 @@ "id": 2 }, { -"content-length": 315, +"content-length": 318, "content-type": "application/pgp-signature", "id": 3 } According to dkg's nifty printmimestructure script, the result computed by notmuch is actually correct. So it looks like different environments result in different content-length headers. I'm not sure if this is a bug that should be tracked down, or we should just sanitize the test data more. [1]: https://buildd.debian.org/status/fetch.php?pkg=notmuch=kfreebsd-i386=0.17%7Erc1-1=1385266740
[PATCH WIP] emacs: show: redesign unread/read logic
The decisions of when to mark messages read in notmuch-show has caused confusion/irritation (several discussions on irc and the mailing list eg the thread starting at id:87hadi0xse.fsf at boo.workgroup). This is an attempt to get some logic that people are happier with. Some examples of the current problems are: notmuch marks sometimes closed messages read, notmuch does not mark messages read if you page down through them, and notmuch removes the unread tag too soon: when you first see the message you do not if you have read it before. The patch separates out two things "seeing" a message and "marking it read". A message is deemed seen if both the top and bottom of the message have both been visible in the buffer's window. This is chosen so that just seeing 1 or 2 lines of a message at the bottom of the window does not mark it seen. A closed message is never marked seen. The seen status is updated via a command-hook (run on every command/key-press) so essentially any change which sees a message should mark it as seen. By default the unread status of seen messages is not updated until the user quits the show buffer, and the user has the option of prefix-arg quit to exit the show buffer without updating the unread status. However, if the user sets the custom variable notmuch-show-update-unread-on-seen then the unread status is updated (ie unread tag is removed) as soon as a message is seen (in the above sense). --- This patch brings the unread handling roughly in line with what I would expect, and is reasonably close to the suggestions from Austin id:20131005162202.GJ21611 at mit.edu and Jani id:87vc1aho64.fsf at nikula.org. It was also clear from the discussion that different people want different things so we will need some customisation possibilities. It is a large patch: I am afraid I don't see a way round that. At the moment there are two things that need fixing: first tree-view assumes the old behaviour so its displayed tags get out of sync with the actual tags. Secondly, a lot of tests fail for the obvious reason that the unread tag is not removed at the same time as before. It would be very helpful if people could test and see whether it works as they would like, and if not say why/when it is doing the wrong thing and what it should do in those cases. Best wishes Mark emacs/notmuch-show.el | 151 - emacs/notmuch-tree.el | 20 +-- 2 files changed, 151 insertions(+), 20 deletions(-) diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el index 784644c..1081eb0 100644 --- a/emacs/notmuch-show.el +++ b/emacs/notmuch-show.el @@ -168,6 +168,10 @@ each attachment handler is logged in buffers with names beginning \" *notmuch-part*\". This option requires emacs version at least 24.3 to work.") +(defvar notmuch-show-seen-plist nil) +(make-variable-buffer-local 'notmuch-show-seen-plist) +(put 'notmuch-show-seen-plist 'permanent-local t) + (defcustom notmuch-show-stash-mlarchive-link-alist '(("Gmane" . "http://mid.gmane.org/;) ("MARC" . "http://marc.info/?i=;) @@ -211,6 +215,15 @@ For example, if you wanted to remove an \"unread\" tag and add a :type '(repeat string) :group 'notmuch-show) +(defcustom notmuch-show-update-unread-on-seen nil + "Update unread tags when seen rathe than when exiting show buffer. + +A message is seen if the top and bottom of the message have both +been visible in the buffer. When this is nil the unread status is +updated on exiting the show buffer. When this is t the unread +status is updated as soon as the message is seen." + :type 'boolean + :group 'notmuch-show) (defmacro with-current-notmuch-show-message ( body) "Evaluate body with current buffer set to the text of current message" @@ -1142,6 +1155,8 @@ function is used." (let ((inhibit-read-only t)) (notmuch-show-mode) +(add-hook 'post-command-hook #'notmuch-show-command-hook nil t) + ;; Don't track undo information for this buffer (set 'buffer-undo-list t) @@ -1213,6 +1228,12 @@ preferences. If invoked with a prefix argument (or RESET-STATE is non-nil) then the state of the buffer (open/closed messages) is reset based on the original query." (interactive "P") + ;; Do not mark seen messages read if we are resetting state. The + ;; idea is that resetting state is asking for the view to be reset + ;; to the current state of the database. + (unless notmuch-show-update-unread-on-seen +(notmuch-show-mark-all-seen-read reset-state)) + (let ((inhibit-read-only t) (state (unless reset-state (notmuch-show-capture-state @@ -1258,6 +1279,8 @@ reset based on the original query." (defvar notmuch-show-mode-map (let ((map (make-sparse-keymap))) (set-keymap-parent map notmuch-common-keymap) + ;; the following overrides the common-keymap quit + (define-key map [remap notmuch-kill-this-buffer] 'notmuch-show-quit-and-mark-read) (define-key
notmuch sha1 implementation broken on (some) big-endian architectures
David Bremner writes: > The following code, when linked with libnotmuch.a and libutil.a does a > passable imitation of sha1sum on amd64 (and I guess also i386) but > computes a different digest on powerpc and probably sparc and s390x. > > In the long run we should maybe outsource hash computations to > e.g. librhash, but I'd like a simpler fix for 0.17, if possible Out of curiousity, I tried out a similar example with librhash, and it works fine on powerpc. #include #include "rhash.h" /* LibRHash interface */ int main(int argc, char *argv[]) { char digest[64]; char output[130]; rhash_library_init(); /* initialize static data */ int res = rhash_file(RHASH_SHA1, argv[1], digest); if(res < 0) { fprintf(stderr, "LibRHash error: %s: %s\n", argv[1], strerror(errno)); return 1; } /* convert binary digest to hexadecimal string */ rhash_print_bytes(output, digest, rhash_get_digest_size(RHASH_SHA1),RHPR_HEX); printf("%s (%s) = %s\n", rhash_get_name(RHASH_SHA1), argv[1], output); return 0; }
notmuch sha1 implementation broken on (some) big-endian architectures
The following code, when linked with libnotmuch.a and libutil.a does a passable imitation of sha1sum on amd64 (and I guess also i386) but computes a different digest on powerpc and probably sparc and s390x. In the long run we should maybe outsource hash computations to e.g. librhash, but I'd like a simpler fix for 0.17, if possible P.S. I blame Austin for adding the "missing-headers" test which found this bug ;). /* 8<- */ #include #include "notmuch.h" char * notmuch_sha1_of_file(const char* filename); int main (int argc, char **argv) { char *digest = notmuch_sha1_of_file (argv[1]); printf("%s %s\n",digest,argv[1]); return 0; }
[PATCH WIP] emacs: show: redesign unread/read logic
The decisions of when to mark messages read in notmuch-show has caused confusion/irritation (several discussions on irc and the mailing list eg the thread starting at id:87hadi0xse.fsf@boo.workgroup). This is an attempt to get some logic that people are happier with. Some examples of the current problems are: notmuch marks sometimes closed messages read, notmuch does not mark messages read if you page down through them, and notmuch removes the unread tag too soon: when you first see the message you do not if you have read it before. The patch separates out two things seeing a message and marking it read. A message is deemed seen if both the top and bottom of the message have both been visible in the buffer's window. This is chosen so that just seeing 1 or 2 lines of a message at the bottom of the window does not mark it seen. A closed message is never marked seen. The seen status is updated via a command-hook (run on every command/key-press) so essentially any change which sees a message should mark it as seen. By default the unread status of seen messages is not updated until the user quits the show buffer, and the user has the option of prefix-arg quit to exit the show buffer without updating the unread status. However, if the user sets the custom variable notmuch-show-update-unread-on-seen then the unread status is updated (ie unread tag is removed) as soon as a message is seen (in the above sense). --- This patch brings the unread handling roughly in line with what I would expect, and is reasonably close to the suggestions from Austin id:20131005162202.gj21...@mit.edu and Jani id:87vc1aho64@nikula.org. It was also clear from the discussion that different people want different things so we will need some customisation possibilities. It is a large patch: I am afraid I don't see a way round that. At the moment there are two things that need fixing: first tree-view assumes the old behaviour so its displayed tags get out of sync with the actual tags. Secondly, a lot of tests fail for the obvious reason that the unread tag is not removed at the same time as before. It would be very helpful if people could test and see whether it works as they would like, and if not say why/when it is doing the wrong thing and what it should do in those cases. Best wishes Mark emacs/notmuch-show.el | 151 - emacs/notmuch-tree.el | 20 +-- 2 files changed, 151 insertions(+), 20 deletions(-) diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el index 784644c..1081eb0 100644 --- a/emacs/notmuch-show.el +++ b/emacs/notmuch-show.el @@ -168,6 +168,10 @@ each attachment handler is logged in buffers with names beginning \ *notmuch-part*\. This option requires emacs version at least 24.3 to work.) +(defvar notmuch-show-seen-plist nil) +(make-variable-buffer-local 'notmuch-show-seen-plist) +(put 'notmuch-show-seen-plist 'permanent-local t) + (defcustom notmuch-show-stash-mlarchive-link-alist '((Gmane . http://mid.gmane.org/;) (MARC . http://marc.info/?i=;) @@ -211,6 +215,15 @@ For example, if you wanted to remove an \unread\ tag and add a :type '(repeat string) :group 'notmuch-show) +(defcustom notmuch-show-update-unread-on-seen nil + Update unread tags when seen rathe than when exiting show buffer. + +A message is seen if the top and bottom of the message have both +been visible in the buffer. When this is nil the unread status is +updated on exiting the show buffer. When this is t the unread +status is updated as soon as the message is seen. + :type 'boolean + :group 'notmuch-show) (defmacro with-current-notmuch-show-message (rest body) Evaluate body with current buffer set to the text of current message @@ -1142,6 +1155,8 @@ function is used. (let ((inhibit-read-only t)) (notmuch-show-mode) +(add-hook 'post-command-hook #'notmuch-show-command-hook nil t) + ;; Don't track undo information for this buffer (set 'buffer-undo-list t) @@ -1213,6 +1228,12 @@ preferences. If invoked with a prefix argument (or RESET-STATE is non-nil) then the state of the buffer (open/closed messages) is reset based on the original query. (interactive P) + ;; Do not mark seen messages read if we are resetting state. The + ;; idea is that resetting state is asking for the view to be reset + ;; to the current state of the database. + (unless notmuch-show-update-unread-on-seen +(notmuch-show-mark-all-seen-read reset-state)) + (let ((inhibit-read-only t) (state (unless reset-state (notmuch-show-capture-state @@ -1258,6 +1279,8 @@ reset based on the original query. (defvar notmuch-show-mode-map (let ((map (make-sparse-keymap))) (set-keymap-parent map notmuch-common-keymap) + ;; the following overrides the common-keymap quit + (define-key map [remap notmuch-kill-this-buffer] 'notmuch-show-quit-and-mark-read) (define-key map Z
notmuch sha1 implementation broken on (some) big-endian architectures
The following code, when linked with libnotmuch.a and libutil.a does a passable imitation of sha1sum on amd64 (and I guess also i386) but computes a different digest on powerpc and probably sparc and s390x. In the long run we should maybe outsource hash computations to e.g. librhash, but I'd like a simpler fix for 0.17, if possible P.S. I blame Austin for adding the missing-headers test which found this bug ;). /* 8- */ #include stdio.h #include notmuch.h char * notmuch_sha1_of_file(const char* filename); int main (int argc, char **argv) { char *digest = notmuch_sha1_of_file (argv[1]); printf(%s %s\n,digest,argv[1]); return 0; } ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: notmuch sha1 implementation broken on (some) big-endian architectures
David Bremner da...@tethera.net writes: The following code, when linked with libnotmuch.a and libutil.a does a passable imitation of sha1sum on amd64 (and I guess also i386) but computes a different digest on powerpc and probably sparc and s390x. In the long run we should maybe outsource hash computations to e.g. librhash, but I'd like a simpler fix for 0.17, if possible Out of curiousity, I tried out a similar example with librhash, and it works fine on powerpc. #include errno.h #include rhash.h /* LibRHash interface */ int main(int argc, char *argv[]) { char digest[64]; char output[130]; rhash_library_init(); /* initialize static data */ int res = rhash_file(RHASH_SHA1, argv[1], digest); if(res 0) { fprintf(stderr, LibRHash error: %s: %s\n, argv[1], strerror(errno)); return 1; } /* convert binary digest to hexadecimal string */ rhash_print_bytes(output, digest, rhash_get_digest_size(RHASH_SHA1),RHPR_HEX); printf(%s (%s) = %s\n, rhash_get_name(RHASH_SHA1), argv[1], output); return 0; } ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
crypto test failures on debian/kfreebsd.
0.17~rc1 is failing several crypto tests on debian/kfreebsd [1] FAIL signature verification --- crypto.2.expected 2013-11-24 04:17:13.0 + +++ crypto.2.output 2013-11-24 04:17:13.0 + @@ -11,7 +11,7 @@ id: 2 }, { -content-length: 315, +content-length: 318, content-type: application/pgp-signature, id: 3 } According to dkg's nifty printmimestructure script, the result computed by notmuch is actually correct. So it looks like different environments result in different content-length headers. I'm not sure if this is a bug that should be tracked down, or we should just sanitize the test data more. [1]: https://buildd.debian.org/status/fetch.php?pkg=notmucharch=kfreebsd-i386ver=0.17%7Erc1-1stamp=1385266740 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Add NEWS entry for notmuch-compact
Ben Gamari bgamari.f...@gmail.com writes: Signed-off-by: Ben Gamari bgamari.f...@gmail.com --- NEWS | 8 1 file changed, 8 insertions(+) pushed. d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: notmuch sha1 implementation broken on (some) big-endian architectures
On Sun, Nov 24 2013, David Bremner da...@tethera.net wrote: The following code, when linked with libnotmuch.a and libutil.a does a passable imitation of sha1sum on amd64 (and I guess also i386) but computes a different digest on powerpc and probably sparc and s390x. In the long run we should maybe outsource hash computations to e.g. librhash, but I'd like a simpler fix for 0.17, if possible P.S. I blame Austin for adding the missing-headers test which found this bug ;). This is interesting problem, I would have guessed that this would fails on LITTLE_ENDIAN machines easier, if ever... ... especially as there is line lib/libsha1.c:52:#if (PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN) ... but... I could not find any (other) matches for PLATFORM_BYTE_ORDER nor IS_LITTLE_ENDIAN in source code or in /usr/include/**/*.h or in /usr/lib/gcc/**/*.h I did some testing and it seems that #if (F == BBBAAARRR) code #endif will have code in output file in case neither of the above are defined... :/ So, this could work: #if endian.h // for BYTE_ORDER LITTLE_ENDIAN and then #if (BYTE_ORDER == LITTLE_ENDIAN) ... to replace lib/libsha1.c:53 (53 now after endian.h added) Please test on BIG_ENDIAN machine... In case this works, then we'd need to inform users that their long/missing Message ID:s are now coded differently in their databases... Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] new: Detect dirent.d_type support at configure time
On Sun, Nov 24 2013, Austin Clements amdra...@mit.edu wrote: Support for dirent.d_type is OS-specific. Previously, we used _DIRENT_HAVE_D_TYPE to detect support for this, but this is apparently a glic-ism (FreeBSD, for example, supports d_type, but does not define this). Since there's no cross-platform way to detect support for dirent.d_type, detect it using a test compile at configure time. --- LGTM. Tomi compat/have_d_type.c | 10 ++ configure| 16 notmuch-new.c| 2 +- 3 files changed, 27 insertions(+), 1 deletion(-) create mode 100644 compat/have_d_type.c diff --git a/compat/have_d_type.c b/compat/have_d_type.c new file mode 100644 index 000..9ca6c6e --- /dev/null +++ b/compat/have_d_type.c @@ -0,0 +1,10 @@ +#include dirent.h + +int main() +{ +struct dirent ent; + +(void) ent.d_type; + +return 0; +} diff --git a/configure b/configure index 1a8e939..d2d193c 100755 --- a/configure +++ b/configure @@ -557,6 +557,17 @@ else fi rm -f compat/have_timegm +printf Checking for dirent.d_type... +if ${CC} -o compat/have_d_type $srcdir/compat/have_d_type.c /dev/null 21 +then +printf Yes.\n +have_d_type=1 +else +printf No (will use stat instead).\n +have_d_type=0 +fi +rm -f compat/have_d_type + printf Checking for standard version of getpwuid_r... if ${CC} -o compat/check_getpwuid $srcdir/compat/check_getpwuid.c /dev/null 21 then @@ -745,6 +756,9 @@ HAVE_STRCASESTR = ${have_strcasestr} # build its own version) HAVE_STRSEP = ${have_strsep} +# Whether struct dirent has d_type (if not, then notmuch will use stat) +HAVE_D_TYPE = ${have_d_type} + # Whether the Xapian version in use supports compaction HAVE_XAPIAN_COMPACT = ${have_xapian_compact} @@ -805,6 +819,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\ \$(VALGRIND_CFLAGS) \\ -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\ -DHAVE_STRSEP=\$(HAVE_STRSEP) \\ +-DHAVE_D_TYPE=\$(HAVE_D_TYPE) \\ -DSTD_GETPWUID=\$(STD_GETPWUID) \\ -DSTD_ASCTIME=\$(STD_ASCTIME) \\ -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT) @@ -813,6 +828,7 @@ CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\ \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\ -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\ -DHAVE_STRSEP=\$(HAVE_STRSEP) \\ + -DHAVE_D_TYPE=\$(HAVE_D_TYPE) \\ -DSTD_GETPWUID=\$(STD_GETPWUID) \\ -DSTD_ASCTIME=\$(STD_ASCTIME) \\ -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT) diff --git a/notmuch-new.c b/notmuch-new.c index ba05cb4..423e188 100644 --- a/notmuch-new.c +++ b/notmuch-new.c @@ -167,7 +167,7 @@ dirent_type (const char *path, const struct dirent *entry) char *abspath; int err, saved_errno; -#ifdef _DIRENT_HAVE_D_TYPE +#if HAVE_D_TYPE /* Mapping from d_type to stat mode_t. We omit DT_LNK so that * we'll fall through to stat and get the real file type. */ static const mode_t modes[] = { -- 1.8.4.rc3 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 1/2] lib: fix byte order test in libsha1.c
From: David Bremner da...@tethera.net Previously PLATFORM_BYTE_ORDER and IS_LITTLE_ENDIAN were not defined, so the little endian code was always compiled in. This will have the effect that the SHA1s on big endian architectures will change (i.e. become actual sha1s). So someone re-indexing their database could conceivable lose tags on messages without a message-id header. --- lib/libsha1.c | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/lib/libsha1.c b/lib/libsha1.c index 5d16f6a..794854b 100644 --- a/lib/libsha1.c +++ b/lib/libsha1.c @@ -49,11 +49,17 @@ extern C #define bswap_32(x) ((rotr32((x), 24) 0x00ff00ff) | (rotr32((x), 8) 0xff00ff00)) -#if (PLATFORM_BYTE_ORDER == IS_LITTLE_ENDIAN) -#define bsw_32(p,n) \ -{ int _i = (n); while(_i--) ((uint32_t*)p)[_i] = bswap_32(((uint32_t*)p)[_i]); } +#ifdef __BYTE_ORDER__ +# if (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) +#define bsw_32(p,n) \ + { int _i = (n); while(_i--) ((uint32_t*)p)[_i] = bswap_32(((uint32_t*)p)[_i]); } +# elif (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) +#define bsw_32(p,n) +# else +#error unknown byte order +# endif #else -#define bsw_32(p,n) +#error macro __BYTE_ORDER__ is not defined #endif #define SHA1_MASK (SHA1_BLOCK_SIZE - 1) -- 1.8.4.2 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 2/2] NEWS: News for big endian sha1 bug fix.
From: David Bremner da...@tethera.net We could give more details about how to migrate tags, but I'm not sure that it's a practical problem, or just a theoretical one. --- NEWS | 14 +- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/NEWS b/NEWS index 3383ecf..31c6284 100644 --- a/NEWS +++ b/NEWS @@ -1,6 +1,18 @@ -Notmuch 0.17~rc1 (2013-11-20) +Notmuch 0.17~rc2 (2013-xx-yy) = +Incompatible change in SHA1 computation +--- + +Previously on big endian architectures like sparc and powerpc the +computation of SHA1 hashes was incorrect. This meant that messages +with overlong or missing message-ids were given different computed +message-ids than on more common little endian architectures like i386 +and amd64. If you use notmuch on a big endian architecture, you are +strongly advised to make a backup of your tags using `notmuch dump` +before this upgrade. It should be possible to migrate the tags using a +script. + Command-Line Interface -- -- 1.8.4.2 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 2/2] NEWS: News for big endian sha1 bug fix.
Quoth da...@tethera.net on Nov 24 at 5:29 pm: From: David Bremner da...@tethera.net We could give more details about how to migrate tags, but I'm not sure that it's a practical problem, or just a theoretical one. --- NEWS | 14 +- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/NEWS b/NEWS index 3383ecf..31c6284 100644 --- a/NEWS +++ b/NEWS @@ -1,6 +1,18 @@ -Notmuch 0.17~rc1 (2013-11-20) +Notmuch 0.17~rc2 (2013-xx-yy) = +Incompatible change in SHA1 computation +--- + +Previously on big endian architectures like sparc and powerpc the +computation of SHA1 hashes was incorrect. This meant that messages +with overlong or missing message-ids were given different computed +message-ids than on more common little endian architectures like i386 +and amd64. If you use notmuch on a big endian architecture, you are +strongly advised to make a backup of your tags using `notmuch dump` +before this upgrade. It should be possible to migrate the tags using a +script. + Should this mention how to find such messages? Something like notmuch dump | awk '/^notmuch-sha1-[0-9a-f]{40} / {system(notmuch search id: $1)}' ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch