Re: Mummi wishlist: API using Message-ID

[Arun Isaac]

> As long as the script shows it's trying and explains why it takes
> time, it should be fine. It could offer a --continue option too :)

Yeah, my thoughts exactly. Definitely beats the two-step manual process
that we have to endure now.

Re: Mummi wishlist: API using Message-ID

[Julien Lepiller]

As long as the script shows it's trying and explains why it takes time, it 
should be fine. It could offer a --continue option too :)

On June 6, 2022 2:03:16 PM GMT+02:00, Ricardo Wurmus  wrote:
>
>Arun Isaac  writes:
>
>>> Once delpoyed to issues.guix.gnu.org you can visit
>>>
>>> https://issues.guix.gnu.org/msgid/HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link
>>
>> This is great news! Using this, we should be able to implement a `mumi
>> send-email' command to easily send N patches to a new issue. Here is
>> how `mumi send-email' might be implemented:
>>
>> 1. Generate patches with `git format-patch --thread' so that there is a
>>Message-ID header.
>>
>> 2. Send the first patch to guix-patc...@gnu.org.
>>
>> 3. Poll https://issues.guix.gnu.org/ to see if the first
>>patch has reached Debbugs/Mumi. Find the issue number of the new
>>issue that was created.
>>
>> 4. Send the remaining patches to the new issue.
>
>This sounds like it could take quite some time, and the work it performs
>is not transactional, so an impatient person cancelling it before
>completion could end up with a bunch of “initial” emails without ever
>sending the rest of the patches.
>
>I think that maybe we should wean mumi off of debbugs and operate on
>received email directly (using debbugs as a storage backend for the time
>being).
>
>-- 
>Ricardo
>

Re: Mummi wishlist: API using Message-ID

[Arun Isaac]

> This sounds like it could take quite some time, and the work it performs
> is not transactional, so an impatient person cancelling it before
> completion could end up with a bunch of “initial” emails without ever
> sending the rest of the patches.

True.

> I think that maybe we should wean mumi off of debbugs and operate on
> received email directly (using debbugs as a storage backend for the time
> being).

Weaning mumi off of debbugs is definitely the better approach, but it
would take more mork, I imagine.

Re: Mummi wishlist: API using Message-ID

[Ricardo Wurmus]

Arun Isaac  writes:

>> Once delpoyed to issues.guix.gnu.org you can visit
>>
>> https://issues.guix.gnu.org/msgid/HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link
>
> This is great news! Using this, we should be able to implement a `mumi
> send-email' command to easily send N patches to a new issue. Here is
> how `mumi send-email' might be implemented:
>
> 1. Generate patches with `git format-patch --thread' so that there is a
>Message-ID header.
>
> 2. Send the first patch to guix-patc...@gnu.org.
>
> 3. Poll https://issues.guix.gnu.org/ to see if the first
>patch has reached Debbugs/Mumi. Find the issue number of the new
>issue that was created.
>
> 4. Send the remaining patches to the new issue.

This sounds like it could take quite some time, and the work it performs
is not transactional, so an impatient person cancelling it before
completion could end up with a bunch of “initial” emails without ever
sending the rest of the patches.

I think that maybe we should wean mumi off of debbugs and operate on
received email directly (using debbugs as a storage backend for the time
being).

-- 
Ricardo

Re: Mummi wishlist: API using Message-ID

[Arun Isaac]

> Once delpoyed to issues.guix.gnu.org you can visit
>
> https://issues.guix.gnu.org/msgid/HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link

This is great news! Using this, we should be able to implement a `mumi
send-email' command to easily send N patches to a new issue. Here is
how `mumi send-email' might be implemented:

1. Generate patches with `git format-patch --thread' so that there is a
   Message-ID header.

2. Send the first patch to guix-patc...@gnu.org.

3. Poll https://issues.guix.gnu.org/ to see if the first
   patch has reached Debbugs/Mumi. Find the issue number of the new
   issue that was created.

4. Send the remaining patches to the new issue.

Regards,
Arun

Re: Mummi wishlist: API using Message-ID

[zimoun]

Hi Ricardo,

On Sat, 04 Jun 2022 at 13:00, Ricardo Wurmus  wrote:

> This is now implemented in mumi.

Really cool!  Thank you!

The bridge between tools becomes easier.  For instance, I use the Emacs
frontend of Notmuch where ’cI’ allow to stash the Message-ID, then I can
directly paste it for referencing.


Cheers,
simon

Re: Mummi wishlist: API using Message-ID

[Ricardo Wurmus]

Ricardo Wurmus  writes:

> zimoun  writes:
>
>> For example .  The number #16 is
>> really difficult to guess. :-)
>>
>> Instead, it is easy to get the Message-ID.  (Using emacs-notmuch, only
>> hit the key ’ci’) Therefore, it could be nice to be able to provide
>> e.g., the URL:
>>
>> 
>>
>> redirecting to .  And maybe the
>> current webpage could provide the Message-ID, easy to copy and then
>> paste in my email reader.
>
> This is now implemented in mumi.
>
> Once delpoyed to issues.guix.gnu.org you can visit
>
> https://issues.guix.gnu.org/msgid/HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link
>
> and it would redirect to https://issues.guix.gnu.org/33899.  It doesn’t
> yet jump to the correct message number, but that’s easy to implement.
> (I only had time for one little hack.)

Mumi commit 9b28ec7d152623692877bcb767e5c654e59e57ed adds an anchor for
the message id and redirects /msgid URLs to issue URL + message id
fragment.

-- 
Ricardo

Re: Mummi wishlist: API using Message-ID

[Ricardo Wurmus]

zimoun  writes:

> For example .  The number #16 is
> really difficult to guess. :-)
>
> Instead, it is easy to get the Message-ID.  (Using emacs-notmuch, only
> hit the key ’ci’) Therefore, it could be nice to be able to provide
> e.g., the URL:
>
> 
>
> redirecting to .  And maybe the
> current webpage could provide the Message-ID, easy to copy and then
> paste in my email reader.

This is now implemented in mumi.

Once delpoyed to issues.guix.gnu.org you can visit

https://issues.guix.gnu.org/msgid/HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link

and it would redirect to https://issues.guix.gnu.org/33899.  It doesn’t
yet jump to the correct message number, but that’s easy to implement.
(I only had time for one little hack.)

-- 
Ricardo

Re: Mummi wishlist: API using Message-ID

[Kyle Meyer]

Kyle Meyer writes:

> For public-inbox URLs, there is one character that needs to be escaped:
> "/" with "%2F" (see ).

Correcting myself: I shouldn't take that help page as a complete
description of what needs to be escaped for public-inbox URLs.  It'd be
better to look at what public-inbox's code does itself, particularly
when it generates the path component of the URL for its Archived-At
header.  It feeds the message ID through this function:

,[ lib/PublicInbox/MID.pm ]
| # RFC3986, section 3.3:
| sub MID_ESC () { '^A-Za-z0-9\-\._~!\$\&\';\(\)\*\+,;=:@' }
| sub mid_escape ($) { uri_escape_utf8($_[0], MID_ESC) }
`

That code came with 9d1e5fad (www: do not unecessarily escape some chars
in paths, 2016-08-14).

So, if you want to perform the same escaping for your custom Elisp
function that generates public-inbox URLs, you can do something like
this:

(let ((unreserved-chars
   (append url-unreserved-chars
   '(?! ?$ ?& ?' ?\( ?\) ?* ?+ ?, ?= ?: ?@ ?\;
  (url-hexify-string "8...@e.com" unreserved-chars)  ; => "8...@e.com"
  (url-hexify-string "8...@e.com" unreserved-chars)  ; => "8...@e.com"
  (url-hexify-string "8/@e.com" unreserved-chars)) ; => "8...@e.com"

Or, if you don't mind the unnecessary escaping (particularly of "@"),
you can just use url-hexify-string with the default set of unreserved
characters.

Re: Mummi wishlist: API using Message-ID

[Arun Isaac]

> I just don’t know for sure if Message-Ids may contain characters that
> are not permitted in URLs.

According to RFC5322, Message-IDs can contain any ASCII character in the
(decimal) ranges 33-91, 93-126. And, many of these characters are not
permitted in URLs, but perhaps modern browsers automatically percent
encode if the user enters unpermitted characters in the URL.


signature.asc
Description: PGP signature

Re: Mummi wishlist: API using Message-ID

[Kyle Meyer]

zimoun writes:

> On Thu, 17 Sep 2020 at 12:00, Ricardo Wurmus  wrote:

>> This is an interesting idea!  I don’t know if it’ll work as a plain URL,
>> because not all characters of a message id might be usable in a URL (you
>> may need to URL-encode it and can’t just paste it in the URL bar of your
>> browser), but it would certainly work as a search query.  The only
>> problem is that we’re not currently indexing the message ids.
>
> I do not know but public-inbox is doing such thing, isn't it?
> I mean, the example with emacs-debbugs, my "custom" function and the
> service :
>
[...]
> Is it not enough?
> The URL,  is it not always ready-to-be-pasted?
> The link contains '@' but it works (on my webbrowser at least :-)).

For public-inbox URLs, there is one character that needs to be escaped:
"/" with "%2F" (see ).

However, in its current form, your custom function has a good chance of
being enough.  Here is the number of message IDs known to
yhetil.org/guix-devel:

$ echo 'SELECT COUNT(*) FROM msgmap' | \
sqlite3 -readonly guix-devel-msgmap.sqlite3
55231

And of those, only 11 have "/" in them:

$ echo 'SELECT mid FROM msgmap' | \
sqlite3 -readonly guix-devel-msgmap.sqlite3 | grep -c "/"
11

Re: Mummi wishlist: API using Message-ID

[Ricardo Wurmus]

zimoun  writes:

> (Sorry for the misspelling of Mumi in the Subject thread.)
>
> (For the record, reading [1], I discovered [2,3] with similar goals
> than Mumi and that the Org-mode community uses public-inbox [4].)
>
> [1] https://bzg.fr/en/org-mode-9.4-is-out-can-you-help.html/
> [2] https://updates.orgmode.org/
> [3] https://github.com/bzg/woof
> [4] https://orgmode.org/list/87r1vd92eg@bzg.fr/
>
> On Thu, 17 Sep 2020 at 12:00, Ricardo Wurmus  wrote:
>> zimoun  writes:
>
>> > Instead, it is easy to get the Message-ID.  (Using emacs-notmuch, only
>> > hit the key ’ci’) Therefore, it could be nice to be able to provide
>> > e.g., the URL:
>> >
>> > 
>> >
>> > redirecting to .
>>
>> This is an interesting idea!  I don’t know if it’ll work as a plain URL,
>> because not all characters of a message id might be usable in a URL (you
>> may need to URL-encode it and can’t just paste it in the URL bar of your
>> browser), but it would certainly work as a search query.  The only
>> problem is that we’re not currently indexing the message ids.
>
> I do not know but public-inbox is doing such thing, isn't it?
> I mean, the example with emacs-debbugs, my "custom" function and the
> service : […]

It might be fine.  I just don’t know for sure if Message-Ids may contain
characters that are not permitted in URLs.

>> > And maybe the
>> > current webpage could provide the Message-ID, easy to copy and then
>> > paste in my email reader.
>>
>> It’s already there but hidden!  Every message is rendered in a
>> div.message.  Inside of that is div.card > div.card-header > div.details
>> (hidden) > div.message-id.
>>
>> With custom CSS you could unhide div.details, and with a custom JS you
>> could grab the contents of div.message-id on the click of a button.
>> Most browsers make it a little hard to augment the CSS and/or JS of a
>> served page, but with a little hacking I’m sure you can extract what you
>> want.
>
> Okish... CSS/JS is not really my cup of tea.  I will try to rehash
> your words when giving a look at Mumi source.

You can give this a try:

* in Icecat hit F12
* select the Inspector (if it isn’t selected yet)
* in the right column (next to :hov and .cls) there’s a plus icon with
  hover text “Add new rule”; click it
* edit the text to say: “.details { display: block !important }”

This should show you all message ids.  It’s possible to make this
permanent but it’s rather tedious with Icecat/Firefox.

If you just need this for scripting, though, you can parse the HTML and
filter the “div” tag with the “message-id” class.  If scripting is
desired I could probably also offer a JSON endpoint that provides all
information in a format that is easily processed.

-- 
Ricardo

Re: Mummi wishlist: API using Message-ID

[zimoun]

(Sorry for the misspelling of Mumi in the Subject thread.)

(For the record, reading [1], I discovered [2,3] with similar goals
than Mumi and that the Org-mode community uses public-inbox [4].)

[1] https://bzg.fr/en/org-mode-9.4-is-out-can-you-help.html/
[2] https://updates.orgmode.org/
[3] https://github.com/bzg/woof
[4] https://orgmode.org/list/87r1vd92eg@bzg.fr/

On Thu, 17 Sep 2020 at 12:00, Ricardo Wurmus  wrote:
> zimoun  writes:

> > Instead, it is easy to get the Message-ID.  (Using emacs-notmuch, only
> > hit the key ’ci’) Therefore, it could be nice to be able to provide
> > e.g., the URL:
> >
> > 
> >
> > redirecting to .
>
> This is an interesting idea!  I don’t know if it’ll work as a plain URL,
> because not all characters of a message id might be usable in a URL (you
> may need to URL-encode it and can’t just paste it in the URL bar of your
> browser), but it would certainly work as a search query.  The only
> problem is that we’re not currently indexing the message ids.

I do not know but public-inbox is doing such thing, isn't it?
I mean, the example with emacs-debbugs, my "custom" function and the
service :

> > Another (Emacs related) example is emacs-debbugs.
> >   C-u M-x debbugs-gnu RET guix RET n y
> >   M-x debbugs-gnu-bugs RET 33899 RET RET
> >   Read messages
> >
> > Well, in the relevant message, I run “M-x debbugs->url” and get the
> > ready-to-be-pasted URL:
> >
> > 
> >
> > with the naive Emacs function:
> >
> > --8<---cut here---start->8---
> > (defun debbugs->url ()
> >   (interactive)
> >   (let ((url "https://yhetil.org/guix-bugs/;)
> > mid)
> > (with-current-buffer gnus-article-buffer
> >   (gnus-summary-toggle-header 1)
> >   (setq mid (message-fetch-field "Message-ID"))
> >   (gnus-summary-toggle-header -1))
> > (while (string-match "[<>]" mid)
> >   (setq mid (replace-match "" t t mid)))
> > (kill-new (concat url mid))
> > (message mid)))
> > --8<---cut here---end--->8---

Is it not enough?
The URL,  is it not always ready-to-be-pasted?
The link contains '@' but it works (on my webbrowser at least :-)).


> Per bug we index the bug id, the submitter, the authors, the subject(s),
> the owner, the severity, the tags, the status, the file name of the
> original Debbugs “.log” file (I forgot why!), and all the message
> bodies.  All of these map to the very same “document” (in Xapian
> parlance), which is a particular “issue”.

Thank you for explaining.


> I think we could also index the message id (and expose it to the
> search), map that to the “issue”, and then find the corresponding anchor
> in the HTML later.  We wouldn’t be able to *directly* map the message id
> to the actual message in the thread, because the search does not operate
> on the concept of messages but only whole bug discussions.  But I think
> this would get us very close to what you propose.

Yes.
If the Message-ID is also indexed, the map is almost there.


> > And maybe the
> > current webpage could provide the Message-ID, easy to copy and then
> > paste in my email reader.
>
> It’s already there but hidden!  Every message is rendered in a
> div.message.  Inside of that is div.card > div.card-header > div.details
> (hidden) > div.message-id.
>
> With custom CSS you could unhide div.details, and with a custom JS you
> could grab the contents of div.message-id on the click of a button.
> Most browsers make it a little hard to augment the CSS and/or JS of a
> served page, but with a little hacking I’m sure you can extract what you
> want.

Okish... CSS/JS is not really my cup of tea.  I will try to rehash
your words when giving a look at Mumi source.


> > Last and off-topic to the Subject :-), the tool ’public-inbox’ exposes
> > such API and then it seems easier to refer to relevant message than
> > going by hand on e.g. 
> > to find it.
>
> I don’t quite understand how this differs from this:
>
> https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=%3C86sgbhz3fe.fsf%40gmail.com%3E=Search%21=guix-devel

Because I did not know. :-)


All the best,
simon

Re: Mummi wishlist: API using Message-ID

[Ricardo Wurmus]

zimoun  writes:

> For example .  The number #16 is
> really difficult to guess. :-)

Yes, it comes from the number of messages we got from Debbugs.

> Instead, it is easy to get the Message-ID.  (Using emacs-notmuch, only
> hit the key ’ci’) Therefore, it could be nice to be able to provide
> e.g., the URL:
>
> 
>
> redirecting to .

This is an interesting idea!  I don’t know if it’ll work as a plain URL,
because not all characters of a message id might be usable in a URL (you
may need to URL-encode it and can’t just paste it in the URL bar of your
browser), but it would certainly work as a search query.  The only
problem is that we’re not currently indexing the message ids.

Per bug we index the bug id, the submitter, the authors, the subject(s),
the owner, the severity, the tags, the status, the file name of the
original Debbugs “.log” file (I forgot why!), and all the message
bodies.  All of these map to the very same “document” (in Xapian
parlance), which is a particular “issue”.

I think we could also index the message id (and expose it to the
search), map that to the “issue”, and then find the corresponding anchor
in the HTML later.  We wouldn’t be able to *directly* map the message id
to the actual message in the thread, because the search does not operate
on the concept of messages but only whole bug discussions.  But I think
this would get us very close to what you propose.

> And maybe the
> current webpage could provide the Message-ID, easy to copy and then
> paste in my email reader.

It’s already there but hidden!  Every message is rendered in a
div.message.  Inside of that is div.card > div.card-header > div.details
(hidden) > div.message-id.

With custom CSS you could unhide div.details, and with a custom JS you
could grab the contents of div.message-id on the click of a button.
Most browsers make it a little hard to augment the CSS and/or JS of a
served page, but with a little hacking I’m sure you can extract what you
want.

>
> Another (Emacs related) example is emacs-debbugs.
>   C-u M-x debbugs-gnu RET guix RET n y
>   M-x debbugs-gnu-bugs RET 33899 RET RET
>   Read messages
>
> Well, in the relevant message, I run “M-x debbugs->url” and get the
> ready-to-be-pasted URL:
>
> 
>
> with the naive Emacs function:
>
> --8<---cut here---start->8---
> (defun debbugs->url ()
>   (interactive)
>   (let ((url "https://yhetil.org/guix-bugs/;)
> mid)
> (with-current-buffer gnus-article-buffer
>   (gnus-summary-toggle-header 1)
>   (setq mid (message-fetch-field "Message-ID"))
>   (gnus-summary-toggle-header -1))
> (while (string-match "[<>]" mid)
>   (setq mid (replace-match "" t t mid)))
> (kill-new (concat url mid))
> (message mid)))
> --8<---cut here---end--->8---
>
>
>
> Last and off-topic to the Subject :-), the tool ’public-inbox’ exposes
> such API and then it seems easier to refer to relevant message than
> going by hand on e.g. 
> to find it.

I don’t quite understand how this differs from this:

https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=%3C86sgbhz3fe.fsf%40gmail.com%3E=Search%21=guix-devel

Here I search for the id of your message (the one I’m responding to);
that’s the (URL-encoded) value of “query”, “idxname” is for the mailing
list to be searched.

It gives you a list of results, not one message directly, but I think
it’s pretty close. 

-- 
Ricardo

Mummi wishlist: API using Message-ID

[zimoun]

Dear,

Today  is really useful and IMHO better than the
classic Debbugs interface [1].

Bug reports (and patches) are still sent by email, the core of the
Debbugs bug tracker.  However, Mummi does not use the unique identifier
Message-ID as address, AFAICT.  Therefore, it can be tricky to link
between my emails reader/writer and the web interface.

Let take an example.  From my typical use case. :-)

My subscription to bug-guix (or guix-patches) sends me emails and I read
them with emacs-notmuch (but it is not Emacs related, since any good
email reader seems able to do the same).

When I want to refer to one specific message, I have to open my web
browser and go to  then scroll a bit to find
the relevant message, click to the date to get the associated number,
copy the URL and go back to my email writer and finally paste the link.

For example .  The number #16 is
really difficult to guess. :-)

Instead, it is easy to get the Message-ID.  (Using emacs-notmuch, only
hit the key ’ci’) Therefore, it could be nice to be able to provide
e.g., the URL:



redirecting to .  And maybe the
current webpage could provide the Message-ID, easy to copy and then
paste in my email reader.


Another (Emacs related) example is emacs-debbugs.
  C-u M-x debbugs-gnu RET guix RET n y
  M-x debbugs-gnu-bugs RET 33899 RET RET
  Read messages

Well, in the relevant message, I run “M-x debbugs->url” and get the
ready-to-be-pasted URL:



with the naive Emacs function:

--8<---cut here---start->8---
(defun debbugs->url ()
  (interactive)
  (let ((url "https://yhetil.org/guix-bugs/;)
mid)
(with-current-buffer gnus-article-buffer
  (gnus-summary-toggle-header 1)
  (setq mid (message-fetch-field "Message-ID"))
  (gnus-summary-toggle-header -1))
(while (string-match "[<>]" mid)
  (setq mid (replace-match "" t t mid)))
(kill-new (concat url mid))
(message mid)))
--8<---cut here---end--->8---



Last and off-topic to the Subject :-), the tool ’public-inbox’ exposes
such API and then it seems easier to refer to relevant message than
going by hand on e.g. 
to find it.

An unofficial (but almost? serving org-mode list [2]) example is
demonstrated here:

   

Does it make to sense to have something similar,
e.g.  serving the ’public-inbox’?


[1] 
https://debbugs.gnu.org/cgi/pkgreport.cgi?package=guix;max-bugs=100;base-order=1;bug-rev=1
[2] 


All the best,
simon