Re: Does Gnus read and sync the read with IMAP?
Michael writes: > AFAIU Gnus stores such stuff in a file named ".newsrc.eld" (AFAIR > ".newsrc.el" in older Emacs versions). Looks like the 'd' has been there at least since 1997; gnus-start.el: eec82323c2e3 (Lars Ingebrigtsen 1997-04-16 2768) (setq buffer-file-name eec82323c2e3 (Lars Ingebrigtsen 1997-04-16 2769)(concat gnus-current-startup-file ".eld")) and before that; gnus.el: aa088210dd (Lars Magne Ingebrigtsen 1997-03-04 01:26:40 + 9645) (write-region 1 (point-max) aa088210dd (Lars Magne Ingebrigtsen 1997-03-04 01:26:40 + 9646) (concat gnus-current-startup-file ".eld") aa088210dd (Lars Magne Ingebrigtsen 1997-03-04 01:26:40 + 9647) nil 'nomesg)) which is as far back the git repository I had lying around seems to go: commit aa088210dd9c90f00466b59c58f60f14b4544e69 (HEAD, tag: v0-5) Author: Lars Magne Ingebrigtsen Date: Tue Mar 4 01:26:40 1997 + Initial revision :-), Adam -- "Do you think the people who make these decisions Adam Sjøgren about what features a project will have are the a...@koldfront.dk *engineers*? Because that's adorable."
Re: Does Gnus read and sync the read with IMAP?
Michael Heerdegen wrote: > AFAIU Gnus stores such stuff in a file named ".newsrc.eld" (AFAIR > ".newsrc.el" in older Emacs versions). And indeed, among other things > there seems to be information about "seen" status of messages in that > file, as well as saved status, ticked and dormant status, all of that > Gnus specific stuff, but nothing related to the "read" (or "old") > status. So I guess the "read" status is fetched every time when you > open the group, along with the rest (list of existing messages etc). Just FYI: For those who use the agent, (info "(gnus) Agent and flags") is also relevant in this regard. --
Re: Does Gnus read and sync the read with IMAP?
Björn Bidar writes: > Dan Christensen writes: > >> On Oct 20, 2023, Eric Abrahamsen wrote: >> >>> Dan Christensen writes: >>> Here's a followup question: how can I make Gnus update the marks for *all* IMAP groups. >>> >>> I once asked Lars about this -- since the IMAP server should be the >>> source of truth for all information about the messages, why does >>> Gnus >>> store its own marks data for IMAP groups at all? Why not just > > How many imap server support imap metadata? Clients such as Thunderbird > already safe their metadata in imap. > >>> refresh >>> from the server at each startup? I think he said that for many large >>> groups that would end up being prohibitively slow. >> >> I agree that it would be too slow for many people, especially with >> remote IMAP servers. So I'd like an optional setting. But I don't >> think it should have anything to do with Gnus shutdown and startup. > > Why would it be to slow? Other mail clients fetch the read > status for each mail when they update the state of each > imap mailbox. I have never tried it, so I don't know if it would be slow or not. But this wouldn't just update the read status, this would remove all existing marks and refresh them from the server: ticks, replied, forwarded, everything. I could imagine that being pretty slow.
Re: Does Gnus read and sync the read with IMAP?
Dan Christensen writes: > On Oct 20, 2023, Eric Abrahamsen wrote: > >> Dan Christensen writes: >> >>> Here's a followup question: how can I make Gnus update the marks >>> for >>> *all* IMAP groups. >> >> I once asked Lars about this -- since the IMAP server should be the >> source of truth for all information about the messages, why does >> Gnus >> store its own marks data for IMAP groups at all? Why not just How many imap server support imap metadata? Clients such as Thunderbird already safe their metadata in imap. >> refresh >> from the server at each startup? I think he said that for many large >> groups that would end up being prohibitively slow. > > I agree that it would be too slow for many people, especially with > remote IMAP servers. So I'd like an optional setting. But I don't > think it should have anything to do with Gnus shutdown and startup. Why would it be to slow? Other mail clients fetch the read status for each mail when they update the state of each imap mailbox. > *Every* time I hit `g', I want the marks to be refreshed from the IMAP > server. Presumably this is the same code that runs when Gnus starts > up, > so it would improve the behaviour at startup as well. > > Dan
Re: Does Gnus read and sync the read with IMAP?
Bob Newell writes: >> 3. But here's the big one, and one that I thought I understood >> but seem to get inconsistent results. At what point does gnus >> update the IMAP server? In other words, I read an article. >> In the summary buffer that article definitely is now shown as >> read. But that doesn't propagate instantly to the IMAP >> server. I can see this by (say) opening a gmail web interface >> and noting that the article I just read on gnus is still >> unread on gmail web. (I will try a few test cases on my own >> to see what I can learn.) > > Although this is getting pretty gmail specific, it makes some > sense. To gmail, INBOX and All Mail are simply labels and an > email is not actually moved from one to the other; /all/ mail > /always/ exists in All Mail and 'moving' from INBOX simply > means removing the INBOX label. (The exception is Trash and > Spam.) > These folders also exist in other imap servers such as dovecot. I'm not sure if they are standard but dovecot creates them by grouping all folders with for example the seen flag. > But gnus treats these as actual groups (call them folders if > you wish). Now, when the email is moved out of INBOX to All > Mail, and you exit INBOX, I'm guessing that the All Mail > 'read' status isn't updated because gnus views that as a truly > separate group. I know GNUS can create virtual groups but I don't think that's transferred of to these Groups coming from Imap folders. https://doc.dovecot.org/configuration_manual/namespace/#core_setting-namespace/mailbox/special_use Based upon RFC 6154. > I think things are a lot simpler with traditional email which > has true IMAP folders. But as a practical matter, if a few > emails are shown as unread, whether in gnus or another client, > it isn't a show-stopper. It's not about inconsistency is an issue I think.
Re: Does Gnus read and sync the read with IMAP?
Bob writes: > At what point does gnus update the IMAP server? In other words, I > read an article. In the summary buffer that article definitely is > now shown as read. Gnus updates the IMAP server when you exit the summary buffer (q). To find out exactly what happens when, you can set nnimap-record-commands in your nnimap server configuration, then the commands will be logged in a buffer called "*imap log*". Best regards, Adam -- "None of them kicks go boomAdam Sjøgren None of them basslines fill the room"a...@koldfront.dk
Re: Does Gnus read and sync the read with IMAP?
> 3. But here's the big one, and one that I thought I understood > but seem to get inconsistent results. At what point does gnus > update the IMAP server? In other words, I read an article. > In the summary buffer that article definitely is now shown as > read. But that doesn't propagate instantly to the IMAP > server. I can see this by (say) opening a gmail web interface > and noting that the article I just read on gnus is still > unread on gmail web. (I will try a few test cases on my own > to see what I can learn.) Appendix: At least with gmail, the IMAP server appears to be pretty consistently updated with 'read' status from gnus when leaving the group. So (say) q-uitting the INBOX group updates the INBOX status on the server, as evidenced by the web interface. But there are still some inconsistencies with the All Mail folder. To wit, if I read an email in gnus INBOX, and then archive it to the All Mail folder (from gnus) /before/ I exit INBOX, then the mail remains 'unread' in All Mail on the web interface. Furthermore if I reload All Mail on gnus, that same email shows up now as unread. Although this is getting pretty gmail specific, it makes some sense. To gmail, INBOX and All Mail are simply labels and an email is not actually moved from one to the other; /all/ mail /always/ exists in All Mail and 'moving' from INBOX simply means removing the INBOX label. (The exception is Trash and Spam.) But gnus treats these as actual groups (call them folders if you wish). Now, when the email is moved out of INBOX to All Mail, and you exit INBOX, I'm guessing that the All Mail 'read' status isn't updated because gnus views that as a truly separate group. It would have updated INBOX but the email is already gone. Gnus doesn't know, hey this is gmail, they do things the gmail way. Similarly the email in question has yet to be seen in the gnus All Mail group, so when that group is entered/refreshed, the email comes up as a new unseen (unread) item. This points to an issue with gnus and gmail, namely that gnus follows a traditional paradigm which doesn't reflect how gmail actually works. I do not view this as a big issue, however, and fixing it would require, I think, massive rework well beyond the limited added value. I think things are a lot simpler with traditional email which has true IMAP folders. But as a practical matter, if a few emails are shown as unread, whether in gnus or another client, it isn't a show-stopper. -- Bob Newell Honolulu, Hawai`i - Via GNU/Linux/Emacs/Gnus/BBDB
Re: Does Gnus read and sync the read with IMAP?
> (defvar nnimap-mark-alist > '((read "\\Seen" %Seen) > (tick "\\Flagged" %Flagged) > (reply "\\Answered" %Answered) > (expire "gnus-expire") > (dormant "gnus-dormant") > (score "gnus-score") > (save "gnus-save") > (download "gnus-download") > (forward "gnus-forward"))) Eric, thanks; this is very helpful. And it leaves a question/comment or maybe a multi-part one :) 1. Yes as you say not all IMAP servers support all the gnus-ish additions. 2. When an already read/seen article becomes unread again, as in the case I tried out, it appears gnus recognizes the change. 3. But here's the big one, and one that I thought I understood but seem to get inconsistent results. At what point does gnus update the IMAP server? In other words, I read an article. In the summary buffer that article definitely is now shown as read. But that doesn't propagate instantly to the IMAP server. I can see this by (say) opening a gmail web interface and noting that the article I just read on gnus is still unread on gmail web. (I will try a few test cases on my own to see what I can learn.) -- Bob Newell Honolulu, Hawai`i - Via GNU/Linux/Emacs/Gnus/BBDB
Re: Does Gnus read and sync the read with IMAP?
On Oct 20, 2023, Eric Abrahamsen wrote: > Dan Christensen writes: > >> Here's a followup question: how can I make Gnus update the marks for >> *all* IMAP groups. > > I once asked Lars about this -- since the IMAP server should be the > source of truth for all information about the messages, why does Gnus > store its own marks data for IMAP groups at all? Why not just refresh > from the server at each startup? I think he said that for many large > groups that would end up being prohibitively slow. I agree that it would be too slow for many people, especially with remote IMAP servers. So I'd like an optional setting. But I don't think it should have anything to do with Gnus shutdown and startup. *Every* time I hit `g', I want the marks to be refreshed from the IMAP server. Presumably this is the same code that runs when Gnus starts up, so it would improve the behaviour at startup as well. Dan
Re: Does Gnus read and sync the read with IMAP?
Dan Christensen writes: > On Oct 19, 2023, Björn Bidar wrote: > >> How can I make Gnus to fetch the read status from Imap? > > In the *Group* buffer, you can make Gnus update the marks from the IMAP > server using `M-g'. > > Here's a followup question: how can I make Gnus update the marks for > *all* IMAP groups. I've tried various incantations. `g' doesn't work, > nor does `M-g' on a topic containing the IMAP groups. In fact, I'd > like Gnus to always refresh IMAP groups in the thorough way that `M-g' > on the group does, since I'm accessing a local IMAP server. I once asked Lars about this -- since the IMAP server should be the source of truth for all information about the messages, why does Gnus store its own marks data for IMAP groups at all? Why not just refresh from the server at each startup? I think he said that for many large groups that would end up being prohibitively slow. I guess that might be true, I don't know. I do think that Gnus should be much, much more willing to dump its marks store and refresh if anything at all seems to be getting out of whack. Maybe we could put in a customization option that would clear out Gnus' nnimap group marks at shutdown, which would force a full refresh at the next startup. That feels like a pretty major change, I'm not sure what the knock-on consequences might be. The other common problem (that Bob Newell ran into recently) is that Gnus' article numbers are purely an internal convention, and don't have any real correspondence to external reality, yet are relied upon pretty heavily within Gnus to "make sense" as an indicator of sequence. When they get out of whack you are SOL. I don't see a great solution to that. Eric
Re: Does Gnus read and sync the read with IMAP?
Michael Heerdegen writes: > Bob Newell writes: > >> This goes well beyond my very limited knowledge so I did an >> experiment. I went to the gmail web interface and found an >> already-opened message in the All Mail folder (really a label) >> that I knew for certain I had originally opened in gnus, so >> therefore gnus would have known about the message. I marked >> it unread (on the web) and moved it back to INBOX. >> >> Then I started up gnus and gnus saw the message as an unread >> INBOX message. Based on this it appears the answer is yes, >> gnus did update the status of this previously seen email. >> However I don't know how much if any status information gnus >> actually keeps (and I don't use the registry) so this may or >> may not be meaningless. It certainly seems as if, when >> reopening the group subsequent to shutting down gnus at some >> point, gnus gets a fresh set of information. > > Thanks for doing this experiment. I hoped for that result. > > AFAIU Gnus stores such stuff in a file named ".newsrc.eld" (AFAIR > ".newsrc.el" in older Emacs versions). And indeed, among other things > there seems to be information about "seen" status of messages in that > file, as well as saved status, ticked and dormant status, all of that > Gnus specific stuff, but nothing related to the "read" (or "old") > status. So I guess the "read" status is fetched every time when you > open the group, along with the rest (list of existing messages etc). Here's the mapping between Gnus' marks and IMAP flags, I believe things like "gnus-save" are set as user flags on servers that support that: (defvar nnimap-mark-alist '((read "\\Seen" %Seen) (tick "\\Flagged" %Flagged) (reply "\\Answered" %Answered) (expire "gnus-expire") (dormant "gnus-dormant") (score "gnus-score") (save "gnus-save") (download "gnus-download") (forward "gnus-forward")))
Re: Does Gnus read and sync the read with IMAP?
On Oct 19, 2023, Björn Bidar wrote: > How can I make Gnus to fetch the read status from Imap? In the *Group* buffer, you can make Gnus update the marks from the IMAP server using `M-g'. Here's a followup question: how can I make Gnus update the marks for *all* IMAP groups. I've tried various incantations. `g' doesn't work, nor does `M-g' on a topic containing the IMAP groups. In fact, I'd like Gnus to always refresh IMAP groups in the thorough way that `M-g' on the group does, since I'm accessing a local IMAP server. Dan
Re: Does Gnus read and sync the read with IMAP?
Bob Newell writes: > On Thu, Oct 19, 2023 at 12:14 PM Björn Bidar > wrote: > >> Bob Newell writes: >> >> >> I noticed that when I read a mail on my phone the read statusis >> >> still unread when I fetch my mails from my imap server is still >> >> unread in >> >> Gnus. >> >> How can I make Gnus to fetch the read status from Imap? >> > >> > This could depend on your email client on your phone. At what >> > point does it sync back to the IMAP server to say that you've >> > read the mail? When the client closes? At certain intervals? >> > You might try closing the client and see if that helps. >> >> The mails that were marked as new in Gnus were in the cur folder of >> the >> imap folder there were from so yes my email client marked them as >> read. >> > > The thing is, your /client/ may show them as read but may or may not > have > uploaded that "being read" information to the IMAP server. Do try > deliberately closing the client on your phone and see if it helps. > You > have to be sure (at least on Android) to actually exit the client and > not > just switch to another app. > > I closed the client in between I think. I will look how Gnus marks > them I have access to the imap server. If the mail in the maildir is no longer in the new directory but in the cur directory then the client has send the read status to the server. In my case (Sailfish OS mail) updates the read status after the mail has been read. >> as read on my imap server. >>
Re: Does Gnus read and sync the read with IMAP?
Bob Newell writes: > This goes well beyond my very limited knowledge so I did an > experiment. I went to the gmail web interface and found an > already-opened message in the All Mail folder (really a label) > that I knew for certain I had originally opened in gnus, so > therefore gnus would have known about the message. I marked > it unread (on the web) and moved it back to INBOX. > > Then I started up gnus and gnus saw the message as an unread > INBOX message. Based on this it appears the answer is yes, > gnus did update the status of this previously seen email. > However I don't know how much if any status information gnus > actually keeps (and I don't use the registry) so this may or > may not be meaningless. It certainly seems as if, when > reopening the group subsequent to shutting down gnus at some > point, gnus gets a fresh set of information. Thanks for doing this experiment. I hoped for that result. AFAIU Gnus stores such stuff in a file named ".newsrc.eld" (AFAIR ".newsrc.el" in older Emacs versions). And indeed, among other things there seems to be information about "seen" status of messages in that file, as well as saved status, ticked and dormant status, all of that Gnus specific stuff, but nothing related to the "read" (or "old") status. So I guess the "read" status is fetched every time when you open the group, along with the rest (list of existing messages etc). Thx, Michael.
Re: Does Gnus read and sync the read with IMAP?
Michael Heerdegen writes: > Bob Newell writes: > >> The thing is, your /client/ may show them as read but may or may not >> have uploaded that "being read" information to the IMAP server. Do >> try deliberately closing the client on your phone and see if it helps. >> You have to be sure (at least on Android) to actually exit the client >> and not just switch to another app. > > Does Gnus update the "being read" information from the server also for > messages it already knows (i.e., has seen in a prior session)? This goes well beyond my very limited knowledge so I did an experiment. I went to the gmail web interface and found an already-opened message in the All Mail folder (really a label) that I knew for certain I had originally opened in gnus, so therefore gnus would have known about the message. I marked it unread (on the web) and moved it back to INBOX. Then I started up gnus and gnus saw the message as an unread INBOX message. Based on this it appears the answer is yes, gnus did update the status of this previously seen email. However I don't know how much if any status information gnus actually keeps (and I don't use the registry) so this may or may not be meaningless. It certainly seems as if, when reopening the group subsequent to shutting down gnus at some point, gnus gets a fresh set of information. -- Bob Newell Honolulu, Hawai`i - Via GNU/Linux/Emacs/Gnus/BBDB
Re: Does Gnus read and sync the read with IMAP?
Bob Newell writes: > The thing is, your /client/ may show them as read but may or may not > have uploaded that "being read" information to the IMAP server. Do > try deliberately closing the client on your phone and see if it helps. > You have to be sure (at least on Android) to actually exit the client > and not just switch to another app. Does Gnus update the "being read" information from the server also for messages it already knows (i.e., has seen in a prior session)? Michael.
Re: Does Gnus read and sync the read with IMAP?
On Thu, Oct 19, 2023 at 12:14 PM Björn Bidar wrote: > Bob Newell writes: > > >> I noticed that when I read a mail on my phone the read statusis > >> still unread when I fetch my mails from my imap server is still > >> unread in > >> Gnus. > >> How can I make Gnus to fetch the read status from Imap? > > > > This could depend on your email client on your phone. At what > > point does it sync back to the IMAP server to say that you've > > read the mail? When the client closes? At certain intervals? > > You might try closing the client and see if that helps. > > The mails that were marked as new in Gnus were in the cur folder of the > imap folder there were from so yes my email client marked them as read. > The thing is, your /client/ may show them as read but may or may not have uploaded that "being read" information to the IMAP server. Do try deliberately closing the client on your phone and see if it helps. You have to be sure (at least on Android) to actually exit the client and not just switch to another app. I closed the client in between I think. I will look how Gnus marks them > as read on my imap server. > -- Bob Newell Honolulu, Hawai`i
Re: Does Gnus read and sync the read with IMAP?
Bob Newell writes: >> I noticed that when I read a mail on my phone the read statusis >> still unread when I fetch my mails from my imap server is still >> unread in >> Gnus. >> How can I make Gnus to fetch the read status from Imap? > > This could depend on your email client on your phone. At what > point does it sync back to the IMAP server to say that you've > read the mail? When the client closes? At certain intervals? > You might try closing the client and see if that helps. The mails that were marked as new in Gnus were in the cur folder of the imap folder there were from so yes my email client marked them as read. I closed the client in between I think. I will look how Gnus marks them as read on my imap server.
Re: Does Gnus read and sync the read with IMAP?
> I noticed that when I read a mail on my phone the read statusis still unread > when I fetch my mails from my imap server is still unread in > Gnus. > How can I make Gnus to fetch the read status from Imap? This could depend on your email client on your phone. At what point does it sync back to the IMAP server to say that you've read the mail? When the client closes? At certain intervals? You might try closing the client and see if that helps. I see the opposite effect. I access my gmail directly from gnus with IMAP. When I read a mail on gnus it isn't immediately marked as read on gmail, and appears in the web interface or on my phone client as unopened sometimes, until I close down gnus. -- Bob Newell Honolulu, Hawai`i - Via GNU/Linux/Emacs/Gnus/BBDB
