Re: org-persist cache for remote files
Lele Gaifax writes: > Hi Ihor, > > I had some trouble testing your patches, and the outcome is still unclear to > me: apparently (again, cannot be sure this is not caused by "improper" > application of the patches) the new version of org-persist, with default value > for o-p-remote-files (100), removes any remote file from the persistent index, > without even prompting for a password. Oops. Try: - (<= org-persist-remote-files remote-files-num)) + (>= org-persist-remote-files remote-files-num)) Best, Ihor
Re: org-persist cache for remote files
Hi Ihor, I had some trouble testing your patches, and the outcome is still unclear to me: apparently (again, cannot be sure this is not caused by "improper" application of the patches) the new version of org-persist, with default value for o-p-remote-files (100), removes any remote file from the persistent index, without even prompting for a password. I will try again (and harder) as time permits. ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. l...@metapensiero.it | -- Fortunato Depero, 1929.
Re: org-persist cache for remote files
Lele Gaifax writes: > I couldn't test yet, but I saw a couple of other glitches, sorry for not > noticing first. Thanks for the proof reading. I updated the patch accordingly. Best, Ihor >From 557e7cec6dd22b09468b82650feed2020f97d781 Mon Sep 17 00:00:00 2001 Message-Id: <557e7cec6dd22b09468b82650feed2020f97d781.1639907677.git.yanta...@gmail.com> From: Ihor Radchenko Date: Sat, 18 Dec 2021 22:14:24 +0800 Subject: [PATCH] org-persist.el: Customise persistence of data for remote files * lisp/org-persist.el (org-persist-remote-files): New defcustom defining strategy for persisting data associated with remote files. Separate strategy is necessary to avoid unexpected password prompts while garbage collecting the cache: remote files may require server login. (org-persist-gc): Decide whether to store remote file data according to `org-persist-remote-files'. Reported in https://list.orgmode.org/87bl1eilt6.fsf@localhost/T/#u --- lisp/org-persist.el | 42 +++--- 1 file changed, 35 insertions(+), 7 deletions(-) diff --git a/lisp/org-persist.el b/lisp/org-persist.el index 90f9a3bc9..8e8ecc196 100644 --- a/lisp/org-persist.el +++ b/lisp/org-persist.el @@ -57,6 +57,23 @@ (defcustom org-persist-directory (expand-file-name :group 'org-persist :type 'directory) +(defcustom org-persist-remote-files 100 + "Whether to keep persistent data for remote files. + +When this variable is nil, never save persistent data associated with +remote files. When `t', always keep the data. When a number, keep up +to that number persistent values for remote files. When +`check-existence', contact remote server containing the file and only +keep the data when the file exists on the server. + +Note that setting value to `check-existence' will require logging in +to remote servers. Emacs may show multiple password prompts." + :group 'org-persist + :type '(choice (const :tag "Never" nil) + (const :tag "Always" t) + (number :tag "Up to X files") + (const :tag "Check if exist on remote" 'check-existence))) + (defvar org-persist-index-file "index" "File name used to store the data index.") @@ -287,15 +304,26 @@ (defun org-persist-read-all-buffer () (defun org-persist-gc () "Remove stored data for not existing files or unregistered variables." - (let (new-index) + (let (new-index (remote-files-num 0)) (dolist (index org-persist--index) - (let ((file (plist-get index :path)) -(persist-file (when (plist-get index :persist-file) -(org-file-name-concat - org-persist-directory - (plist-get index :persist-file) + (let* ((file (plist-get index :path)) + (file-remote (when file (file-remote-p file))) + (persist-file (when (plist-get index :persist-file) + (org-file-name-concat + org-persist-directory + (plist-get index :persist-file) (when (and file persist-file) - (if (file-exists-p file) + (when (and file-remote persist-file) +(cl-incf remote-files-num)) + (if (if (not file-remote) + (file-exists-p file) +(pcase org-persist-remote-files + ('t t) + ('check-existence + (file-exists-p file)) + ((pred #'numberp) + (<= org-persist-remote-files remote-files-num)) + (_ nil))) (push index new-index) (when (file-exists-p persist-file) (delete-file persist-file) -- 2.32.0
Re: org-persist cache for remote files
Lele Gaifax writes: >> + :group 'org-persist >> + :type '(choice (const :tag "Never" nil) >> + (const :tag "Always" t) >> + (number :tag "Keep note more than X files") > > Now that you explained the meaning, maybe better to be explicit here, > mentioning "X remote files"? Sigh, re-reading, I see one more glitch there: s/Keep note/Keep not/ I'm sorry, but real work swallowed all my day, and could not try this out yet. Hopefully I will be able to do some test tomorrow morning. bye, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. l...@metapensiero.it | -- Fortunato Depero, 1929.
Re: org-persist cache for remote files
Ihor Radchenko writes: > Lele Gaifax writes: > >> In the meanwhile, here below some notes: > > Thanks! See the updated patch. Great, you're so fast! :-) I couldn't test yet, but I saw a couple of other glitches, sorry for not noticing first. > ... > +(defcustom org-persist-remote-files 100 > + "Whether to keep persistent data for remote files. > + > +When this variable is nil, never save persitent data associated with > +remote files. When `t', always keep the data. When > +`check-existence', contact remote server containing the file and only > +keep the data when the file exists on the server. When a number, keep > +up to that number persistent values for remote files. > ... s/persitent/persistent > +Note that the last option `check-existence' may cause Emacs to show > +password prompts to log in." I'd remove "last" above, since `check-existence' is not the last (described) option anymore. > + :group 'org-persist > + :type '(choice (const :tag "Never" nil) > + (const :tag "Always" t) > + (number :tag "Keep note more than X files") Now that you explained the meaning, maybe better to be explicit here, mentioning "X remote files"? Thanks&bye, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. l...@metapensiero.it | -- Fortunato Depero, 1929.
Re: org-persist cache for remote files
Lele Gaifax writes: > In the meanwhile, here below some notes: Thanks! See the updated patch. Best, Ihor >From 92878fab54c26800562409f3c686f065b7523a61 Mon Sep 17 00:00:00 2001 Message-Id: <92878fab54c26800562409f3c686f065b7523a61.1639838789.git.yanta...@gmail.com> From: Ihor Radchenko Date: Sat, 18 Dec 2021 22:14:24 +0800 Subject: [PATCH] org-persist.el: Customise persistence of data for remote files * lisp/org-persist.el (org-persist-remote-files): New defcustom defining strategy for persisting data associated with remote files. Separate strategy is necessary to avoid unexpected password promtps while garbage collecting the cache: remote files may require server login. (org-persist-gc): Decide whether to store remote file data according to `org-persist-remote-files'. Reported in https://list.orgmode.org/87bl1eilt6.fsf@localhost/T/#u --- lisp/org-persist.el | 42 +++--- 1 file changed, 35 insertions(+), 7 deletions(-) diff --git a/lisp/org-persist.el b/lisp/org-persist.el index 90f9a3bc9..4c0673c72 100644 --- a/lisp/org-persist.el +++ b/lisp/org-persist.el @@ -57,6 +57,23 @@ (defcustom org-persist-directory (expand-file-name :group 'org-persist :type 'directory) +(defcustom org-persist-remote-files 100 + "Whether to keep persistent data for remote files. + +When this variable is nil, never save persitent data associated with +remote files. When `t', always keep the data. When +`check-existence', contact remote server containing the file and only +keep the data when the file exists on the server. When a number, keep +up to that number persistent values for remote files. + +Note that the last option `check-existence' may cause Emacs to show +password prompts to log in." + :group 'org-persist + :type '(choice (const :tag "Never" nil) + (const :tag "Always" t) + (number :tag "Keep note more than X files") + (const :tag "Check if exist on remote" 'check-existence))) + (defvar org-persist-index-file "index" "File name used to store the data index.") @@ -287,15 +304,26 @@ (defun org-persist-read-all-buffer () (defun org-persist-gc () "Remove stored data for not existing files or unregistered variables." - (let (new-index) + (let (new-index (remote-files-num 0)) (dolist (index org-persist--index) - (let ((file (plist-get index :path)) -(persist-file (when (plist-get index :persist-file) -(org-file-name-concat - org-persist-directory - (plist-get index :persist-file) + (let* ((file (plist-get index :path)) + (file-remote (when file (file-remote-p file))) + (persist-file (when (plist-get index :persist-file) + (org-file-name-concat + org-persist-directory + (plist-get index :persist-file) (when (and file persist-file) - (if (file-exists-p file) + (when (and file-remote persist-file) +(cl-incf remote-files-num)) + (if (if (not file-remote) + (file-exists-p file) +(pcase org-persist-remote-files + ('t t) + ('check-existence + (file-exists-p file)) + ((pred #'numberp) + (<= org-persist-remote-files remote-files-num)) + (_ nil))) (push index new-index) (when (file-exists-p persist-file) (delete-file persist-file) -- 2.32.0
Re: org-persist cache for remote files
Ihor Radchenko writes: > Lele Gaifax writes: > >>> Maybe we should better make this a user option? >> >> Yes, that's what I had in mind: when (say) org-persist-do-not-auto-gc-remotes >> is enabled, the cleanup procedure would ignore remote Org documents and a new >> explicit (interactive) org-persist-gc-remotes would take care of them. >> >>> Or we can go a different route and just not garbage collect remote files >>> until cache gets too large. >> >> Yes, a threshold would work too. > > Can you test the attached 2 patches? They do not seem to break local > files, but I do not have remote file caches to test on. Yes, I'll be able to do some test in a couple of hours. Thank you! In the meanwhile, here below some notes: > ... > Subject: [PATCH] org-persist.el: Customise persistence of data for remote > files > > * lisp/org-persist.el (org-persist-remote-files): New defcustom > defining strategy for persisting data associated with remote files. > Separate strategy is necessary to avoid unexpected password promtps > while garbage collecting the cache: remote files may require server > login. > ... s/promtps/prompts > ... > +(defcustom org-persist-remote-files 100 > + "Whether to keep persistent data for remote files. > + > +When this variable is nil, never save persitent data associated with > +remote files. When `t', always keep the data. When > +`check-existance', contact remote server containing the file and only > +keep the data when the file exists on the server. s/check-existance/check-existence Also, shouldn't the doc mention the meaning of the numberp, expecially since that's the default? > +Note that the last option `check-existance' may cause Emacs to show > +password prompts to log in." Thanks again, bye, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. l...@metapensiero.it | -- Fortunato Depero, 1929.
Re: org-persist cache for remote files
Lele Gaifax writes: >> Maybe we should better make this a user option? > > Yes, that's what I had in mind: when (say) org-persist-do-not-auto-gc-remotes > is enabled, the cleanup procedure would ignore remote Org documents and a new > explicit (interactive) org-persist-gc-remotes would take care of them. > >> Or we can go a different route and just not garbage collect remote files >> until cache gets too large. > > Yes, a threshold would work too. Can you test the attached 2 patches? They do not seem to break local files, but I do not have remote file caches to test on. Best, Ihor >From 592608d3c48cb1e63f9b4427576f0698ca190e9a Mon Sep 17 00:00:00 2001 Message-Id: <592608d3c48cb1e63f9b4427576f0698ca190e9a.1639837163.git.yanta...@gmail.com> From: Ihor Radchenko Date: Sat, 18 Dec 2021 22:13:24 +0800 Subject: [PATCH] org-persist.el: Make org-persist-directory a custom variable * lisp/org-persist.el (org-persist): Create org-persist customisation group. (org-persist-directory): Convert into defcustom. --- lisp/org-persist.el | 34 +- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/lisp/org-persist.el b/lisp/org-persist.el index f06d4fc61..90f9a3bc9 100644 --- a/lisp/org-persist.el +++ b/lisp/org-persist.el @@ -35,19 +35,27 @@ (declare-function org-back-to-heading "org" (&optional invisible-ok)) (declare-function org-next-visible-heading "org" (arg)) (declare-function org-at-heading-p "org" (&optional invisible-not-ok)) -(defvar org-persist-directory (expand-file-name -(org-file-name-concat - (let ((cache-dir (when (fboundp 'xdg-cache-home) -(xdg-cache-home - (if (or (seq-empty-p cache-dir) - (not (file-exists-p cache-dir)) - (file-exists-p (org-file-name-concat - user-emacs-directory - "org-persist"))) - user-emacs-directory - cache-dir)) - "org-persist/")) - "Directory where the data is stored.") + +(defgroup org-persist nil + "Persistent cache for Org mode." + :tag "Org persist" + :group 'org) + +(defcustom org-persist-directory (expand-file-name + (org-file-name-concat +(let ((cache-dir (when (fboundp 'xdg-cache-home) + (xdg-cache-home + (if (or (seq-empty-p cache-dir) + (not (file-exists-p cache-dir)) + (file-exists-p (org-file-name-concat + user-emacs-directory + "org-persist"))) + user-emacs-directory +cache-dir)) +"org-persist/")) + "Directory where the data is stored." + :group 'org-persist + :type 'directory) (defvar org-persist-index-file "index" "File name used to store the data index.") -- 2.32.0 >From de89eca758ca1972d75c8830aee224376a1e364e Mon Sep 17 00:00:00 2001 Message-Id: From: Ihor Radchenko Date: Sat, 18 Dec 2021 22:14:24 +0800 Subject: [PATCH] org-persist.el: Customise persistence of data for remote files * lisp/org-persist.el (org-persist-remote-files): New defcustom defining strategy for persisting data associated with remote files. Separate strategy is necessary to avoid unexpected password promtps while garbage collecting the cache: remote files may require server login. (org-persist-gc): Decide whether to store remote file data according to `org-persist-remote-files'. Reported in https://list.orgmode.org/87bl1eilt6.fsf@localhost/T/#u --- lisp/org-persist.el | 41 ++--- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/lisp/org-persist.el b/lisp/org-persist.el index 90f9a3bc9..613290afc 100644 --- a/lisp/org-persist.el +++ b/lisp/org-persist.el @@ -57,6 +57,22 @@ (defcustom org-persist-directory (expand-file-name :group 'org-persist :type 'directory) +(defcustom org-persist-remote-files 100 + "Whether to keep persistent data for remote files. + +When this variable is nil, never save persitent data associated with +remote files. When `t', always keep the data. When +`check-existance', contact remote server containing the file and only +keep the data when the file exists on the server. + +Note that the last option `check-existance' may cause Emacs to show +password prompts to log in." + :group 'org-persist + :type '(choice (const :tag "Never" nil) + (const :tag "Always" t) + (number :tag "Keep note more than X files") + (const :tag "Check if exist on remote" 'check-existence))) + (defvar org-persist-index-file "index" "
Re: org-persist cache for remote files
Ihor Radchenko writes: > Lele Gaifax writes: > >> I wonder if there is some mechanism I could use to either prevent caching >> of non-local documents or to avoid the check on existence in the >> org-persist-gc. > > I can easily make a change that always garbage-collect non-local > documents without trying to connect to remote server. This can be one > solution of your problem. However, I am not sure if it is the best > solution. > > May I know more about your use cases? Do you frequently open large > remote Org files? If you do, trashing their cache may not be optimal. I do not open remote Org files very often, but surely happens now and then, typically from my laptop to see/edit something I keep on my desktop PC. > Maybe we should better make this a user option? Yes, that's what I had in mind: when (say) org-persist-do-not-auto-gc-remotes is enabled, the cleanup procedure would ignore remote Org documents and a new explicit (interactive) org-persist-gc-remotes would take care of them. > Or we can go a different route and just not garbage collect remote files > until cache gets too large. Yes, a threshold would work too. Thanks a lot, ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. l...@metapensiero.it | -- Fortunato Depero, 1929.
org-persist cache for remote files (was: org-persist)
Lele Gaifax writes: > Digging down, I found that it was due to the persistent cache, that contained > several entries about remote (Tramp) documents: upon closing Emacs, the > org-persist-gc function is triggered, and the index is traversed to remove > dangling entries, performing a file-exist-p on each filename. For remote > files, that obviously mean prompting for the remote password (when the > ssh-agent does not carry it) or spending a considerable time in the network > round trip. Thanks for reporting! I am not using tramp regularly and I did not think about this use case. > I wonder if there is some mechanism I could use to either prevent caching of > non-local documents or to avoid the check on existence in the org-persist-gc. I can easily make a change that always garbage-collect non-local documents without trying to connect to remote server. This can be one solution of your problem. However, I am not sure if it is the best solution. May I know more about your use cases? Do you frequently open large remote Org files? If you do, trashing their cache may not be optimal. Maybe we should better make this a user option? Or we can go a different route and just not garbage collect remote files until cache gets too large. WDYT? Best, Ihor