Re: [O] [ox-publish, patch] More flexible sitemaps
Hello, Rasmus writes: > Sorry about the slow reply. Sometimes there's not enough time. So true. > I obviously agree with your criticism. It’s not obvious how to do this > properly. > > If we provide a backend we’d have to move the index preparation to > beginning of each export process as :publishing-function can be a list. > Also, I’m not sure how we’d know about the backend. Before the exporting > starts, we at most know the names of the functions right? If one of the > publishing functions is anonymous we don’t even have that. > > Perhaps the best way is to move keywords back to ox, though I’d rather opt > for something else. You're right, "ox-publish" is mostly back-end agnostic so the information is not readily available. However, we're creating an Org file here. We don't need to have parsed values for keywords. So, what about simply looking for, e.g., #+SUBTITLE value in the document and use it instead as a template replacement? IOW, what about eschewing so-called "environment" altogether? >>> +;; Call function to build sitemap based on files and the project-plist. >>> +(let* ((style (or (plist-get project-plist :sitemap-style) 'tree)) >>> + (fun (intern (format "org-publish-org-sitemap-as-%s" style >> >> Side note : I think this is a bit too smart. It prevents, e.g., from >> grepping for a function name. Maybe >> >> (if (eq style 'something) #'... #') >> >> would be better. > > The point is that it should be easy to supply your own functions. All > sorts of requirements for the index/sitemap could come up, and it is > important to not limit to the tree-style and the flat-style. But that's outside the scope of the patch currently considered. My point is that we should avoid forging function names as a UI. > I might want to provide an index ordered by keywords, say. > > Perhaps styles should be stores in an alist with elements like > > (STYLE STYLE-FUNCTION) ? Some functions may handle multiple styles, as `org-publish-org-sitemap' does. Besides, any user-provided sitemap function is likely to handle its own style without relying on :sitemap-style property. Therefore, providing :sitemap-function and :sitemap-style is enough, there's no need to merge them together. WDYT? Regards, -- Nicolas Goaziou
Re: [O] [ox-publish, patch] More flexible sitemaps
Hi,
Sorry about the slow reply. Sometimes there's not enough time.
Nicolas Goaziou writes:
>> + (unless (or (file-directory-p file) (directory-name-p file))
>
> What is `directory-name-p'?
Oh, it's new in Emacs-25. Thanks for pointing that out!
The following is about the index function, which is the most "difficult"
part.
>> + (let* ((org-inhibit-startup t)
>> + (visiting (find-buffer-visiting file))
>> + (buffer (or visiting (find-file-noselect file)))
>> + (case-fold-search t))
>> +(with-current-buffer buffer
>> + ;; protect local variables in open buffers
>> + (org-export-with-buffer-copy
>> + (let* ((backends (append (list nil)
>> +(mapcar 'org-export-backend-name
>> +
>> org-export-registered-backends)))
>> + (value (cl-loop for backend in backends
>> + when (org-no-properties
>> +(org-element-interpret-data
>> + (plist-get (org-export-get-environment backend)
>> +prop)))
>> + return it)))
>
> Return value of `org-element-interpret-data' is never nil so the loop
> always returns the first back-end.
>
> In any case, this is sub-optimal since common keywords are retrieved for
> each back-end. Also, two back-ends could use the same keyword, with
> different values (e.g., using `parsed' or not).
>
> One option could be to change the policy about keywords in ox.el and
> move KEYWORDS and SUBTITLE there. Unfortunately, there is still a change
> to miss some cases.
>
> Another option would be to provide BACKEND to sitemap-function. I think
> it can be backward-compatible if we make it an optional argument.
I obviously agree with your criticism. It’s not obvious how to do this
properly.
If we provide a backend we’d have to move the index preparation to
beginning of each export process as :publishing-function can be a list.
Also, I’m not sure how we’d know about the backend. Before the exporting
starts, we at most know the names of the functions right? If one of the
publishing functions is anonymous we don’t even have that.
Perhaps the best way is to move keywords back to ox, though I’d rather opt
for something else.
>> + ;; Call function to build sitemap based on files and the project-plist.
>> + (let* ((style (or (plist-get project-plist :sitemap-style) 'tree))
>> +(fun (intern (format "org-publish-org-sitemap-as-%s" style
>
> Side note : I think this is a bit too smart. It prevents, e.g., from
> grepping for a function name. Maybe
>
> (if (eq style 'something) #'... #')
>
> would be better.
The point is that it should be easy to supply your own functions. All
sorts of requirements for the index/sitemap could come up, and it is
important to not limit to the tree-style and the flat-style. I might want
to provide an index ordered by keywords, say.
Perhaps styles should be stores in an alist with elements like
(STYLE STYLE-FUNCTION) ?
>> +@item @code{:sitemap-preamble}
>> +@item @code{:sitemap-postamble}
>> +@tab Preamble and postamble for sitemap. Useful for inserting
>> +@code{#+OPTIONS}, footers etc. See @code{org-publish-sitemap-preamble}
>> +for details.
>
> I don't understand the "useful for inserting @code{#+OPTIONS}" part.
> Maybe some examples could be useful. This part of the manual is rather
> terse.
It might be useful to suppress the printing of the title or the author or
similar.
Rasmus
--
I feel emotional landscapes they puzzle me
Re: [O] [ox-publish, patch] More flexible sitemaps
Robert Klein writes: > are those Patches still being worked on? Yes, but they haven't been pushed yet. Rasmus -- May the Force be with you
Re: [O] [ox-publish, patch] More flexible sitemaps
Hi, are those Patches still being worked on? Thanks and best regards Robert On Wed, 01 Jun 2016 17:34:56 +0200 Nicolas Goaziou wrote: > Hello, > > Rasmus writes: > > > This was by far the hardest part... > > Thank you. Some comments follow. > > > +(defun org-publish-find-property (file property &optional reset) > > + "Find the PROPERTY of FILE in project. > > +PROPERTY can be a string or a symbol-property." > > Could you also document RESET argument? > [rest deleted from reply]
Re: [O] [ox-publish, patch] More flexible sitemaps
Hello, Rasmus writes: > This was by far the hardest part... Thank you. Some comments follow. > +(defun org-publish-find-property (file property &optional reset) > + "Find the PROPERTY of FILE in project. > +PROPERTY can be a string or a symbol-property." Could you also document RESET argument? > + (unless (or (file-directory-p file) (directory-name-p file)) What is `directory-name-p'? > +(let ((prop (cond ((stringp property) > + (intern (downcase (format ":%s" property > + (t property The following might be more robust: (if (keywordp property) property (intern (downcase (format ":%s" property))) > + (and (not reset) (org-publish-cache-get-file-property file prop nil t)) (unless reset ...) Anyway, you don't seem to use the return value. If this is used for side-effect only, you may want to call `org-publish-initialize-cache' instead. > + (let* ((org-inhibit-startup t) > + (visiting (find-buffer-visiting file)) > + (buffer (or visiting (find-file-noselect file))) > + (case-fold-search t)) > +(with-current-buffer buffer > + ;; protect local variables in open buffers > + (org-export-with-buffer-copy > + (let* ((backends (append (list nil) > +(mapcar 'org-export-backend-name > +org-export-registered-backends))) > + (value (cl-loop for backend in backends > + when (org-no-properties > +(org-element-interpret-data > + (plist-get > (org-export-get-environment backend) > +prop))) > + return it))) Return value of `org-element-interpret-data' is never nil so the loop always returns the first back-end. In any case, this is sub-optimal since common keywords are retrieved for each back-end. Also, two back-ends could use the same keyword, with different values (e.g., using `parsed' or not). One option could be to change the policy about keywords in ox.el and move KEYWORDS and SUBTITLE there. Unfortunately, there is still a change to miss some cases. Another option would be to provide BACKEND to sitemap-function. I think it can be backward-compatible if we make it an optional argument. > +(defcustom org-publish-sitemap-file-entry-format "%- [[file:%link][%TITLE]]" >"Format string for site-map file entry. > -You could use brackets to delimit on what part the link will be. > > -%t is the title. > -%a is the author. > -%d is the date formatted using `org-publish-sitemap-date-format'." > +Most keywords can be accessed using a the name of the keyword > +prefixed with \"%\", e.g. \"%TITLE\" or \"%KEYWORDS\". In > +addition, the following keywords are defined. > + > + %fullpath > +The full path of the file. > + > + %relpath > +The relative path of the file. > + > + %filename > + %f > +The filename. > + > + %link > + %l > +The link. > + > + %* > +Leveled headline relative to the base directory. > + > + %* > +Leveled item relative to the base directory. > + > + %author > +The author of the document, the project, or `user-full-name'. > + > + %date > + Date formatted using `org-publish-sitemap-date-format'. > + > +See also `org-publish-sitemap-dir-entry-format'." Could it be possible to use single-char placeholders and `format-spec', which is more robust than replace-match (e.g., it is possibl to escape percent signs)... > +(defcustom org-publish-sitemap-dir-entry-format "%- %f" > + "Format string for site-map file entry. > + > +The following keywords are defined. > + > + %fullpath > +The full path of the file. > + > + %relpath > +The relative path of the file. > + > + %filename > + %f > +The filename. > + > + %link > + %l > +The link. > + > + %* > +Leveled headline relative to the base directory. > + > + %* > +Leveled item relative to the base directory. > + > + %author > +The author of the document, the project, or `user-full-name'. > + > + %date > + Date formatted using `org-publish-sitemap-date-format'. Ditto. > @@ -1292,11 +1394,11 @@ Returns value on success, else nil." > (defun org-publish-cache-ctime-of-src (file) >"Get the ctime of FILE as an integer." >(let ((attr (file-attributes > -(expand-file-name (or (file-symlink-p file) file) > - (file-name-directory file) > + (expand-file-name (or (file-symlink-p file) file) > + (file-name-directory file) Maybe (file-truename file) instead. > + `:sitemap-preamble' > + `:sitemap-postamble' > + > +Preamble and postamble for sitemap. Useful for inserting > +#+OPTIONS: keywords, footers etc. See > +`org-publish-sit
Re: [O] [ox-publish, patch] More flexible sitemaps
Hi, Thanks for the comments. Nicolas Goaziou writes: >> (org-publish-find-subtitle): New function. >> (org-publish-org-sitemap-as-tree): New function. >> (org-publish--find-property): Find arbirary property. >> (org-publish-project-alist): Document changes. >> * doc/org.texi (Sitemap): Update documentation. > > All in all, I think this deserves to be split into 3 patches: one for > the preamble-postamble feature, another one for implementing > `org-publish--find-property' and associated refactoring, and the latter > for the sitemap itself. This was by far the hardest part... >> +(autoload 'message-flatten-list "message") >> +(autoload 'dired-tree-lessp "dired-aux") > > I hope we can avoid these. In particular, why are you using > `dired-tree-lessp' instead of `org-publish-compare-directory-files'? AFAIK, org-publish-compare-directory-files can’t be used on its own. It requires variables to be bound around it, e.g. org-publish-sitemap-ignore-case. See org-publish-get-base-files. (I don’t know why the project plist isn’t being passed around.) Anyway, I’m not using neither of these files anymore. Aside: the way ox-publish order files is not super, IMO, as it takes directory into account even if one use a flat list and chronological ordering. That’s a problem for another day, though. >> +(defun org-publish--tree-assoc (key tree) >> +(defun org-pubish--order-files-by-dir-tree (files) > > I don't understand why you need the 2 functions above. You are working > with plain lists, not nested ones. Besides, once the file name are > standardized, isn't tree order equivalent to lexicographic one? Removed. >> +(defun org-publish-find-subtitle (file &optional reset) >> + "Find the title of FILE in project." >> + (org-publish--find-property file :subtitle reset)) > > I don't think this would work. :subtitle is not defined in default > export properties, it is back-end specific. `org-export-get-environment' > without any argument, doesn't catch these. You need to somehow provide > it the back-end. You are right. The "problem" is that the sitemap is done in a before the backend is known. So we can’t do (org-export-get-environment BACKEND). OTOH, only allowing ox.el keywords is too limiting IMO. E.g. a blog sitemap wouldn’t be able to include keywords, though a desirable format for each entry is TITLE DATE KEYWORDS #+INCLUDE: "file.org::lead" But KEYWORDS is specific to ox-html... A couple of workarounds: 1. Guess the backend(s) from the name of :publishing-function. This is not very robust... 2. Find the keyword by search in the file, thus sidestepping org-export-get-environment. It is is not without problems. E.g. if my file contains this, I’d probably only key1. #+Keywords: key1 #+Keywords: key2 3. Go through backend for org-export-get-environment until something is Found. I do this now. Rasmus -- This is the kind of tedious nonsense up with which I will not put >From 4e18d9141c509b03cc5579517547f6e7e73088f3 Mon Sep 17 00:00:00 2001 From: Rasmus Date: Fri, 27 May 2016 16:46:14 +0200 Subject: [PATCH 1/6] ox-publish: Tiny refactor * lisp/ox-publish.el: (org-publish-compare-directory-files): (org-publish-get-base-files-1): Change test. * lisp/ox-publish.el (org-publish-initialize-cache): Tiny fix. --- lisp/ox-publish.el | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lisp/ox-publish.el b/lisp/ox-publish.el index 8ccba99..d60562b 100644 --- a/lisp/ox-publish.el +++ b/lisp/ox-publish.el @@ -467,7 +467,7 @@ matching the regexp SKIP-DIR when recursing through BASE-DIR." (and skip-file (string-match skip-file fnd)) (not (file-exists-p (file-truename f))) (not (string-match match fnd))) - (cl-pushnew f org-publish-temp-files))) + (cl-pushnew f org-publish-temp-files :test #'file-equal-p))) (defun org-publish-get-base-files (project &optional exclude-regexp) "Return a list of all files in PROJECT. @@ -511,7 +511,7 @@ matching filenames." (setq org-publish-temp-files nil) (when org-publish-sitemap-requested (cl-pushnew (expand-file-name (concat base-dir sitemap-filename)) - org-publish-temp-files)) + org-publish-temp-files :test #'file-equal-p)) (org-publish-get-base-files-1 base-dir recurse match ;; FIXME distinguish exclude regexp ;; for skip-file and skip-dir? @@ -1172,7 +1172,7 @@ If FREE-CACHE, empty the cache." (if cexists (load-file cache-file) (setq org-publish-cache - (make-hash-table :test 'equal :weakness nil :size 100)) + (make-hash-table :test #'equal :weakness nil :size 100)) (org-publish-cache-set ":project:" project-name) (org-publish-cache-set ":cache-file:" cache-file)) (unless cexists (org-publish-write-cache-file nil -- 2.8.3 >From 410dc552de2e846f7eed9f98275acffba45f5be0 Mon Sep 17 00:00:00 2001 From: Rasmus Date: Fri, 27 May 2016 15:46:31 +0200 Subject: [PATCH 2/6] ox-publish: Get pr
Re: [O] [ox-publish, patch] More flexible sitemaps
Hello, Rasmus writes: > I've long wanted to use ox to auto-generate something that looks like a > blog index. > > This patch makes ox sitemaps a bit more flexible. For instance, it would > allow me to use something like this for ‘:sitemap-file-entry-format’, > > :sitemap-file-entry-format "* [[file:%l][%t]] > #+include: \"%f::lead\" > > [[file:%l][Read more]]" > > Which would come out as; > > * [[file:link][Title]] > #+Include: "file.org::lead" > > [[File:link][Read more]] > > For the tests I did, it matches the "old" sitemap for list and tree. > > WDYT? It sounds interesting. Usual nitpicking follows. > * lisp/ox-publish.el (org-publish-sitemap-file-entry-format): Support > more formatters. > (org-publish-sitemap-dir-entry-format): New defcustom. > (org-publish-org-sitemap): Use new variables and functions. > (org-publish-org-sitemap-as-list): New function. > (org-publish--tree-assoc): New function. > (org-pubish--order-files-by-dir-tree): New function. > (org-publish-find-title): New function. This is not a new function. > (org-publish-find-subtitle): New function. > (org-publish-org-sitemap-as-tree): New function. > (org-publish--find-property): Find arbirary property. > (org-publish-project-alist): Document changes. > * doc/org.texi (Sitemap): Update documentation. All in all, I think this deserves to be split into 3 patches: one for the preamble-postamble feature, another one for implementing `org-publish--find-property' and associated refactoring, and the latter for the sitemap itself. > +(autoload 'message-flatten-list "message") > +(autoload 'dired-tree-lessp "dired-aux") I hope we can avoid these. In particular, why are you using `dired-tree-lessp' instead of `org-publish-compare-directory-files'? > @@ -399,6 +469,7 @@ This splices all the components into the list." > (defvar org-publish-sitemap-requested) > (defvar org-publish-sitemap-date-format) > (defvar org-publish-sitemap-file-entry-format) > +(defvar org-publish-sitemap-dir-entry-format) The above is not necessary. > + (files (nreverse > + ;; Sitemap shouldn't list itself. > + (cl-delete-if (lambda (f) > + (equal (file-truename f) > + (file-truename sitemap-filename))) See `file-equal-p'. > +(org-publish-get-base-files > + project > + (plist-get project-plist :exclude) >(sitemap-title (or (plist-get project-plist :sitemap-title) > - (concat "Sitemap for project " (car project > - (sitemap-style (or (plist-get project-plist :sitemap-style) > - 'tree)) > - (sitemap-sans-extension > - (plist-get project-plist :sitemap-sans-extension)) > + (concat "Sitemap for project " (car project >(visiting (find-buffer-visiting sitemap-filename)) > - file sitemap-buffer) > -(with-current-buffer > - (let ((org-inhibit-startup t)) > - (setq sitemap-buffer > - (or visiting (find-file sitemap-filename > + (sitemap-buffer (or visiting (find-file sitemap-filename))) > + (insert-pre-or-postamble (function (lambda (pre-or-postamble) No need to wrap `function' around `lambda'. Also, it doesn't "insert" anything, does it? IOW, isn't the name a bit confusing ? > + (when pre-or-postamble You can include the `when' in the cond: (cond ((not pre-or-postamble) nil) ((stringp pre-or-postamble) ...) ...) > + (cond ((stringp > pre-or-postamble) pre-or-postamble) > + ((listp pre-or-postamble) > +(mapconcat 'identity > preamble "\n")) > + ((functionp > pre-or-postamble) > +(funcall > pre-or-postamble project-plist)) > + (t (error (concat > "unknown `:sitemap-preamble' or " > + > "`:sitemap-postamble' format") > +(with-current-buffer (let ((org-inhibit-startup t)) sitemap-buffer) You can drop the `let' part, which is a no-op here. > + ;; Insert sitemap-preamble. > + (funcall insert-pre-or-postamble > +(plist-get project-plist :sitemap-preamble)) > + ;; Call function to build sitemap based on files and the project-plist. > + (insert (funcall (intern > + (concat "org-publish-org-sitemap-as-" > + (symbol-name (or (plist-get project-plist > :sitemap-style) 'tree > +files project-plist)) (intern (format "org-publish-org-sitemap-as-%s" (or (plist-get ...) 'tree)
