Hi,
Thanks for the notes. Hopefully patch one if good now. Patch two needs
tests, but I can write those if we agree to impose minlevel automatically.
Nicolas Goaziou <m...@nicolasgoaziou.fr> writes:
> AFAICT, there's no reason to include a rule about whitespace separating
> anything. Just make sure that any INCLUDE keyword that doesn't have
> a :minlevel property gets one set to 1+N, where N is the current level
> (or 0 if at top level).
>
> Another option is to delay insertion of included files: expand them
> completely in different strings, then replace keywords with appropriate
> strings. IOW, just make sure expansion doesn't happen sequentially.
OK. Solution one sounds easier. A quick attempt, without tests, is given
in the second patch. I'll add patches if you agree with the easy
approach. It seems to work, though I'm not sure if the matching of
headlines which should have :minlevel added is robust enough.
>> Objects can be extracted via =#+INCLUDE= using file links. It is
>> -possible to include only the contents of the object. See manual for
>> +possible to include only the contents of the object. Further,
>> +footnotes are now supported when using =#+INCLUDE=. See manual for
>
> This is not quite true. Footnotes are already supported with INCLUDE
> keywords. This is the combination of :lines and footnotes that is new.
> It is more a bugfix than a new feature.
Right. Removed.
>> + (goto-char (point-min))
>> + (while (and (search-forward-regexp org-footnote-re nil t))
>> + (let* ((reference (org-element-context))
>> + (type (org-element-type reference))
>> + (label (org-element-property :label reference)))
>> + (when (and label (eq type 'footnote-reference))
>> + (unless (org-footnote-get-definition label)
>> + (save-excursion
>> + (org-footnote-create-definition label)
>> + ;; We do not need an error here since ox
>> + ;; will complain if a footnote is missing.
>> + (insert (or (gethash label footnotes) "")))))))
>
> Why is the above necessary? Shouldn't you only insert footnotes
> definitions at the end of the master document (i.e. when INCLUDED is
> nil)?
Indeed. Thanks for the hint!
> I think a `maphash' is enough.
>
> Also, looking for every footnote reference sounds tedious. You should
> simply insert every footnote definition collected there, and filter out
> unnecessary definitions at another level (e.g., before storing it in the
> hash table).
Thanks!
>> + (when id
>> + (unless (eq major-mode 'org-mode)
>> + (let ((org-inhibit-startup t)) (org-mode)))
>
> Is it necessary?
I think org-with-wide-buffer is sufficient.
>> + (forward-char 4)
>> + (insert (format "%d-" id))
>> + (and (not (eq footnote-type 'inline))
>> + (let ((new-label (org-element-property
>> + :label (org-element-context))))
>
> Why do you need to parse the new label, since you know it already:
>
> (concat (format "%d-" id) label)
Almost, but label contains fn: first, so the above would be e.g. 1-fn:1.
I didn't see an elegant way of doing it at first, thus I used elements,
but now I just use regexp-replace... I solved in another way.
>> + (save-restriction
>> + (save-excursion
>> + (widen)
>
> `save-restriction' + `save-excursion' + `widen' = `org-with-wide-buffer'
Cool.
>> + (puthash new-label
>> + (org-element-normalize-string
>> + (buffer-substring
>> + (org-element-property
>> + :contents-begin definition)
>> + (org-element-property
>> + :contents-end definition)))
>> + footnotes))
>
> Here you could check if :contents-begin is within LINES, in which case
> the definition needs not be inserted at the end of the master document.
Good idea. I did it a bit more elaborated since footnotes can in
principle also be before the definition. I don't check for the end.
Thanks,
Rasmus
--
Slowly unravels in a ball of yarn and the devil collects it
>From 5d79c76c6a93666a1521a5d5eefe3d79bda3d00d Mon Sep 17 00:00:00 2001
From: rasmus <ras...@gmx.us>
Date: Tue, 9 Dec 2014 12:40:52 +0100
Subject: [PATCH 1/2] ox.el: Fix footnote-bug in #+INCLUDE-keyword
* ox.el (org-export--prepare-file-contents): Preserve footnotes
when using the LINES argument. New optional argument FOOTNOTES.
(org-export-expand-include-keyword): New optional argument
FOOTNOTES.
* test-ox.el: Add test for INCLUDE with :lines and footnotes.
---
lisp/ox.el | 116 +++++++++++++++++++++++++++++++++++++-----------
testing/lisp/test-ox.el | 70 +++++++++++++++++++++++++++++
2 files changed, 159 insertions(+), 27 deletions(-)
diff --git a/lisp/ox.el b/lisp/ox.el
index 9d9e794..99c4e9b 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3052,18 +3052,22 @@ locally for the subtree through node properties."
(car key)
(if (org-string-nw-p val) (format " %s" val) ""))))))))
-(defun org-export-expand-include-keyword (&optional included dir)
+(defun org-export-expand-include-keyword (&optional included dir footnotes)
"Expand every include keyword in buffer.
Optional argument INCLUDED is a list of included file names along
with their line restriction, when appropriate. It is used to
avoid infinite recursion. Optional argument DIR is the current
working directory. It is used to properly resolve relative
-paths."
+paths. Optional argument FOOTNOTES is a hash-table used for
+storing and resolving footnotes. It is created automatically."
(let ((case-fold-search t)
(file-prefix (make-hash-table :test #'equal))
- (current-prefix 0))
+ (current-prefix 0)
+ (footnotes (or footnotes (make-hash-table :test #'equal)))
+ (include-re "^[ \t]*#\\+INCLUDE:"))
(goto-char (point-min))
- (while (re-search-forward "^[ \t]*#\\+INCLUDE:" nil t)
+ ;; Expand INCLUDE keywords.
+ (while (re-search-forward include-re nil t)
(let ((element (save-match-data (org-element-at-point))))
(when (eq (org-element-type element) 'keyword)
(beginning-of-line)
@@ -3155,15 +3159,44 @@ paths."
file location only-contents lines)
lines)))
(org-mode)
- (insert
- (org-export--prepare-file-contents
- file lines ind minlevel
- (or (gethash file file-prefix)
- (puthash file (incf current-prefix) file-prefix)))))
+ (insert (org-export--prepare-file-contents
+ file lines ind minlevel
+ (or (gethash file file-prefix)
+ (puthash file (incf current-prefix) file-prefix))
+ footnotes)))
(org-export-expand-include-keyword
(cons (list file lines) included)
- (file-name-directory file))
- (buffer-string)))))))))))))
+ (file-name-directory file)
+ footnotes)
+ (buffer-string)))))
+ ;; Expand footnotes after all files have been
+ ;; included. Footnotes are stored in an
+ ;; `org-footnote-section' if that variable is
+ ;; non-nil. Otherwise they are stored close to the definition.
+ (when (and (not included) (> (hash-table-count footnotes) 0))
+ (org-with-wide-buffer
+ (goto-char (point-min))
+ (if org-footnote-section
+ (progn
+ (or (search-forward-regexp
+ (concat "^\\*[ \t]+"
+ (regexp-quote org-footnote-section)
+ "[ \t]*$")
+ nil t)
+ (and
+ (goto-char (point-max))
+ (insert (format "* %s" org-footnote-section))))
+ (insert "\n")
+ (maphash (lambda (ref def)
+ (insert (format "[%s] %s" ref def) "\n"))
+ footnotes))
+ ;; `org-footnote-section' is nil. Insert definitions close to references.
+ (maphash (lambda (ref def)
+ (save-excursion
+ (search-forward (format "[%s]" ref))
+ (org-footnote-create-definition ref)
+ (insert def "\n")))
+ footnotes))))))))))))
(defun org-export--inclusion-absolute-lines (file location only-contents lines)
"Resolve absolute lines for an included file with file-link.
@@ -3227,8 +3260,8 @@ Return a string of lines to be included in the format expected by
(while (< (point) end) (incf counter) (forward-line))
counter))))))))
-(defun org-export--prepare-file-contents (file &optional lines ind minlevel id)
- "Prepare the contents of FILE for inclusion and return them as a string.
+(defun org-export--prepare-file-contents (file &optional lines ind minlevel id footnotes)
+ "Prepare contents of FILE for inclusion and return it as a string.
When optional argument LINES is a string specifying a range of
lines, include only those lines.
@@ -3246,7 +3279,11 @@ file should have.
Optional argument ID is an integer that will be inserted before
each footnote definition and reference if FILE is an Org file.
This is useful to avoid conflicts when more than one Org file
-with footnotes is included in a document."
+with footnotes is included in a document.
+
+Optional argument FOOTNOTES is a hash-table to store footnotes in
+the included document.
+"
(with-temp-buffer
(insert-file-contents file)
(when lines
@@ -3309,19 +3346,44 @@ with footnotes is included in a document."
;; become file specific and cannot collide with footnotes in other
;; included files.
(when id
- (goto-char (point-min))
- (while (re-search-forward org-footnote-re nil t)
- (let ((reference (org-element-context)))
- (when (memq (org-element-type reference)
- '(footnote-reference footnote-definition))
- (goto-char (org-element-property :begin reference))
- (forward-char)
- (let ((label (org-element-property :label reference)))
- (cond ((not label))
- ((org-string-match-p "\\`[0-9]+\\'" label)
- (insert (format "fn:%d-" id)))
- (t (forward-char 3) (insert (format "%d-" id)))))))))
- (org-element-normalize-string (buffer-string))))
+ (let* ((lines (and lines (split-string lines "-")))
+ (lbeg (and lines (string-to-number (car lines))))
+ (lend (and lines (string-to-number (cadr lines)))))
+ (goto-char (point-min))
+ (while (re-search-forward org-footnote-re nil t)
+ (let* ((reference (org-element-context))
+ (type (org-element-type reference))
+ (footnote-type (org-element-property :type reference))
+ (label (org-element-property :label reference)))
+ (when (eq type 'footnote-reference)
+ (goto-char (org-element-property :begin reference))
+ (when label
+ (forward-char 4)
+ (insert (format "%d-" id))
+ (and (not (eq footnote-type 'inline))
+ (org-with-wide-buffer
+ (org-footnote-goto-definition label)
+ (beginning-of-line)
+ (org-skip-whitespace)
+ (forward-char 4)
+ (insert (format "%d-" id))
+ (let ((definition (org-element-context)))
+ (when (and lines
+ (or (< lend (line-number-at-pos
+ (org-element-property
+ :contents-begin definition)))
+ (> lbeg (line-number-at-pos
+ (org-element-property
+ :contents-begin definition)))))
+ (puthash (org-element-property :label definition)
+ (org-element-normalize-string
+ (buffer-substring
+ (org-element-property
+ :contents-begin definition)
+ (org-element-property
+ :contents-end definition)))
+ footnotes)))))))))))
+ (org-element-normalize-string (buffer-string))))
(defun org-export-execute-babel-code ()
"Execute every Babel code in the visible part of current buffer."
diff --git a/testing/lisp/test-ox.el b/testing/lisp/test-ox.el
index 9a0e787..140c0a8 100644
--- a/testing/lisp/test-ox.el
+++ b/testing/lisp/test-ox.el
@@ -919,6 +919,76 @@ Footnotes[fn:1], [fn:test] and [fn:inline:anonymous footnote].
(org-element-map (org-element-parse-buffer)
'footnote-reference
(lambda (ref) (org-element-property :label ref)))))))))))
+ ;; Footnotes are supported by :lines-like elements and unnecessary
+ ;; footnotes are dropped.
+ (should
+ (= 3
+ (length
+ (delete-dups
+ (let ((contents "
+* foo
+Footnotes[fn:1]
+* bar
+Footnotes[fn:2], [fn:test], and [fn:inline:anonymous footnote]
+
+[fn:1] Footnote 1
+[fn:2] Footnote 1
+* Footnotes
+[fn:test] Footnote \"test\""))
+ (org-test-with-temp-text-in-file contents
+ (let ((file (buffer-file-name)))
+ (org-test-with-temp-text
+ (format "#+INCLUDE: \"%s::*bar\"
+" file)
+ (org-export-expand-include-keyword)
+ (org-element-map (org-element-parse-buffer)
+ 'footnote-definition
+ (lambda (ref) (org-element-property :label ref)))))))))))
+ ;; Footnotes within :lines are not collected in the Footnotes section
+ (should
+ (not (member org-footnote-section
+ (let ((contents "
+* foo
+Footnote[fn:test]
+* bar
+Footnote[fn:1], and [fn:inline:anonymous footnote]
+
+[fn:1] Footnote 1
+* Footnotes
+[fn:test] Footnote \"test\""))
+ (org-test-with-temp-text-in-file contents
+ (let ((file (buffer-file-name)))
+ (org-test-with-temp-text
+ (format "#+INCLUDE: \"%s::*bar\"
+ " file)
+ (org-export-expand-include-keyword)
+ (org-element-map (org-element-parse-buffer)
+ 'headline (lambda (elt) (org-element-property :title elt))))))))))
+ ;; Footnotes outside of scope are placed in Footnotes section.
+ (should
+ (let ((org-footnote-section "Footnotes")
+(contents "
+* foo
+Footnote[fn:1]
+* bar
+baz
+[fn:1] Footnote 1"))
+ (org-test-with-temp-text-in-file contents
+ (let ((file (buffer-file-name)))
+ (org-test-with-temp-text
+ (format "#+INCLUDE: \"%s::*foo\" " file)
+ (org-export-expand-include-keyword)
+ (<
+ (org-element-map (org-element-parse-buffer)
+ 'headline (lambda (head) (and (equal org-footnote-section
+ (org-element-property :raw-value head))
+ (org-element-property :begin head)))
+ nil t)
+ (org-element-map (org-element-parse-buffer)
+ 'footnote-definition (lambda (def) (and (equal "fn:1-1"
+ (org-element-property :label def))
+ (org-element-property :begin def)))
+ nil t)))))))
;; If only-contents is non-nil only include contents of element.
(should
(equal
--
2.2.0
>From f6426704852aa84f7b8fa4efda8560eb66a73a9a Mon Sep 17 00:00:00 2001
From: Rasmus <ras...@gmx.us>
Date: Thu, 18 Dec 2014 16:48:49 +0100
Subject: [PATCH 2/2] ox.el: Guess the :minlevel for INCLUDE-keywords
* ox.el (org-export-expand-include-keyword): Guess :minlevel if
missing.
---
lisp/ox.el | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/lisp/ox.el b/lisp/ox.el
index 99c4e9b..cba624c 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3066,6 +3066,18 @@ storing and resolving footnotes. It is created automatically."
(footnotes (or footnotes (make-hash-table :test #'equal)))
(include-re "^[ \t]*#\\+INCLUDE:"))
(goto-char (point-min))
+ ;; Add :minlevel to all include words that no explicitly have one.
+ (save-excursion
+ (while (re-search-forward "^[ \t]*#\\+INCLUDE: \\(.*\\)$" nil t)
+ (let ((matched (match-string 1)))
+ (unless (or (string-match-p ":minlevel" matched)
+ (string-match-p "\\(\\<src\\(?: +\\(.*\\)\\)?\\|\\<example\\>\\)" matched))
+ (goto-char (line-end-position))
+ (insert (format " :minlevel %d"
+ (1+ (save-excursion
+ (if (search-backward-regexp org-heading-regexp nil t)
+ (length (match-string 1))
+ 0)))))))))
;; Expand INCLUDE keywords.
(while (re-search-forward include-re nil t)
(let ((element (save-match-data (org-element-at-point))))
--
2.2.0
