Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Nicolas Goaziou]

Hello,

"Daniel P. Gomez"  writes:

> I've fixed the implementation of `org-export--prepare-file-contents`
> so
> that links are relative to the included file. A patch is attached.
>
> I've also created two org files in the test examples directory:
> "includer-with-links.org" and "subdir/includee-with-links.org". My
> goal
> was to add a test in "test-ox.el" `test-org-export/expand-include`
> that
> checks whether, after exporting the "includer-with-links.org" file to
> Org, the headings *Source and *Target are the same.
>
> I did not figure out how to do this cleanly, so any help implementing
> the test itself would be greatly appreciated.

I applied your patch and wrote a few tests. Let me know if it works for
you.

BTW, what's your status wrt FSF papers? I don't remember if you have
signed them already.

Thank you!

Regards,

-- 
Nicolas Goaziou0x80A93738

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Daniel P. Gomez]

I've fixed the implementation of 
`org-export--prepare-file-contents` so

that links are relative to the included file. A patch is attached.

I've also created two org files in the test examples directory:
"includer-with-links.org" and "subdir/includee-with-links.org". My 
goal
was to add a test in "test-ox.el" `test-org-export/expand-include` 
that
checks whether, after exporting the "includer-with-links.org" file 
to

Org, the headings *Source and *Target are the same.

I did not figure out how to do this cleanly, so any help 
implementing

the test itself would be greatly appreciated.

Regards,

Daniel

>From b47dcf43067cd57e2ee3c1f8e4dfea94bca7d14b Mon Sep 17 00:00:00 2001
From: Daniel Gomez 
Date: Fri, 2 Mar 2018 14:46:41 +0100
Subject: [PATCH 1/1] Fix file links when using #+INCLUDE

---
 lisp/ox.el  | 36 ++---
 testing/examples/includer-with-links.org| 19 +
 testing/examples/subdir/includee-with-links.org | 12 +
 3 files changed, 64 insertions(+), 3 deletions(-)
 create mode 100644 testing/examples/includer-with-links.org
 create mode 100644 testing/examples/subdir/includee-with-links.org

diff --git a/lisp/ox.el b/lisp/ox.el
index bd49a8a26..f9b2d8095 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3257,7 +3257,8 @@ avoid infinite recursion.  Optional argument DIR is the current
 working directory.  It is used to properly resolve relative
 paths.  Optional argument FOOTNOTES is a hash-table used for
 storing and resolving footnotes.  It is created automatically."
-  (let ((case-fold-search t)
+  (let ((includer-file (buffer-file-name (buffer-base-buffer)))
+	(case-fold-search t)
 	(file-prefix (make-hash-table :test #'equal))
 	(current-prefix 0)
 	(footnotes (or footnotes (make-hash-table :test #'equal)))
@@ -3373,7 +3374,8 @@ storing and resolving footnotes.  It is created automatically."
 			 (or
 			  (gethash file file-prefix)
 			  (puthash file (cl-incf current-prefix) file-prefix))
-			 footnotes)))
+			 footnotes
+			 includer-file)))
 		 (org-export-expand-include-keyword
 		  (cons (list file lines) included)
 		  (file-name-directory file)
@@ -3451,7 +3453,7 @@ Return a string of lines to be included in the format expected by
 		   counter
 
 (defun org-export--prepare-file-contents
-(file  lines ind minlevel id footnotes)
+(file  lines ind minlevel id footnotes includer-file)
   "Prepare contents of FILE for inclusion and return it as a string.
 
 When optional argument LINES is a string specifying a range of
@@ -3476,6 +3478,34 @@ Optional argument FOOTNOTES is a hash-table to store footnotes in
 the included document."
   (with-temp-buffer
 (insert-file-contents file)
+;; Adapt all file links within the included document that
+;; contain relative paths in order to make these paths
+;; relative to the base document, or absolute
+(goto-char (point-min))
+(while (re-search-forward org-any-link-re nil t)
+  (let ((link (save-excursion
+		(backward-char)
+		(org-element-context
+	(when (string= "file" (org-element-property :type link))
+	  (let* ((old-path (org-element-property :path link))
+		 (new-path (let ((abspath (expand-file-name
+ old-path
+ (file-name-directory file
+			 (if includer-file
+ (file-relative-name
+  abspath (file-name-directory includer-file))
+			   abspath
+	(insert (let ((new (org-element-copy link)))
+		  (org-element-put-property new :path new-path)
+		  (when (org-element-property :contents-begin link)
+			(org-element-adopt-elements
+			 new
+			 (buffer-substring
+			  (org-element-property :contents-begin link)
+			  (org-element-property :contents-end link
+		  (delete-region (org-element-property :begin link)
+ (org-element-property :end link))
+		  (org-element-interpret-data new)))
 (when lines
   (let* ((lines (split-string lines "-"))
 	 (lbeg (string-to-number (car lines)))
diff --git a/testing/examples/includer-with-links.org b/testing/examples/includer-with-links.org
new file mode 100644
index 0..f97b76375
--- /dev/null
+++ b/testing/examples/includer-with-links.org
@@ -0,0 +1,19 @@
+* Source
+
+#+INCLUDE: "./subdir/includee-with-links.org::*Links" :only-contents t
+
+
+* Target
+
+File link with description
+[[file:subdir/setupfile2.org][A setupfile]]
+
+File link pointing to a specific header
+[[file:normal.org::top]]
+
+Bracketed file link without description
+[[file:subdir/setupfile.org]]
+
+Plain file link
+file:subdir/setupfile.org
+
diff --git a/testing/examples/subdir/includee-with-links.org b/testing/examples/subdir/includee-with-links.org
new file mode 100644
index 0..1f734ab86
--- /dev/null
+++ b/testing/examples/subdir/includee-with-links.org
@@ -0,0 +1,12 @@
+* Links
+File link with description
+[[file:setupfile2.org][A setupfile]]
+
+File link pointing to a 

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Nicolas Goaziou]

Hello,

Daniel P Gomez  writes:

> I noticed that (buffer-file-name (buffer-base-buffer)) will always
> return nil because `org-export--prepare-file-contents` is called in
> `org-export-include-keyword` after a call to `with-temp-buffer`. Two
> possible solutions to this issue would be either 1. passing the
> includer-file as an extra parameter to
> `org-export--prepare-file-contents` and then using
> `file-relative-name` to generate a relative path, or alternatively 2 .
> passing the matched string that points to the file to be included.
> Example:
>
> #+INCLUDE: "directory/file.org"
>
> Here, if file.org contains a link [[other/image.png]], then all one
> has to do is append the (file-name-directory matched) to the old-path.
> In this example this would result in directory/other/image.png.
>
> This second solution does not require a call to (buffer-file-name
> (buffer-base-buffer)), but seems hackish in the sense that we would
> pass 2 redundant arguments to `org-export-prepare-file-contents`: both
> the expanded and the non-expanded include-file filename.
> Perhaps I'm missing a simpler 3rd solution?

I think solution 1 is fine.

> If we opt for solution 1 then new-path could be made relative here
>>  ;; (org-element-put-property new :path new-path)
>
> (org-element-put-property
>new :path
>(if includer-file
>(file-relative-name
> new-path (file-name-directory includer-file))
>  new-path))

Indeed. However, the (if includer-file ...) should be integrated in
new-path binding, IMO.

> I will attempt to write them once the implementation is completed.

Great. Thank you!

Regards,

-- 
Nicolas Goaziou0x80A93738

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Daniel P Gomez]

Hi,


> `org-export--prepare-file-contents' is called from the including
> document, so you can get its path with (buffer-file-name
> (buffer-base-buffer)).
>
> However, we need to handle the case where the including buffer is not
> associated to a file, i.e., the Sexp above returns nil.
>

I noticed that (buffer-file-name (buffer-base-buffer)) will always
return nil because `org-export--prepare-file-contents` is called in
`org-export-include-keyword` after a call to `with-temp-buffer`. Two
possible solutions to this issue would be either 1. passing the
includer-file as an extra parameter to
`org-export--prepare-file-contents` and then using
`file-relative-name` to generate a relative path, or alternatively 2 .
passing the matched string that points to the file to be included.
Example:

#+INCLUDE: "directory/file.org"

Here, if file.org contains a link [[other/image.png]], then all one
has to do is append the (file-name-directory matched) to the old-path.
In this example this would result in directory/other/image.png.

This second solution does not require a call to (buffer-file-name
(buffer-base-buffer)), but seems hackish in the sense that we would
pass 2 redundant arguments to `org-export-prepare-file-contents`: both
the expanded and the non-expanded include-file filename.
Perhaps I'm missing a simpler 3rd solution?


> It would be nice to add a comment explaining what we are going to do.
>
Of course.

;; Adapt all file links within the included document that
;; contain relative paths in order to make these paths
;; relative to the base document, or absolute

> I suggest the following inner part:
>
> (when (string= "file" (org-element-property :type link))
>   (let* ((old-path (org-element-property :path link))
>  (new-path (expand-file-name old-path (file-name-directory 
> file

I noticed that the call to delete here will break
`org-element-adopt-elements` later on since the real buffer positions
will be changed and we are still querying for old, invalid positions
stored in link.
> ;; (delete-region (org-element-property :begin link)
> ;;   (org-element-property :end link))
> (insert (let ((new (org-element-copy link)))

If we opt for solution 1 then new-path could be made relative here
>  ;; (org-element-put-property new :path new-path)

(org-element-put-property
   new :path
   (if includer-file
   (file-relative-name
new-path (file-name-directory includer-file))
 new-path))

>   (when (org-element-property :contents-begin link)
> (org-element-adopt-elements new
>   (buffer-substring (org-element-property :contents-begin 
> link)
> (org-element-property :contents-end 
> link

Deleting immediately before the insertion works.
(delete-region (org-element-property :begin link)
   (org-element-property :end link))

>   (org-element-interpret-data new)
>
> Also, would you mind adding a test in "text-ox.el", within
> `test-org-export/expand-include'?

I will attempt to write them once the implementation is completed.

Thanks for the support.

Regards,

Daniel

> Regards,
>
> --
> Nicolas Goaziou

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Nicolas Goaziou]

Hello,

Daniel P Gomez  writes:

> I've adapted the code such that it handles both bracketed and
> unbracketed links, and links with descriptions.
> As it is now, the changes are always automatically applied.

Thank you.

> I couldn't find a simple way of rewriting links without making them
> absolute, as `org-export--prepare-file-contents` does not have access
> to the path of the including file, only of the included file.

`org-export--prepare-file-contents' is called from the including
document, so you can get its path with (buffer-file-name
(buffer-base-buffer)).

However, we need to handle the case where the including buffer is not
associated to a file, i.e., the Sexp above returns nil.

> +(goto-char (point-min))
> +(while (re-search-forward org-any-link-re nil t)
> +  (let ((link (save-excursion
> + (backward-char)
> + (org-element-context

It would be nice to add a comment explaining what we are going to do.

> + (when (string= (org-element-property :type link) "file")
> +   (let* ((has-bracket (string=
> +(org-element-property :format link) "bracket"))
> +  (has-content (org-element-property :contents-begin link))
> +  (old-path (org-element-property :path link))
> +  (new-path (expand-file-name old-path
> +  (file-name-directory file)))
> +  (raw-new-link
> +   (concat "file:" new-path))
> +  (new-link
> +   (cond
> +((and has-bracket (not has-content))
> + (concat "[[" raw-new-link "]]"))
> +((and has-bracket has-content)
> + (let ((description
> +(buffer-substring
> + (org-element-property :contents-begin link)
> + (org-element-property :contents-end link
> +   (concat "[[" raw-new-link "][" description "]]")))
> +(t raw-new-link
> + (apply #'delete-region (list (org-element-property :begin link)
> +  (org-element-property :end link)))
> + (insert new-link)

I suggest the following inner part:

(when (string= "file" (org-element-property :type link))
  (let* ((old-path (org-element-property :path link))
 (new-path (expand-file-name old-path (file-name-directory file
(delete-region (org-element-property :begin link)
   (org-element-property :end link))
(insert (let ((new (org-element-copy link)))
  (org-element-put-property new :path new-path)
  (when (org-element-property :contents-begin link)
(org-element-adopt-elements new
  (buffer-substring (org-element-property :contents-begin 
link)
(org-element-property :contents-end 
link
  (org-element-interpret-data new)
  
Also, would you mind adding a test in "text-ox.el", within
`test-org-export/expand-include'?

Regards,

-- 
Nicolas Goaziou

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Daniel P Gomez]

I've adapted the code such that it handles both bracketed and
unbracketed links, and links with descriptions.
As it is now, the changes are always automatically applied.

> Also, rewriting needs not be always absolute path, if both directories
> share a common root.

I couldn't find a simple way of rewriting links without making them
absolute, as `org-export--prepare-file-contents` does not have access
to the path of the including file, only of the included file.



On Thu, Mar 1, 2018 at 11:42 PM, Nicolas Goaziou  wrote:
> Hello,
>
> Daniel P Gomez  writes:
>
>> Currently when passing the :absolute-paths toggle to an include
>> derivative, as in :
>>
>> #+INCLUDE: file.org :absolute-paths t
>>
>> The function `org-export--prepare-file-contents` will automatically
>> deduce the directory from file.org and adapt links by calling:
>> `(new-path (expand-file-name old-path (file-name-directory file)))`.
>> I could either make this a default option, such that links get
>> corrected but users can overwrite this by calling `:absolute-paths
>> nil`, or I could completely remove this toggle and always correct
>> links no matter what.
>
> I am wondering about the latter. If there is no reason to keep broken
> file names, it makes sense to automatically apply it.
>
> Also, rewriting needs not be always absolute path, if both directories
> share a common root.
>
>> One question regarding the implementation, currently I'm deleting the
>> link with a call to `delete-region` and using `(insert "[[file:"
>> new-path "]]")` to insert the corrected one. This does not take into
>> consideration whether links are bracketed or not ( is there a
>> functional difference if links are not bracketed?).
>
> Yes. White spaces are handled differently in each link type.
>
>> Also, my approach completely disregards link descriptions, which may
>> be relevant if the linked file would be, for example, an html
>> document. Would there be a cleaner org approach to replace the path
>> keeping the description?
>
> You could check :contents-begin and :contents-end for the link. If they
> are not nil, extract this region, and insert it again in the new link.
>
> Regards,
>
> --
> Nicolas Goaziou0x80A93738


0001-Fix-file-links-when-using-INCLUDE.patch
Description: Binary data

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Nicolas Goaziou]

Hello,

Daniel P Gomez  writes:

> Currently when passing the :absolute-paths toggle to an include
> derivative, as in :
>
> #+INCLUDE: file.org :absolute-paths t
>
> The function `org-export--prepare-file-contents` will automatically
> deduce the directory from file.org and adapt links by calling:
> `(new-path (expand-file-name old-path (file-name-directory file)))`.
> I could either make this a default option, such that links get
> corrected but users can overwrite this by calling `:absolute-paths
> nil`, or I could completely remove this toggle and always correct
> links no matter what.

I am wondering about the latter. If there is no reason to keep broken
file names, it makes sense to automatically apply it.

Also, rewriting needs not be always absolute path, if both directories
share a common root.

> One question regarding the implementation, currently I'm deleting the
> link with a call to `delete-region` and using `(insert "[[file:"
> new-path "]]")` to insert the corrected one. This does not take into
> consideration whether links are bracketed or not ( is there a
> functional difference if links are not bracketed?).

Yes. White spaces are handled differently in each link type.

> Also, my approach completely disregards link descriptions, which may
> be relevant if the linked file would be, for example, an html
> document. Would there be a cleaner org approach to replace the path
> keeping the description?

You could check :contents-begin and :contents-end for the link. If they
are not nil, extract this region, and insert it again in the new link.

Regards,

-- 
Nicolas Goaziou0x80A93738

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Daniel P Gomez]

Currently when passing the :absolute-paths toggle to an include
derivative, as in :

#+INCLUDE: file.org :absolute-paths t

The function `org-export--prepare-file-contents` will automatically
deduce the directory from file.org and adapt links by calling:
`(new-path (expand-file-name old-path (file-name-directory file)))`.
I could either make this a default option, such that links get
corrected but users can overwrite this by calling `:absolute-paths
nil`, or I could completely remove this toggle and always correct
links no matter what.

One question regarding the implementation, currently I'm deleting the
link with a call to `delete-region` and using `(insert "[[file:"
new-path "]]")` to insert the corrected one. This does not take into
consideration whether links are bracketed or not ( is there a
functional difference if links are not bracketed?). Also, my approach
completely disregards link descriptions, which may be relevant if the
linked file would be, for example, an html document.  Would there be a
cleaner org approach to replace the path keeping the description?

Thank you.

Regards,
Daniel

On Thu, Mar 1, 2018 at 7:32 PM, Nicolas Goaziou  wrote:
> Hello,
>
> Daniel P Gomez  writes:
>
>> If there is any interest, I've created a patch adding support for a
>> :absolute-paths functionality.
>> As I said, I don't have much experience writing Elisp, so any
>> suggestions on how to improve the code would be appreciated. If there
>> is any interest in adding this feature to Org I could try to clean it
>> up.
>
> Thank you.
>
> Would it make sense to automatically deduce this directory from included
> file and apply this automatically?
>
> Regards,
>
> --
> Nicolas Goaziou

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Nicolas Goaziou]

Hello,

Daniel P Gomez  writes:

> If there is any interest, I've created a patch adding support for a
> :absolute-paths functionality.
> As I said, I don't have much experience writing Elisp, so any
> suggestions on how to improve the code would be appreciated. If there
> is any interest in adding this feature to Org I could try to clean it
> up.

Thank you.

Would it make sense to automatically deduce this directory from included
file and apply this automatically?

Regards,

-- 
Nicolas Goaziou

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Daniel P Gomez]

If there is any interest, I've created a patch adding support for a
:absolute-paths functionality.
As I said, I don't have much experience writing Elisp, so any
suggestions on how to improve the code would be appreciated. If there
is any interest in adding this feature to Org I could try to clean it
up.

Patch attached.


On Wed, Feb 28, 2018 at 9:11 PM, Daniel P Gomez  wrote:
> Sorry for the ignorance, but where could I look up how to use a parse
> tree filter? And where would these modifications make sense?
> Any pointers towards documentation, functions, or any help of any kind
> would be appreciated. I must say I am a bit lost.
>
> Thank you in advance,
>
> Daniel
>
> On Wed, Feb 28, 2018 at 6:51 PM, Nicolas Goaziou  
> wrote:
>> Hello,
>>
>> Daniel P Gomez  writes:
>>
>>> If I include a file child.org in parent.org using the #+INCLUDE
>>> derivative, and the following two conditions are true:
>>>
>>> 1. child.org and parent.org are not in the same directory nor share
>>> directory structure,
>>> 2. child.org contains file links with relative file paths,
>>>
>>> then exporting parent.org will produce a file with broken links.
>>>
>>> Is there a way to have org sort this out?
>>>
>>> Or even make this optional,
>>> say by using:
>>>
>>> #+INCLUDE: child.org :fix-paths t
>>>
>>> I guess the quickest (but perhaps not cleanest) way to have this
>>> feature would be to convert relative paths in file links into absolute
>>> paths within `org-export-expand-include-keyword`. I'm not that well
>>> versed in Elisp yet to do this yet, though.
>>>
>>> Are there any known solutions to this, or suggestions on how to get it
>>> working?
>>
>> As you suggest, I would use a parse tree filter that turns every
>> relative file link into an absolute one.
>>
>> Regards,
>>
>> --
>> Nicolas Goaziou


0001-Add-support-for-absolute-paths-in-INCLUDE-derivative.patch
Description: Binary data

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Daniel P Gomez]

Sorry for the ignorance, but where could I look up how to use a parse
tree filter? And where would these modifications make sense?
Any pointers towards documentation, functions, or any help of any kind
would be appreciated. I must say I am a bit lost.

Thank you in advance,

Daniel

On Wed, Feb 28, 2018 at 6:51 PM, Nicolas Goaziou  wrote:
> Hello,
>
> Daniel P Gomez  writes:
>
>> If I include a file child.org in parent.org using the #+INCLUDE
>> derivative, and the following two conditions are true:
>>
>> 1. child.org and parent.org are not in the same directory nor share
>> directory structure,
>> 2. child.org contains file links with relative file paths,
>>
>> then exporting parent.org will produce a file with broken links.
>>
>> Is there a way to have org sort this out?
>>
>> Or even make this optional,
>> say by using:
>>
>> #+INCLUDE: child.org :fix-paths t
>>
>> I guess the quickest (but perhaps not cleanest) way to have this
>> feature would be to convert relative paths in file links into absolute
>> paths within `org-export-expand-include-keyword`. I'm not that well
>> versed in Elisp yet to do this yet, though.
>>
>> Are there any known solutions to this, or suggestions on how to get it
>> working?
>
> As you suggest, I would use a parse tree filter that turns every
> relative file link into an absolute one.
>
> Regards,
>
> --
> Nicolas Goaziou

Re: [O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Nicolas Goaziou]

Hello,

Daniel P Gomez  writes:

> If I include a file child.org in parent.org using the #+INCLUDE
> derivative, and the following two conditions are true:
>
> 1. child.org and parent.org are not in the same directory nor share
> directory structure,
> 2. child.org contains file links with relative file paths,
>
> then exporting parent.org will produce a file with broken links.
>
> Is there a way to have org sort this out?
>
> Or even make this optional,
> say by using:
>
> #+INCLUDE: child.org :fix-paths t
>
> I guess the quickest (but perhaps not cleanest) way to have this
> feature would be to convert relative paths in file links into absolute
> paths within `org-export-expand-include-keyword`. I'm not that well
> versed in Elisp yet to do this yet, though.
>
> Are there any known solutions to this, or suggestions on how to get it
> working?

As you suggest, I would use a parse tree filter that turns every
relative file link into an absolute one.

Regards,

-- 
Nicolas Goaziou

[O] How to keep correct filepaths when using the #+INCLUDE derivative?

[Daniel P Gomez]

Dear org users,

If I include a file child.org in parent.org using the #+INCLUDE
derivative, and the following two conditions are true:

1. child.org and parent.org are not in the same directory nor share
directory structure,
2. child.org contains file links with relative file paths,

then exporting parent.org will produce a file with broken links.

Is there a way to have org sort this out? Or even make this optional,
say by using:

#+INCLUDE: child.org :fix-paths t

I guess the quickest (but perhaps not cleanest) way to have this
feature would be to convert relative paths in file links into absolute
paths within `org-export-expand-include-keyword`. I'm not that well
versed in Elisp yet to do this yet, though.

Are there any known solutions to this, or suggestions on how to get it working?

Thanks in advance,

Daniel