Re: [O] [RFC] [PATCH] [babel] read description lists as lists of lists

2014-09-28 Thread Thorsten Jolitz
Aaron Ecay aarone...@gmail.com writes:

Hi Aaron,

 So the question is, how to provide a consistent language-agnostic view
 of org structure to other languages. 

well, that would be the parse-tree normally, its a nested list
containing all info about org structure. 

I tried to make two Lisps talk to each other (Emacs Lisp and PicoLisp),
and initially thought it would be easy - just send lists back and
forth. But when these lists are parse-trees produced by the Org parser
framework, its not that straight-forward anymore, because these lists
are send as strings and then read again by the other side, and there are
lots of peculiarities in the string representation of Org parse-trees
that make other languages choke, even if they otherwise understand lists
very well.

As an example, this nested list representation of this created
headline is simple enough, PicoLisp can read it as-is:

#+NAME: hl1
#+BEGIN_SRC emacs-lisp :results verbatim
  (org-dp-create 'headline Hallo World 'data '(:name foo)
 :level 2
 :title 2nd level
 :todo-keyword NEXT
 :tags '(office)
 :priority ?B)
#+END_SRC

#+results: hl1
: (headline (:level 2 :title 2nd level :todo-keyword NEXT :tags
(office) :priority 66 :name foo) (section nil Hallo World))

#+BEGIN_SRC emacs-lisp :var lst=hl1
 (length lst)
#+END_SRC

#+results:
: 131

#+BEGIN_SRC picolisp :var lst=hl1 :results pp
 (in NIL (length lst))
#+END_SRC

#+results:
: 131

#+BEGIN_SRC picolisp :var lst=hl1 :results pp
 (in NIL (last (car (str lst
#+END_SRC

#+results:
: (section nil Hallo World)

So this can be read by PicoLisp (and probably many other list
processing languages), except that nil is not NIL in PicoLisp.

But when parsing this Org buffer its easy to see that syntax and
semantics of Emacs Lisp circular lists as well as of strings with
text-properties cannot easily be consumed by other languages (expecially
when # is their comment-start character). Some post-processing is
necessary, e.g. suppressing text properties, nil - NIL, \n - ^J, ^ -
\^, # - \# etc., and I doubt this can be done in a language agnostic
way. And then the circular relationships needs to be resolved to
something the target language can understand...

#+BEGIN_SRC emacs-lisp :results pp
 (org-element-parse-buffer)
#+END_SRC

#+results: (org-data nil (headline (:raw-value --text follows this
line-- :begin 1 :end 17295 :pre-blank 0 :contents-begin 30
:contents-end 17295 :level 1 :priority nil :tags nil :todo-keyword nil
:todo-type nil :post-blank 0 :footnote-section-p nil :archivedp nil
:commentedp nil :post-affiliated 1 :title (#(--text follows this
line-- 0 26 (:parent #1))) :parent #0) [...cut...] (paragraph (:begin
17274 :end 17295 :contents-begin 17274 :contents-end 17295 :post-blank 0
:post-affiliated 17274 :parent #3) #(-- \ncheers,\nThorsten\n 0 21
(:parent #4)

-- 
cheers,
Thorsten




Re: [O] [RFC] [PATCH] [babel] read description lists as lists of lists

2014-09-28 Thread Nicolas Goaziou
Aaron Ecay aarone...@gmail.com writes:

 Let’s back up a step.  The representation I am targeting with my change
 is what babel uses to ship a list off as input to code in a babel block.
 This code could be emacs lisp, but it could also be R, python, etc.  So
 the question is, how to provide a consistent language-agnostic view of
 org structure to other languages.  The resultant structure doesn’t hang
 around inside babel, it just gets handed off to a code block.

I know. But another internal representation is an additional maintenance
burden.

 It’s not work that I’m afraid of: I offered to rewrite both babel and
 radio lists in terms of org-elements.  Maybe I am insane, as you imply.

I never wrote, implied or even thought you were insane.

 What if I rewrote org-babel-read-list in terms of org-elements?  That
 would satisfy me wrt. babel, and wouldn’t necessitate disturbing
 org-list-parse-list, radio lists, or indeed anything outside of babel.

That is even better.


Regards,

-- 
Nicolas Goaziou



Re: [O] [RFC] [PATCH] [babel] read description lists as lists of lists

2014-09-27 Thread Aaron Ecay
Hi Nicolas,

2014ko irailak 26an, Nicolas Goaziou-ek idatzi zuen:
 
 Why?  Babel’s representation is for babel.
 
 Which I strongly frown upon.

Let’s back up a step.  The representation I am targeting with my change
is what babel uses to ship a list off as input to code in a babel block.
This code could be emacs lisp, but it could also be R, python, etc.  So
the question is, how to provide a consistent language-agnostic view of
org structure to other languages.  The resultant structure doesn’t hang
around inside babel, it just gets handed off to a code block.

 
 org-list-parse-list/-to-generic’s is for radio lists (although as I’ve
 said this connection seems accidental rather than essential).  Babel
 calls org-list-parse-list, but I don’t see why it should be forbidden
 from doing more processing on the result before passing it along
 (indeed, it already does some processing to remove the list type
 indicators, remove nested structure, etc.).
 
 It is best to use as much common ground as possible. We should strive to
 decrease need for such processing, not the other way.
 
 As I already stated in my first answer, in the long run, it is the only
 sane way to proceed. I agree it is less work to simply tweak Babel right
 now and ignore the whole Org ecosystem, but it does no good to Org as
 a whole.

It’s not work that I’m afraid of: I offered to rewrite both babel and
radio lists in terms of org-elements.  Maybe I am insane, as you imply.

What I’m afraid of is old and disused sort-of-APIs like
org-list-parse-list calcifying and preventing good things from happening
to parts of org that people actually use.

What if I rewrote org-babel-read-list in terms of org-elements?  That
would satisfy me wrt. babel, and wouldn’t necessitate disturbing
org-list-parse-list, radio lists, or indeed anything outside of babel.

 
 `org-list-parse-list' handles nested lists just fine. Another advantage
 of not re-inventing the wheel in every part of Org.

I know.  But babel’s processing of parse-list’s output strips the nested
structure:

#+name: a-list
- foo
- bar
  - abc
  - def
- baz

#+begin_src emacs-lisp :var lst=a-list
  (pp-to-string lst)
#+end_src

#+RESULTS:
: (foo bar baz)

That’s because it’s hard to come up with a good representation of a nested
list in a language-agnostic way.  “List of strings” is a straightforward
datatype in every language babel supports, but not all of them have a
convenient “labeled n-ary tree with string leaves” (which you’d need for
arbitrarily nested lists).

Thanks,

-- 
Aaron Ecay



Re: [O] [RFC] [PATCH] [babel] read description lists as lists of lists

2014-09-26 Thread Nicolas Goaziou
Hello,

Aaron Ecay aarone...@gmail.com writes:

 Isn’t the org-element format also easy to work on?  It requires a bit
 more than just car and cdr, but it’s well documented and used in many
 places across the code base (= cognitive burden to use is lower).  It’s
 also easy to produce in the sense that org-element.el already exists for
 independent reasons; we just have to use it.

It is not as easy to produce ex nihilo, i.e., without any Org syntax
under point. But, really, I do not mind if both radio lists and Babel
move to this internal syntax. It will require much more work, though.

Also, it doesn't mean we can remove or replace `org-list-parse-list' and
`org-list-to-generic'.

 Radio lists is a feature, org-list-to-generic is an implementation.  We
 can change the implementation without changing the user-visible aspects
 of the feature.  IOW, nothing about the user-facing functionality of
 org-list-to-generic requires it to accept a particular type of argument
 (as long as that arg is some representation or other of a list).

I agree.

 One approach would be to detect when it’s called from a non-org-mode
 buffer, and copy the text into a temporary org-mode buffer for parsing.
 Then org-element would be available.

Of course, if the internal representation is changed to Elements', that
is probably the way to go.

 IDK.  You’re probably in a better position to know that than I am.  There’s
 only one message even mentioning them (very tangentially) in my 2-ish years
 of messages from the list: http://mid.gmane.org/87obc6scty@pank.eu.
 I’m not advocating their removal or deprecation, but they certainly seem
 like the tail and not the dog when considering what parts of org ought to
 wag what others.

I think you are missing my point.

Again, I'm fine with any improvement needed for Babel, but other, even
remotely, related parts should be moved along. This is about
consistency. I certainly don't want to see various parts of Org drift
away. Or, to put it differently: mind the tail, do not act as if the dog
had none.

 Why?  Babel’s representation is for babel.

Which I strongly frown upon.

 org-list-parse-list/-to-generic’s is for radio lists (although as I’ve
 said this connection seems accidental rather than essential).  Babel
 calls org-list-parse-list, but I don’t see why it should be forbidden
 from doing more processing on the result before passing it along
 (indeed, it already does some processing to remove the list type
 indicators, remove nested structure, etc.).

It is best to use as much common ground as possible. We should strive to
decrease need for such processing, not the other way.

As I already stated in my first answer, in the long run, it is the only
sane way to proceed. I agree it is less work to simply tweak Babel right
now and ignore the whole Org ecosystem, but it does no good to Org as
a whole.

 I dunno if I’d call my proposal an “internal plain list representation,”
 but rather “babel’s interpretation of plain lists.”

See above.

 Ordered and unordered lists are lists of strings (exactly as now).
 Description lists are lists of 2-element lists, each of the form
 (“TERM” “DESCRIPTION”) (unlike now, when they are lists of strings of
 the form “TERM :: DESCRIPTION”).

 It might be nice to handle nested lists somehow, if a sensible design
 can be created, but it looks like babel just discards them currently.
 So I propose to leave this unchanged, for the present at least:

`org-list-parse-list' handles nested lists just fine. Another advantage
of not re-inventing the wheel in every part of Org.


Regards,

-- 
Nicolas Goaziou0x80A93738



Re: [O] [RFC] [PATCH] [babel] read description lists as lists of lists

2014-09-24 Thread Nicolas Goaziou
Hello,

Aaron Ecay aarone...@gmail.com writes:

 I think I can remove these three functions (-parse-list, -to-subtree,
 and -to-generic), and rewrite their callers to use org-element.  Thus,
 the org-list-parse-list format would be eradicated from the code base
 incl. contrib (AFAICT).  Can I do that, or do I need to care about
 preserving backwards compatibility with external callers of these
 functions?  If backwards compatibility must be preserved, may I mark
 these functions as deprecated and what is the minimum period (measured
 in calendar time and/or org versions) that should pass before their
 removal?

You cannot do that. This is not about backwards compatibility.

`org-list-parse-list' generates an easy to produce and work on internal
representation for lists (similar to what `org-table-to-lisp' does for
tables). `org-list-to-generic' is used for radio lists (similar to
`org-table-to-generic'): it is expected to consume
a `org-list-parse-list'-like return value.

IOW both functions are important and are not meant to be replaced by
Elements (however, at some point `org-list-to-generic' should use
ox.el, but that's for another day).

Note that since `org-list-parse-list' is meant for extraneous buffer, it
cannot rely on Elements. It shouldn't even use `org-list-struct' because
I plan to make this function use Elements, too.

 The babel feature is compelling to me (and I guess Chuck) on its
 own.  It’s familiar (e.g. in the case of tables) that babel gets to
 have its own data format for org elements.

It's the same for lists. Internal representation for lists should come
from org-list.el, not from Babel. Internal representation for tables
comes from org-table.el, too.

 I’m happy to undertake the above-described demolition job on
 org-list-parse-list in order to offset the added complexity from the
 babel change (we can call it a cap-and-trade system). But given that
 org-list-parse-list is a marginal part of the code base and perhaps
 moribund in the era of org-element

I don't consider radio lists moribund. Are they?

 I don’t really think it’s worth it (to me) to try and engineer an
 improvement to it in order to enable the babel feature.

It is not (or should not be) a Babel feature.

Anyway, it's not about rewriting `org-list-parse-list', but if Babel
understands a new representation for plain lists, this function should
be able to generate it and `org-list-to-generic' should be able to
interpret it.

Could you detail the exact specifications of the suggested internal
plain list representation?


Regards,

-- 
Nicolas Goaziou



Re: [O] [RFC] [PATCH] [babel] read description lists as lists of lists

2014-09-24 Thread Aaron Ecay
Hi Nicolas,

Thanks for the discussion.

2014ko irailak 24an, Nicolas Goaziou-ek idatzi zuen:
 
 You cannot do that. This is not about backwards compatibility.
 
 `org-list-parse-list' generates an easy to produce and work on internal
 representation for lists (similar to what `org-table-to-lisp' does for
 tables). 

Isn’t the org-element format also easy to work on?  It requires a bit
more than just car and cdr, but it’s well documented and used in many
places across the code base (= cognitive burden to use is lower).  It’s
also easy to produce in the sense that org-element.el already exists for
independent reasons; we just have to use it.

 `org-list-to-generic' is used for radio lists (similar to
 `org-table-to-generic'): it is expected to consume a
 `org-list-parse-list'-like return value.

Radio lists is a feature, org-list-to-generic is an implementation.  We
can change the implementation without changing the user-visible aspects
of the feature.  IOW, nothing about the user-facing functionality of
org-list-to-generic requires it to accept a particular type of argument
(as long as that arg is some representation or other of a list).

 
 IOW both functions are important and are not meant to be replaced by
 Elements (however, at some point `org-list-to-generic' should use
 ox.el, but that's for another day).

...doesn’t using ox.el entail using elements?

 
 Note that since `org-list-parse-list' is meant for extraneous buffer, it
 cannot rely on Elements. It shouldn't even use `org-list-struct' because
 I plan to make this function use Elements, too.

One approach would be to detect when it’s called from a non-org-mode
buffer, and copy the text into a temporary org-mode buffer for parsing.
Then org-element would be available.

 
 The babel feature is compelling to me (and I guess Chuck) on its
 own.  It’s familiar (e.g. in the case of tables) that babel gets to
 have its own data format for org elements.
 
 It's the same for lists. Internal representation for lists should come
 from org-list.el, not from Babel. Internal representation for tables
 comes from org-table.el, too.

Hmm.  I had missed that, you are correct.

 
 I’m happy to undertake the above-described demolition job on
 org-list-parse-list in order to offset the added complexity from the
 babel change (we can call it a cap-and-trade system). But given that
 org-list-parse-list is a marginal part of the code base and perhaps
 moribund in the era of org-element
 
 I don't consider radio lists moribund. Are they?

IDK.  You’re probably in a better position to know that than I am.  There’s
only one message even mentioning them (very tangentially) in my 2-ish years
of messages from the list: http://mid.gmane.org/87obc6scty@pank.eu.
I’m not advocating their removal or deprecation, but they certainly seem
like the tail and not the dog when considering what parts of org ought to
wag what others.

(I’d hope their usefulness would eventually naturally wane as org
becomes compelling enough that people commit to it wholesale, rather
than relying just on the list editing features while living in another
document composition regime.  Whether it’s worth keeping them around
as a sort of training wheels I don’t really have an opinion on.)

 
 I don’t really think it’s worth it (to me) to try and engineer an
 improvement to it in order to enable the babel feature.
 
 It is not (or should not be) a Babel feature.
 
 Anyway, it's not about rewriting `org-list-parse-list', but if Babel
 understands a new representation for plain lists, this function should
 be able to generate it and `org-list-to-generic' should be able to
 interpret it.

Why?  Babel’s representation is for babel.
org-list-parse-list/-to-generic’s is for radio lists (although as I’ve
said this connection seems accidental rather than essential).  Babel
calls org-list-parse-list, but I don’t see why it should be forbidden
from doing more processing on the result before passing it along
(indeed, it already does some processing to remove the list type
indicators, remove nested structure, etc.).

 
 Could you detail the exact specifications of the suggested internal
 plain list representation?

I dunno if I’d call my proposal an “internal plain list representation,”
but rather “babel’s interpretation of plain lists.”

Ordered and unordered lists are lists of strings (exactly as now).
Description lists are lists of 2-element lists, each of the form
(“TERM” “DESCRIPTION”) (unlike now, when they are lists of strings of
the form “TERM :: DESCRIPTION”).

It might be nice to handle nested lists somehow, if a sensible design
can be created, but it looks like babel just discards them currently.
So I propose to leave this unchanged, for the present at least:

#+name: data
- foo
- bar
  - baz
  - quux
- abc

#+begin_src emacs-lisp :var data=data
(pp-to-string data)
#+end_src

#+RESULTS:
: (foo bar abc)

-- 
Aaron Ecay



Re: [O] [RFC] [PATCH] [babel] read description lists as lists of lists

2014-09-22 Thread Aaron Ecay
Hi Nicolas,

Thanks for your feedback.

2014ko irailak 20an, Nicolas Goaziou-ek idatzi zuen:
 
 The problem I see here is that you're introducing yet another internal
 representation for lists (along with element's and
 org-list-parse-list's). Worse, it can only be discovered when reading
 the docstring of a Babel internal function and will only benefit to
 Babel.
 
 If this new internal representation is better than current one, by all
 means, improve `org-list-parse-list', and document it in
 
   (info (org) Radio lists)
 
 This is more work, but, IMO, it is also the only sane way to proceed.

Indeed.  The internals of org-list are not pretty.  org-list-parse-list
has few callers:
- org-list-make-subtree
- org-babel-read-list
- org-toggle-heading

org-list-to-subtree has only two callers:
- org-list-make-subtree
- org-toggle-heading

org-list-to-generic also has only two callers:
- org-babel-insert-result
- org-list-to-subtree

I think I can remove these three functions (-parse-list, -to-subtree,
and -to-generic), and rewrite their callers to use org-element.  Thus,
the org-list-parse-list format would be eradicated from the code base
incl. contrib (AFAICT).  Can I do that, or do I need to care about
preserving backwards compatibility with external callers of these
functions?  If backwards compatibility must be preserved, may I mark
these functions as deprecated and what is the minimum period (measured
in calendar time and/or org versions) that should pass before their
removal?

The babel feature is compelling to me (and I guess Chuck) on its
own.  It’s familiar (e.g. in the case of tables) that babel gets to
have its own data format for org elements.  I’m happy to undertake
the above-described demolition job on org-list-parse-list in order
to offset the added complexity from the babel change (we can call it
a cap-and-trade system).  But given that org-list-parse-list is a
marginal part of the code base – and perhaps moribund in the era of
org-element – I don’t really think it’s worth it (to me) to try and
engineer an improvement to it in order to enable the babel feature.

WDYT?

-- 
Aaron Ecay



Re: [O] [RFC] [PATCH] [babel] read description lists as lists of lists

2014-09-20 Thread Nicolas Goaziou
Hello,

Aaron Ecay aarone...@gmail.com writes:

 The attached patch makes babel read description lists as lists of the
 following format: ((term description) ...).  The present default is
 to simply read in the text of each list item, yielding:
 (term :: description ...).

Thank you.

 Of course, it’s possible to interconvert between the two formats, but I
 think the greater structure of this proposal makes things easier for
 babel authors.  (Another way of thinking of the proposal is that it
 treats description lists like two-column tables.)

 What do people think?

The problem I see here is that you're introducing yet another internal
representation for lists (along with element's and
org-list-parse-list's). Worse, it can only be discovered when reading
the docstring of a Babel internal function and will only benefit to
Babel.

If this new internal representation is better than current one, by all
means, improve `org-list-parse-list', and document it in

  (info (org) Radio lists)

This is more work, but, IMO, it is also the only sane way to proceed.

 +THE result is a list of strings \(the list items), unless the

You only need to escape parenthesis at the beginning of a line.

 + (mapcar (lambda (el)
 +   (let ((s (split-string el  :: )))
 + (list (nth 0 s) (mapconcat #'identity (cdr s)  :: 

This is really awkward. You can use a regexp to extract the tag.


Regards,

-- 
Nicolas Goaziou



Re: [O] [RFC] [PATCH] [babel] read description lists as lists of lists

2014-09-19 Thread Charles Berry
Aaron Ecay aaronecay at gmail.com writes:

 
 
 Hello all,
 
 The attached patch makes babel read description lists as lists of the
 following format: ((term description) ...).  The present default is
 to simply read in the text of each list item, yielding:
 (term :: description ...).
 
 Of course, it’s possible to interconvert between the two formats, but I
 think the greater structure of this proposal makes things easier for
 babel authors.  (Another way of thinking of the proposal is that it
 treats description lists like two-column tables.)
 
 What do people think?

With that change 

#+BEGIN_SRC R :var a=my-desc-list
a
#+END_SRC

returns a data.frame with one column of terms and one of descriptions!

Which, no doubt, is what you were thinking.

This will be handy for creating DESCRIPTION files for R packages.

I love it. 

FWIW, rgrep-ing org-babel-\(ref-resolve\)\|\(read-result\) and clicking
on the links, I didn't catch any obvious problem cases.

HTH,

Chuck