Re: [O] Literate Programming - Continue a Source Block?
"Sebastien Vauban" writes: > The only case that pops up to my mind now, of such a use case where > overwriting could be "needed" (well, let's say useful) is for some pedagogical > document that one would write, where code is constructed from a simplistic > (and buggy) approach to a correct one. > > One could say: the code to do that is this one, and show the block contents. > > Then, discover problems to it, explain them in the document, and make a new > version of the block with the same name (for tangling reasons). > > Then, highlight some limitations of the new code, fix them in a new block with > the same name, etc. > > Does that mean it needs to be implemented? Up to you... I gave a similar example earlier, and I think Eric said that can be handled by just changing the tangle header for the code you overwrite. Not as convenient, though.
Re: [O] Literate Programming - Continue a Source Block?
Hi Eric and Neeum, Eric Schulte wrote: >> Overwriting is still not supported, but I don't know if that's all that >> important (I don't have an immediate need for it). And noweb by default >> did not have it either, so perhaps it's not needed for most tasks > > This was my thinking. > >> (OTOH, you may want to think about what the best solution is if later >> on you decide to add overwriting capability). > > If someone finds a real need for overwriting code blocks, hopefully the > specifics of their need will point towards an implementation. The only case that pops up to my mind now, of such a use case where overwriting could be "needed" (well, let's say useful) is for some pedagogical document that one would write, where code is constructed from a simplistic (and buggy) approach to a correct one. One could say: the code to do that is this one, and show the block contents. Then, discover problems to it, explain them in the document, and make a new version of the block with the same name (for tangling reasons). Then, highlight some limitations of the new code, fix them in a new block with the same name, etc. Does that mean it needs to be implemented? Up to you... Best regards, Seb -- Sebastien Vauban
Re: [O] Literate Programming - Continue a Source Block?
Neeum Zawan writes: > Eric Schulte writes: > >> How about the following solution, which is based on a new :noweb-ref >> header argument. >> >> When expanding ``noweb'' style references the bodies of all code block >> with /either/ a block name matching the reference name /or/ a :noweb-ref >> header argument matching the reference name will be concatenated >> together to form the replacement text. >> >> By setting this header argument at the sub-tree or file level, simple >> code block concatenation may be achieved. For example, when tangling >> the following Org-mode file, the bodies of code blocks will be >> concatenated into the resulting pure code file. > > Hi, > > Your example is not completely clear. I noticed you didn't put any names > for the source blocks that use the noweb-ref. Is it necessary not to > name them, The source code blocks may have names, as before all code block names should be unique or the behavior may be undefined. > or can one name them but their names will still have to be unique and > are orthogonal to concatenation (i.e. they have names, but the > concatenation is due to whatever the argument of noweb-ref is)? > The value of the :noweb-ref header argument supersedes the name of the code block for noweb expansion. To preserve existing semantics and to avoid requiring the use of the :noweb-ref header argument in simple cases, the code block name will be used to resolve noweb references in blocks which do not have a :noweb-ref header argument. > > I think either way, this solution serves my purpose. My original > suggestion was to have a header whose value would be "append" to allow > for concatenation if the name matches a previous source block. It seems > your solution above is the same except instead of looking at the name of > the source block, it looks at the argument of noweb-ref. > Yup, happy this sounds like it should work for your original need. > > Overwriting is still not supported, but I don't know if that's all that > important (I don't have an immediate need for it). And noweb by default > did not have it either, so perhaps it's not needed for most tasks This was my thinking. > (OTOH, you may want to think about what the best solution is if later > on you decide to add overwriting capability). > If someone finds a real need for overwriting code blocks, hopefully the specifics of their need will point towards an implementation. > > Thanks. Hope to see this or something like it soon. > You welcome, hope this turns out to be helpful -- Eric -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: > How about the following solution, which is based on a new :noweb-ref > header argument. > > When expanding ``noweb'' style references the bodies of all code block > with /either/ a block name matching the reference name /or/ a :noweb-ref > header argument matching the reference name will be concatenated > together to form the replacement text. > > By setting this header argument at the sub-tree or file level, simple > code block concatenation may be achieved. For example, when tangling > the following Org-mode file, the bodies of code blocks will be > concatenated into the resulting pure code file. Hi, Your example is not completely clear. I noticed you didn't put any names for the source blocks that use the noweb-ref. Is it necessary not to name them, or can one name them but their names will still have to be unique and are orthogonal to concatenation (i.e. they have names, but the concatenation is due to whatever the argument of noweb-ref is)? I think either way, this solution serves my purpose. My original suggestion was to have a header whose value would be "append" to allow for concatenation if the name matches a previous source block. It seems your solution above is the same except instead of looking at the name of the source block, it looks at the argument of noweb-ref. Overwriting is still not supported, but I don't know if that's all that important (I don't have an immediate need for it). And noweb by default did not have it either, so perhaps it's not needed for most tasks (OTOH, you may want to think about what the best solution is if later on you decide to add overwriting capability). Thanks. Hope to see this or something like it soon.
Re: [O] Literate Programming - Continue a Source Block?
Am 08.06.2011 17:20, schrieb Neeum Zawan: [...] Now the original noweb allows what I'm asking for. If you begin a source block with a name of an existing block but append an "=" symbol, it knows to append to that source block. It would be great if org-mode could add that capability. Another approach is that if multiple source blocks with the same name are found, the default behavior is to concatenate all those source blocks together (and then add a header option for overwrite if the user wanted to overwrite instead of append). Thoughts? I missed this feature too - I used it (so called additive macros) very often with funnelweb.
Re: [O] Literate Programming - Continue a Source Block?
Neeum Zawan writes: > Eric Schulte writes: >> It would be possible to also implement the concatenation behavior during >> noweb expansion, however I'd prefer to first wait for a response to my >> recent other email to this thread asking for a more clear explication of >> existing noweb behavior. >> >> The only remaining times when such concatenation behavior could be >> implemented would be during block reference expansion, and during block >> evaluation, but I think expanding at those times would be unnecessary >> and confusing. > > I'm having trouble understanding the difference between what you discuss > in the first paragraph and in the second. What's the difference between > noweb expansion and block reference expansion? > By block reference I meant when variables are assigned to the values of blocks through reference. So in the following example the "number" block has its body returned through "reference expansion". #+source: number #+begin_src emacs-lisp 4 #+end_src #+begin_src emacs-lisp :var num=number (* num 2) #+end_src Best -- Eric -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
Achim Gratz writes:
> Eric Schulte writes:
>> Does anyone on this list know the noweb system well enough to specify
>> its behavior in this regard, and to describe what functional changes
>> would be required to bring Babel into line with noweb behavior?
>
> Far from knowing it well, but the basics are, well basic:
>
> Documentation text is started via '@', code chunks are defined via
> '<>=' and a given chunk can be extended by '<>='
> with the same name again, which allows to intermingle documentation and
> definition in a linear fashion. Code chunk references are just
> '<>' without the equal sign and can be used non-linearly.
> For comparison with Babel the noweb Hackers Guide is probably a good
> source since it describes the pipeline representation which is used by
> the tools (and should maybe be emulated by the Babel backend).
>
> http://www.cs.tufts.edu/~nr/noweb/guide.html
>
Hi Achim,
Thanks for sending along this reference. That combined with the related
"Literate Programming Simplified" [1] serve as a very interesting
introduction to Noweb. I have not studied noweb previously and I am
impressed by the simplicity of the pipelined filter-based design.
Noweb and Babel are certainly very different beasts. Noweb achieves
sparse simplicity allowing use with a variety of tools while Babel lives
inside the very rich environment of Org-mode documents. Noweb enables
literate programming LP while babel is a reproducible research tool
which also provides support for LP.
It seems that lifting a solution whole-piece from noweb will not make
sense, as noweb does not have a need for unique code block names in the
same way as babel.
How about the following solution, which is based on a new :noweb-ref
header argument.
When expanding ``noweb'' style references the bodies of all code block
with /either/ a block name matching the reference name /or/ a :noweb-ref
header argument matching the reference name will be concatenated
together to form the replacement text.
By setting this header argument at the sub-tree or file level, simple
code block concatenation may be achieved. For example, when tangling
the following Org-mode file, the bodies of code blocks will be
concatenated into the resulting pure code file.
#+begin_src sh :tangle yes :noweb yes :shebang #!/bin/sh
<>
#+end_src
* the mount point of the fullest disk
:PROPERTIES:
:noweb-ref: fullest-disk
:END:
** query all mounted disks
#+begin_src sh
df \
#+end_src
** strip the header row
#+begin_src sh
|sed '1d' \
#+end_src
** sort by the percent full
#+begin_src sh
|awk '{print $5 " " $6}'|sort -n |tail -1 \
#+end_src
** extract the mount point
#+begin_src sh
|awk '{print $2}'
#+end_src
This should provide feature-parity with noweb, and satisfy most LP needs.
Best -- Eric
Footnotes:
[1] www.cs.tufts.edu/~nr/pubs/lpsimp.pdf
--
Eric Schulte
http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: > Rather than feeling our way forward step by step it seems that simply > following the behavior of noweb would both > 1. allow for easy transition between noweb and babel > 2. benefit from the years of experience and design accumulated in the >noweb project > > Does anyone on this list know the noweb system well enough to specify > its behavior in this regard, and to describe what functional changes > would be required to bring Babel into line with noweb behavior? Looking at the basic functionality: <<...>>= starts a code block. Further uses of it will just append. No delimiter for end (it ends when you start another code/doc block). It can be referenced using <<...>> (without the = symbol). The location of the reference is not relevant. So by default, no overwriting or ordering. However, noweb is extensible, and Achim linked to the syntax one can use to extend it. I haven't read those details. Hope that helps.
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: > Does anyone on this list know the noweb system well enough to specify > its behavior in this regard, and to describe what functional changes > would be required to bring Babel into line with noweb behavior? Far from knowing it well, but the basics are, well basic: Documentation text is started via '@', code chunks are defined via '<>=' and a given chunk can be extended by '<>=' with the same name again, which allows to intermingle documentation and definition in a linear fashion. Code chunk references are just '<>' without the equal sign and can be used non-linearly. For comparison with Babel the noweb Hackers Guide is probably a good source since it describes the pipeline representation which is used by the tools (and should maybe be emulated by the Babel backend). http://www.cs.tufts.edu/~nr/noweb/guide.html Notably, noweb assumes a tree (or forest) of code chunks of which the root nodes are to be tangled. Babel probably needs to take care of another two layers of complexity since it goes at least one step further than LP: after the equivalent of tangling all root chunks (producing the runnable programs) it needs to produce and integrate back into the documentation their results by running them and optionally keeping track of session information if they are not independent. That still assumes that the final documentation can be arrived at after running all programs and collecting their results, but the real fun starts when one of those programs produces another program that also needs to be run. :-) Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Waldorf MIDI Implementation & additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: > It would be possible to also implement the concatenation behavior during > noweb expansion, however I'd prefer to first wait for a response to my > recent other email to this thread asking for a more clear explication of > existing noweb behavior. > > The only remaining times when such concatenation behavior could be > implemented would be during block reference expansion, and during block > evaluation, but I think expanding at those times would be unnecessary > and confusing. I'm having trouble understanding the difference between what you discuss in the first paragraph and in the second. What's the difference between noweb expansion and block reference expansion? As for actual noweb behavior, I don't really know - will have to play with it and let you know...
Re: [O] Literate Programming - Continue a Source Block?
Hi Neeum, You are correct, the current implementation only specially concatenates blocks during tangling, *not* during noweb resolution. It would be possible to also implement the concatenation behavior during noweb expansion, however I'd prefer to first wait for a response to my recent other email to this thread asking for a more clear explication of existing noweb behavior. The only remaining times when such concatenation behavior could be implemented would be during block reference expansion, and during block evaluation, but I think expanding at those times would be unnecessary and confusing. Thanks -- Eric Neeum Zawan writes: > Eric Schulte writes: > >> Could you try the attached example file? I first evaluated the >> following elisp code to set the combination variable's value to append. > > Your example works if there are no noweb references. > > See the modified one where I have noweb references. Note that when > expanding the reference, it inserts only the first block it finds. > > > > > #+begin_src emacs-lisp :tangle yes :noweb yes > <> > > Random text > > <> > #+end_src > > > > * continued code blocks >:PROPERTIES: >:tangle: yes >:comments: yes >:END: > #+srcname: foo > #+begin_src emacs-lisp > (message "foo:%S" 1) > #+end_src > > #+begin_src emacs-lisp > (message "un-named") > #+end_src > #+srcname: bar > #+begin_src emacs-lisp > (message "bar:%S" 1) > #+end_src > > #+srcname: foo > #+begin_src emacs-lisp > (message "foo:%S" 2) > #+end_src > > #+srcname: bar > #+begin_src emacs-lisp > (message "bar:%S" 2) > #+end_src > > #+begin_src emacs-lisp :tangle no :results silent > (with-temp-buffer > (insert-file-contents "scraps.el") > (eval-buffer)) > #+end_src -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
Rather than feeling our way forward step by step it seems that simply following the behavior of noweb would both 1. allow for easy transition between noweb and babel 2. benefit from the years of experience and design accumulated in the noweb project Does anyone on this list know the noweb system well enough to specify its behavior in this regard, and to describe what functional changes would be required to bring Babel into line with noweb behavior? Thanks -- Eric Achim Gratz writes: > Eric Schulte writes: >> Thanks for sharing your thoughts. How would you feel about moving away >> from special source block names and moving towards implementing this >> behavior with a header argument? > > I'm not feeling strongly either way... I'm wanting to use Babel for some > stuff at work, but haven't got around to learn it well enough to set > things in motion. So my current investment is still small. :-) > >> Specifically two header arguments >> which would both take the name of a named code block, these header >> arguments would be as follows >> >> - append :: will append the current code block's body to the named code >> block during tangling >> >> - overwrite :: will overwrite the named code block's body with the body >>of the current code block >> >> for simple concatenate the value of the append header argument could be >> easily set on the file or subtree level. > > This looks good, but it sort of turns the tables (I hope I get this > right, it was a longer day than I hoped). With noweb references, you > need to resolve the name of the source block at the target. With the > header arguments you propose you need to resolve the target block name > at the source. Both ways are useful, but it seems that it would be > difficult to track all targets from a single source even if supposedly > one could have a list of targets (that would only be useful for > "overwrite" unless you are really lucky). This asymmetry is > inescapeable when you try to use source blocks like "include files". > Another problem arises when you need to (again) reorder source blocks at > the target. You can't do that from the source, since not only would the > source then need to know all targets, it would also need to know all > other sources and their ordering in all targets. > >> I feel that for many reasons (most of which have been discussed in >> relation to a header argument solution earlier in this thread) such a >> solution would be simpler and more consistent with the rest of Babel >> than the current name-based solution. > > I'd think if there was an API to get at the contents of the header > arguments (perhaps using or modeled after the properties API), both the > forward and backward resolution could use the same internal mechanism > and noweb references could be changed (or extended) to (optionally) use > a match with some header argument (e.g. :id or :srcname). So let's say > that'd then allow: > > <<:id "bla">> > > to build a list of all source blocks with that header argument and > tangle them in list order. And since noweb references now really expect > a list, you can just as well feed them a custom function that cobbles > that list from multiple header arguments or properties. > > Thinking along those lines I'm beginning to wonder if not the whole > header should be put into a drawer (I've often wished this was the case > for #+TBLFM). > > > > Regards, > Achim. -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: > Could you try the attached example file? I first evaluated the > following elisp code to set the combination variable's value to append. Your example works if there are no noweb references. See the modified one where I have noweb references. Note that when expanding the reference, it inserts only the first block it finds. #+begin_src emacs-lisp :tangle yes :noweb yes <> Random text <> #+end_src * continued code blocks :PROPERTIES: :tangle: yes :comments: yes :END: #+srcname: foo #+begin_src emacs-lisp (message "foo:%S" 1) #+end_src #+begin_src emacs-lisp (message "un-named") #+end_src #+srcname: bar #+begin_src emacs-lisp (message "bar:%S" 1) #+end_src #+srcname: foo #+begin_src emacs-lisp (message "foo:%S" 2) #+end_src #+srcname: bar #+begin_src emacs-lisp (message "bar:%S" 2) #+end_src #+begin_src emacs-lisp :tangle no :results silent (with-temp-buffer (insert-file-contents "scraps.el") (eval-buffer)) #+end_src
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: > Thanks for sharing your thoughts. How would you feel about moving away > from special source block names and moving towards implementing this > behavior with a header argument? I'm not feeling strongly either way... I'm wanting to use Babel for some stuff at work, but haven't got around to learn it well enough to set things in motion. So my current investment is still small. :-) > Specifically two header arguments > which would both take the name of a named code block, these header > arguments would be as follows > > - append :: will append the current code block's body to the named code > block during tangling > > - overwrite :: will overwrite the named code block's body with the body >of the current code block > > for simple concatenate the value of the append header argument could be > easily set on the file or subtree level. This looks good, but it sort of turns the tables (I hope I get this right, it was a longer day than I hoped). With noweb references, you need to resolve the name of the source block at the target. With the header arguments you propose you need to resolve the target block name at the source. Both ways are useful, but it seems that it would be difficult to track all targets from a single source even if supposedly one could have a list of targets (that would only be useful for "overwrite" unless you are really lucky). This asymmetry is inescapeable when you try to use source blocks like "include files". Another problem arises when you need to (again) reorder source blocks at the target. You can't do that from the source, since not only would the source then need to know all targets, it would also need to know all other sources and their ordering in all targets. > I feel that for many reasons (most of which have been discussed in > relation to a header argument solution earlier in this thread) such a > solution would be simpler and more consistent with the rest of Babel > than the current name-based solution. I'd think if there was an API to get at the contents of the header arguments (perhaps using or modeled after the properties API), both the forward and backward resolution could use the same internal mechanism and noweb references could be changed (or extended) to (optionally) use a match with some header argument (e.g. :id or :srcname). So let's say that'd then allow: <<:id "bla">> to build a list of all source blocks with that header argument and tangle them in list order. And since noweb references now really expect a list, you can just as well feed them a custom function that cobbles that list from multiple header arguments or properties. Thinking along those lines I'm beginning to wonder if not the whole header should be put into a drawer (I've often wished this was the case for #+TBLFM). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf microQ V2.22R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] Literate Programming - Continue a Source Block?
Achim Gratz writes: > Eric Schulte writes: >> I think "append" is just as clear as concatenate, > > Fair enough, in my mind "append" needs an existing entity, but I guess > it could be an empty one. > >> and collate implies shuffling which is not happening. > > Well, I was getting ahead of myself... I hope you don't mind me harping > on about this. > >> I'm wary of adding too much duplicate functionality. It is already >> possible to organize the tangling of many named code blocks using noweb >> reference expansion (a feature which I've used myself on real projects >> in the past). This existing method allows for unique block names and >> for arbitrary tangling order. > > The examples presented in this thread pointed out one difficulty with > how noweb references currently work, namely that you need to know in > advance all block names for finally tangling them and that when adding > or removing things you need to remember to keep these references current > in potentially many places. > >> Simplicity is the only reason that the new name-based appending behavior >> was implemented, simplicity which (in my opinion) is lost when the user >> defines a naming and sorting schema. > > My general objection is that since you now require that the names be all > the same, there is no way to distinguish between those blocks anymore. > This will come back to bite you when you then later need to make a > distinction since you'll then have to rework everything to unique names > again. I consider that sort of simplicity a trap for the unwary. > > Splitting between a basename and an index extension would instead allow > for appending with the basename and still getting at individual blocks > using the full name (with the index part). I've had one specific > use-case in mind where this would be needed. You'd normally just > concatenate all source blocks (let's assume for the moment that the > index part is separated from the basename by "::") > > <> > > Let's say you have three source blocks and need to point-patch the > second one for this run: > > <> > <> > <> > > If the three source blocks are all just named "source", there's no way > to do this short of changing their names and all the references that use > them. > > The index part doesn't have to be numeric, it just needs to be > orderable. Anyone not using index extensions gets an implicit one by > order of appearance, so that wouldn't require any code duplication and > you can in this case still say > > <> > > Being able to provide custom naming and sorting schemes could be > deferred to hook functions the user has to provide, as they will indeed > be rarely needed. So an implementation that keeps current behaviour > would provide an implicit indexer (by order of appearance), an empty > sorter and three selectors (all=append, first last). Anyone who needs > more than that has to customize those functions, but I suppose the more > common application scenarios will sneak into the baseline sooner or > later. The move from implicit to explicit indexing should be supported > by providing another hook in the implicit indexer. > > > Regards, > Achim. Hi Achim, Thanks for sharing your thoughts. How would you feel about moving away from special source block names and moving towards implementing this behavior with a header argument? Specifically two header arguments which would both take the name of a named code block, these header arguments would be as follows - append :: will append the current code block's body to the named code block during tangling - overwrite :: will overwrite the named code block's body with the body of the current code block for simple concatenate the value of the append header argument could be easily set on the file or subtree level. I feel that for many reasons (most of which have been discussed in relation to a header argument solution earlier in this thread) such a solution would be simpler and more consistent with the rest of Babel than the current name-based solution. Thoughts? -- Eric -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
Hi Neeum, Neeum Zawan writes: > Eric Schulte writes: > >> Hi Neeum, >> >> Thanks for your feedback. Your point is well taken about the >> flexibility of header arguments, and the ability of a header argument >> based solution to overwrite blocks. >> >> I would mention that variables such as the newly introduced >> `org-babel-tangle-named-block-combination' may be easily set on a >> per-file bases using file local variables---basically adding a line like >> the following to the top of your Org-mode file. > > Somehow I couldn't get your new function to work. The variable is set to > append, and I removed all org-modes from my system except the latest > from git. I even ensured that this code gets executed: > > (append (mapconcat #'identity > named "\n")) > > However, the output I get is the same as before - it just uses the first > source block. > Could you try the attached example file? I first evaluated the following elisp code to set the combination variable's value to append. #+begin_src emacs-lisp (setq org-babel-tangle-named-block-combination 'append) #+end_src I then call org-babel-tangle in the attached Org-mode file to generate the attached elisp file. * continued code blocks :PROPERTIES: :tangle: yes :comments: yes :END: #+source: foo #+begin_src emacs-lisp (message "foo:%S" 1) #+end_src #+begin_src emacs-lisp (message "un-named") #+end_src #+source: bar #+begin_src emacs-lisp (message "bar:%S" 1) #+end_src #+source: foo #+begin_src emacs-lisp (message "foo:%S" 2) #+end_src #+source: bar #+begin_src emacs-lisp (message "bar:%S" 2) #+end_src #+begin_src emacs-lisp :tangle no :results silent (with-temp-buffer (insert-file-contents "scraps.el") (eval-buffer)) #+end_src combined-tangle-example.el Description: application/emacs-lisp > > Incidentally, why do we need "\n" as a separator? What if I wanted to > do (in Python) > > def func(a, b, <>): > stuff > > In other words, I want to add more arguments later on. Wouldn't a \n > mess things up here? > Ah, this is a good point. I've just changed the code to avoid inserting superfluous newlines. Thanks! Cheers -- Eric > > > Thanks. > > -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: > I think "append" is just as clear as concatenate, Fair enough, in my mind "append" needs an existing entity, but I guess it could be an empty one. > and collate implies shuffling which is not happening. Well, I was getting ahead of myself... I hope you don't mind me harping on about this. > I'm wary of adding too much duplicate functionality. It is already > possible to organize the tangling of many named code blocks using noweb > reference expansion (a feature which I've used myself on real projects > in the past). This existing method allows for unique block names and > for arbitrary tangling order. The examples presented in this thread pointed out one difficulty with how noweb references currently work, namely that you need to know in advance all block names for finally tangling them and that when adding or removing things you need to remember to keep these references current in potentially many places. > Simplicity is the only reason that the new name-based appending behavior > was implemented, simplicity which (in my opinion) is lost when the user > defines a naming and sorting schema. My general objection is that since you now require that the names be all the same, there is no way to distinguish between those blocks anymore. This will come back to bite you when you then later need to make a distinction since you'll then have to rework everything to unique names again. I consider that sort of simplicity a trap for the unwary. Splitting between a basename and an index extension would instead allow for appending with the basename and still getting at individual blocks using the full name (with the index part). I've had one specific use-case in mind where this would be needed. You'd normally just concatenate all source blocks (let's assume for the moment that the index part is separated from the basename by "::") <> Let's say you have three source blocks and need to point-patch the second one for this run: <> <> <> If the three source blocks are all just named "source", there's no way to do this short of changing their names and all the references that use them. The index part doesn't have to be numeric, it just needs to be orderable. Anyone not using index extensions gets an implicit one by order of appearance, so that wouldn't require any code duplication and you can in this case still say <> Being able to provide custom naming and sorting schemes could be deferred to hook functions the user has to provide, as they will indeed be rarely needed. So an implementation that keeps current behaviour would provide an implicit indexer (by order of appearance), an empty sorter and three selectors (all=append, first last). Anyone who needs more than that has to customize those functions, but I suppose the more common application scenarios will sneak into the baseline sooner or later. The move from implicit to explicit indexing should be supported by providing another hook in the implicit indexer. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: > Hi Neeum, > > Thanks for your feedback. Your point is well taken about the > flexibility of header arguments, and the ability of a header argument > based solution to overwrite blocks. > > I would mention that variables such as the newly introduced > `org-babel-tangle-named-block-combination' may be easily set on a > per-file bases using file local variables---basically adding a line like > the following to the top of your Org-mode file. Somehow I couldn't get your new function to work. The variable is set to append, and I removed all org-modes from my system except the latest from git. I even ensured that this code gets executed: (append (mapconcat #'identity named "\n")) However, the output I get is the same as before - it just uses the first source block. Incidentally, why do we need "\n" as a separator? What if I wanted to do (in Python) def func(a, b, <>): stuff In other words, I want to add more arguments later on. Wouldn't a \n mess things up here? Thanks.
Re: [O] Literate Programming - Continue a Source Block?
Hi Neeum, Thanks for your feedback. Your point is well taken about the flexibility of header arguments, and the ability of a header argument based solution to overwrite blocks. I would mention that variables such as the newly introduced `org-babel-tangle-named-block-combination' may be easily set on a per-file bases using file local variables---basically adding a line like the following to the top of your Org-mode file. # -*- org-babel-tangle-named-block-combination:append -*- While there is no way to specify this on the block level, I would tend to think that specifications such as ":tangle no" or simply renaming the block would suffice in most cases. This still does not address the subtree level. I think that a general solution for setting variable values at the subtree level might be widely useful and could be something worth pursuing Org-wide. Moving forward from here, my proposal would be that we all try out the current support using noweb references and the new named-block behavior and discover experientially if and what new features would be desirable. On a related note I hereby declare that the newly implemented named-block behavior should be considered experimental, and support for it may be removed if a better solution presents itself. Cheers -- Eric Neeum Zawan writes: > Eric Schulte writes: I like the concision of the "=original-name" syntax used by noweb, but I would lean towards the use of a ":noweb-append" type header argument as suggested above because currently the names of blocks in Babel carry no semantic content and I'd prefer to leave it this way. >>> >>> I suppose it may also break compatibility in case someone out there >>> uses >>> the =symbol. >>> >>> Had it been thought of earlier, I would have preferred the default >>> behavior being append if you have multiple blocks of the same name, >>> and an explicit option *not* to append but to overwrite, but your >>> idea makes the most sense with respect to preserving backward >>> compatibility. >>> >>> In addition to append, there probably should be another option for >>> overwriting instead of appending (neither is possible right now). >>> >> >> I've just pushed up a patch which implements optional block >> combination during tangling. Specifically a new customization >> variable named `org-babel-tangle-named-block-combination' is >> introduced which can take the following values > > This does solve the problems I have, but I think the noweb-append header > option discussed earlier is much more flexible. The potential problems > with a custom variable is that it can't be set at a per-buffer or > per-document level. I was thinking along the lines of: > > :noweb-append option > > where option is one of: > > - nil (the default) > > - append (appends to the block of the same name as it is up to that > point in the document - acts as nil if this is the first block of that > name). > > - overwrite (overwrites the block as it is up to that point - acts as > nil if this is the first block). > > I think these three provide all the abilities of what you proposed, but > allows for much more. Some additional benefits: > > 1. Can be set at a per-buffer level or a per-block level. > > 2. Can selectively append/overwrite. One scenario where this would be >useful is where you may have had some source blocks that had been >appended, but then later on as the project evolves, you decide to >rewrite much of that code. You can then just do an overwrite >(i.e. erases all that you had up to that point), and then again allow >for the new code to be evolved with possible future appends (so >multiple appends/overwrites in one document). You may have reason to >keep the old code in the document for some reason or other. If that >didn't make sense I can explain in more detail. > >> Hopefully this gets at the behavior you're after. I'd be interested >> to hear any thought you have on this new functionality. > > I don't want to make it sound like I'm complaining above. What you've > proposed probably takes care of my current needs (and I imagine is a > bit easier to code than what I've proposed) - but I just think having > a new header for the source block would make it much more flexible. > > I haven't yet tried the new patch - I'll have to figure out how to do a > custom babel install (at the moment I get it via orgmode which is > installed system-wide). Is it possible for me to just install babel in > my custom emacs directory and not have it impact other aspects of > org-mode? > > Thanks for the quick commit! -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
Achim Gratz writes: > Eric Schulte writes: > >> append the bodies of all blocks of the same name are appended >>during tangling > > several blocks with the same name seem a bit dubious, would it not be > cleaner to have an index part to the block name and a range expression > for the concatenation during tangling? I might want to tangle them in > different order than their appearance in the source, for instance. For my purposes, an index would be too much to maintain, and while I wouldn't mind it as an option, I wouldn't want to *have* to use them if all I want is to append and/or overwrite. The solution I proposed in my response to Eric may meet you halfway there in some ways. When I initially wrote about this problem, my goal was simply to enable what could easily be done with noweb (and of course, I actually had a need for it). I actually have very little experience with LP, so I don't know in practice what will work and what won't. The initial benefit that I can see with indexing is that one may want to add some lines in the middle of a block. However, this can only really work if you just happen to end/begin your source blocks at that point. If you didn't know in advance that you'd add some lines in the middle, then you likely didn't end/begin your blocks at that point, and you'll have to go and artificially begin/end your blocks there. This may seem confusing to a reader, so it makes sense to put an explicit noweb style reference. If, on the other hand, you knew it in advance, you probably should put the noweb reference any way. My comments are from the perspective of my use case for LP, and I can see people may use LP for other reasons, so if you have a better scenario, let us know.
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: >>> >>> I like the concision of the "=original-name" syntax used by noweb, but I >>> would lean towards the use of a ":noweb-append" type header argument as >>> suggested above because currently the names of blocks in Babel carry no >>> semantic content and I'd prefer to leave it this way. >> >> I suppose it may also break compatibility in case someone out there uses >> the =symbol. >> >> Had it been thought of earlier, I would have preferred the default >> behavior being append if you have multiple blocks of the same name, and >> an explicit option *not* to append but to overwrite, but your idea makes >> the most sense with respect to preserving backward compatibility. >> >> In addition to append, there probably should be another option for >> overwriting instead of appending (neither is possible right now). >> > > I've just pushed up a patch which implements optional block combination > during tangling. Specifically a new customization variable named > `org-babel-tangle-named-block-combination' is introduced which can take > the following values This does solve the problems I have, but I think the noweb-append header option discussed earlier is much more flexible. The potential problems with a custom variable is that it can't be set at a per-buffer or per-document level. I was thinking along the lines of: :noweb-append option where option is one of: - nil (the default) - append (appends to the block of the same name as it is up to that point in the document - acts as nil if this is the first block of that name). - overwrite (overwrites the block as it is up to that point - acts as nil if this is the first block). I think these three provide all the abilities of what you proposed, but allows for much more. Some additional benefits: 1. Can be set at a per-buffer level or a per-block level. 2. Can selectively append/overwrite. One scenario where this would be useful is where you may have had some source blocks that had been appended, but then later on as the project evolves, you decide to rewrite much of that code. You can then just do an overwrite (i.e. erases all that you had up to that point), and then again allow for the new code to be evolved with possible future appends (so multiple appends/overwrites in one document). You may have reason to keep the old code in the document for some reason or other. If that didn't make sense I can explain in more detail. > Hopefully this gets at the behavior you're after. I'd be interested to > hear any thought you have on this new functionality. I don't want to make it sound like I'm complaining above. What you've proposed probably takes care of my current needs (and I imagine is a bit easier to code than what I've proposed) - but I just think having a new header for the source block would make it much more flexible. I haven't yet tried the new patch - I'll have to figure out how to do a custom babel install (at the moment I get it via orgmode which is installed system-wide). Is it possible for me to just install babel in my custom emacs directory and not have it impact other aspects of org-mode? Thanks for the quick commit! -- My neighbor has a circular driveway. He never leaves home.
Re: [O] Literate Programming - Continue a Source Block?
Achim Gratz writes: > Eric Schulte writes: > >> append the bodies of all blocks of the same name are appended >>during tangling > > Shouldn't this be called "concat(enate)" or "collate"? I think "append" is just as clear as concatenate, and collate implies shuffling which is not happening. > Or, since several blocks with the same name seem a bit dubious, would > it not be cleaner to have an index part to the block name and a range > expression for the concatenation during tangling? I might want to > tangle them in different order than their appearance in the source, > for instance. > I'm wary of adding too much duplicate functionality. It is already possible to organize the tangling of many named code blocks using noweb reference expansion (a feature which I've used myself on real projects in the past). This existing method allows for unique block names and for arbitrary tangling order. Simplicity is the only reason that the new name-based appending behavior was implemented, simplicity which (in my opinion) is lost when the user defines a naming and sorting schema. Cheers -- Eric > > > Achim. -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: > append the bodies of all blocks of the same name are appended >during tangling Shouldn't this be called "concat(enate)" or "collate"? Or, since several blocks with the same name seem a bit dubious, would it not be cleaner to have an index part to the block name and a range expression for the concatenation during tangling? I might want to tangle them in different order than their appearance in the source, for instance. Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] Literate Programming - Continue a Source Block?
>> >> I like the concision of the "=original-name" syntax used by noweb, but I >> would lean towards the use of a ":noweb-append" type header argument as >> suggested above because currently the names of blocks in Babel carry no >> semantic content and I'd prefer to leave it this way. > > I suppose it may also break compatibility in case someone out there uses > the =symbol. > > Had it been thought of earlier, I would have preferred the default > behavior being append if you have multiple blocks of the same name, and > an explicit option *not* to append but to overwrite, but your idea makes > the most sense with respect to preserving backward compatibility. > > In addition to append, there probably should be another option for > overwriting instead of appending (neither is possible right now). > I've just pushed up a patch which implements optional block combination during tangling. Specifically a new customization variable named `org-babel-tangle-named-block-combination' is introduced which can take the following values nilthe default, no behavior is changed append the bodies of all blocks of the same name are appended during tangling first only the body of the first block of any given name is kept during tangling last only the body of the last block of any given name is kept during tangling > > Also, just on the side, I'm not sure it's documented anywhere what > happens if you have multiple source code blocks of the same name. At the > moment, it seems only the first is used (I would have expected the > last). > Yes, currently block names are intended to be unique, and some of the Babel functionality (e.g., named block evaluation) make this assumption. The behavior of multiple blocks with the same name is undefined behavior. I've expanded the relevant documentation. > >> Thanks for the motivating example and the thorough explanation of >> behavior. >> >> I'll certainly put this on my long-term development queue, however, that >> does not guarantee an implementation in the near future. If anyone is >> interested in this functionality and is up for writing some elisp I am >> happy to offer advice and code pointers immediately. > > Wish I knew elisp. Anyway, hopefully someone will get it done one day. > Hopefully this gets at the behavior you're after. I'd be interested to hear any thought you have on this new functionality. Cheers -- Eric -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
On Wed, Jun 8, 2011 at 5:20 PM, Neeum Zawan wrote: > n this case, yes. In a real programming project, it could be a number > of them. For example, I may have a code block dedicated to > imports/includes which I want to be on the top of the file - and I may > have to append to that when adding a new feature. Ooh! Ooh! Me too! I was just wanting this the other day. I wanted to be able to do: #+srcname: requires #+begin_src ruby require 'library_a' #+end_src ... #+srcname: requires #+begin_src ruby require 'library_b' #+end_src And then tie it all together with: #+begin_src ruby :tangle foo.rb <> #+end But no such luck. <> just picked up the most recent block with that srcname, IIRC. -- Avdi Grimm http://avdi.org
Re: [O] Literate Programming - Continue a Source Block?
Eric Schulte writes: >>> Second solution: create one sole block that will be tangled, and which >>> contains your other blocks (using the <> syntax), in the order you >>> want. >> >> I had thought of this, but I find it somewhat lacking. Consider my >> example above. I could have created a <> in my >> <> block. However: >> >> 1. That requires me to know I'm going to need it later when I write >>visual-config. >> > > nit picking here, but while this does require a small edit to > <>, there is no need for prior knowledge of the need for > <>. > >> >> 2. If I didn't know I'd need it, I'd have to continually modify various >> parts of the org document every time I add a new feature to my code. I >> find this suboptimal and error prone. >> > > Technically only one edit per new block introduced, which does not seem > overly onerous. In this case, yes. In a real programming project, it could be a number of them. For example, I may have a code block dedicated to imports/includes which I want to be on the top of the file - and I may have to append to that when adding a new feature. And then the actual code for the new feature may require edits to other parts of one's program. Ideally, everything should be decoupled, but reality rarely follows the ideal ;-) >> Now the original noweb allows what I'm asking for. If you begin a >> source block with a name of an existing block but append an "=" >> symbol, it knows to append to that source block. >> >> It would be great if org-mode could add that capability. > > I agree, and the functionality you describe shouldn't be overly > difficult to implement. > > I like the concision of the "=original-name" syntax used by noweb, but I > would lean towards the use of a ":noweb-append" type header argument as > suggested above because currently the names of blocks in Babel carry no > semantic content and I'd prefer to leave it this way. I suppose it may also break compatibility in case someone out there uses the =symbol. Had it been thought of earlier, I would have preferred the default behavior being append if you have multiple blocks of the same name, and an explicit option *not* to append but to overwrite, but your idea makes the most sense with respect to preserving backward compatibility. In addition to append, there probably should be another option for overwriting instead of appending (neither is possible right now). Also, just on the side, I'm not sure it's documented anywhere what happens if you have multiple source code blocks of the same name. At the moment, it seems only the first is used (I would have expected the last). > Thanks for the motivating example and the thorough explanation of > behavior. > > I'll certainly put this on my long-term development queue, however, that > does not guarantee an implementation in the near future. If anyone is > interested in this functionality and is up for writing some elisp I am > happy to offer advice and code pointers immediately. Wish I knew elisp. Anyway, hopefully someone will get it done one day. Thanks.
Re: [O] Literate Programming - Continue a Source Block?
> > The above is somewhat artificial, but in a proper programming project > something like this will occur frequently: A new feature will be added > at some later point and I'll want to update various blocks of code. > Currently the best method is that suggested previously/below of using named references in the target code block, as suggested below. While it shouldn't be overly difficult to add the behavior your described with something like a ":noweb-append" header argument, e.g., #+begin_src emacs-lisp :noweb-append visual-config ... #+end_src This behavior is not currently implemented. > >> Second solution: create one sole block that will be tangled, and which >> contains your other blocks (using the <> syntax), in the order you want. > > I had thought of this, but I find it somewhat lacking. Consider my > example above. I could have created a <> in my > <> block. However: > > 1. That requires me to know I'm going to need it later when I write >visual-config. > nit picking here, but while this does require a small edit to <>, there is no need for prior knowledge of the need for <>. > > 2. If I didn't know I'd need it, I'd have to continually modify various > parts of the org document every time I add a new feature to my code. I > find this suboptimal and error prone. > Technically only one edit per new block introduced, which does not seem overly onerous. > > 3. For me, one of the main motivations to do LP is to document the > evolution of the project, such that someone else can read the document > and understand. Part of this is that I want all aspects of a single > feature to appear under a single heading. When the reader reads the > document, he shouldn't have to see too many aspects of the code early on > that will be explained much later in the document. > I agree, this is a motivating example. > > Now the original noweb allows what I'm asking for. If you begin a > source block with a name of an existing block but append an "=" > symbol, it knows to append to that source block. > > It would be great if org-mode could add that capability. I agree, and the functionality you describe shouldn't be overly difficult to implement. I like the concision of the "=original-name" syntax used by noweb, but I would lean towards the use of a ":noweb-append" type header argument as suggested above because currently the names of blocks in Babel carry no semantic content and I'd prefer to leave it this way. > Another approach is that if multiple source blocks with the same name > are found, the default behavior is to concatenate all those source > blocks together (and then add a header option for overwrite if the > user wanted to overwrite instead of append). > > Thoughts? > Thanks for the motivating example and the thorough explanation of behavior. I'll certainly put this on my long-term development queue, however, that does not guarantee an implementation in the near future. If anyone is interested in this functionality and is up for writing some elisp I am happy to offer advice and code pointers immediately. Best -- Eric -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Literate Programming - Continue a Source Block?
"Sebastien Vauban" writes: > Hi Neeum, > > Neeum Zawan wrote: >> With noweb, one can continue a source block that one started >> earlier. Can this not be done with Babel? >> >> If not, I'm struggling a little with how to do LP using Babel... > > Of course, this can be done here as well: simply reuse the same "tangle" > target (file), and blocks are concatenated in the order they appear. But this will only allow me to append to the end of the file (up to that point), correct? I want to append to a source block that may be in the middle. Consider the following example: (Begin-example) #+BABEL: :noweb yes * Outline The structure of the configuration file will be: #+srcname: outline #+begin_src emacs-lisp :tangle .emacs :exports code ;; Store configurations that impact appearance <> ;; Languages configurations <> ;; GNUS configurations <> #+end_src * General Appearance This section controls the general appearance (color of cursor, fonts, etc). #+srcname: visual-config #+begin_src emacs-lisp :exports code ;; Config here. #+end_src * GNUS configuration #+srcname: gnus-config #+begin_src emacs-lisp :exports code ;; Config here. #+end_src * Languages #+srcname: lang-config #+begin_src emacs-lisp :exports code <> <> #+end_src ** Python Now here's where the problem begins. I want to add some configurations that impact some Python settings. I can dump those in python-config. However, I also want to add some visual settings for python-mode (change font colors, etc). For whatever reason, I think that part should go into visual-config. How do I now append the relevant portion to visual-config while I'm in _this_ portion of the org document? (End-example) The above is somewhat artificial, but in a proper programming project something like this will occur frequently: A new feature will be added at some later point and I'll want to update various blocks of code. > Second solution: create one sole block that will be tangled, and which > contains your other blocks (using the <> syntax), in the order you want. I had thought of this, but I find it somewhat lacking. Consider my example above. I could have created a <> in my <> block. However: 1. That requires me to know I'm going to need it later when I write visual-config. 2. If I didn't know I'd need it, I'd have to continually modify various parts of the org document every time I add a new feature to my code. I find this suboptimal and error prone. 3. For me, one of the main motivations to do LP is to document the evolution of the project, such that someone else can read the document and understand. Part of this is that I want all aspects of a single feature to appear under a single heading. When the reader reads the document, he shouldn't have to see too many aspects of the code early on that will be explained much later in the document. Now the original noweb allows what I'm asking for. If you begin a source block with a name of an existing block but append an "=" symbol, it knows to append to that source block. It would be great if org-mode could add that capability. Another approach is that if multiple source blocks with the same name are found, the default behavior is to concatenate all those source blocks together (and then add a header option for overwrite if the user wanted to overwrite instead of append). Thoughts?
Re: [O] Literate Programming - Continue a Source Block?
Hi Neeum, Neeum Zawan wrote: > With noweb, one can continue a source block that one started > earlier. Can this not be done with Babel? > > If not, I'm struggling a little with how to do LP using Babel... Of course, this can be done here as well: simply reuse the same "tangle" target (file), and blocks are concatenated in the order they appear. Second solution: create one sole block that will be tangled, and which contains your other blocks (using the <> syntax), in the order you want. Best regards, Seb -- Sebastien Vauban
