Re: [O] Not overwriting unchanged source code files when tangling
On 11/19/2011 01:32 PM, Holger Hoefling wrote: Hi everyone, I wanted to thank everyone for their helpful suggestions and wanted to share the best solutions I heard of and found. One solution is to include a rule in the makefile for every sourcecode file that that copies it and only updates the copy if something has changed (see Nick's email below). Or tangle to one directory, and then copy to another; then there's one rule for all of the transitions. work/%: tangle/% @cmp --silent $ $@ || ( echo Updating $@ ; cp $ $@ ) I liked Nick's cmp invocation; I started out with diff. :) - Allen S. Rout
Re: [O] Not overwriting unchanged source code files when tangling
Allen S. Rout a...@ufl.edu wrote: On 11/19/2011 01:32 PM, Holger Hoefling wrote: Hi everyone, I wanted to thank everyone for their helpful suggestions and wanted to share the best solutions I heard of and found. One solution is to include a rule in the makefile for every sourcecode file that that copies it and only updates the copy if something has changed (see Nick's email below). Or tangle to one directory, and then copy to another; then there's one rule for all of the transitions. work/%: tangle/% @cmp --silent $ $@ || ( echo Updating $@ ; cp $ $@ ) I liked Nick's cmp invocation; I started out with diff. :) I first saw it in Kernighan and Pike's Unix Programming Environment (1984!). I'm sure the trick went back even further. Nick
Re: [O] Not overwriting unchanged source code files when tangling
Such tools are for example scons is a make alternative that rethinks make's older timestamp = rebuild From www.scons.org/architecture/index.html *(S)Cons decides if a file was out-of-date by using MD5 checksums of the contents of files, not timestamps. * SCons also comes to mind because you are using make for R not (traditional) programming *Most of the build tools just mentioned were written by programmers and for programmers. The fact that most programmer-friendly utilities do a poor job of fulfilling the needs of non-programmers prompted Greg Wilson to organize the Software Carpentry competition in January 2000. Software Carpentry was an open design contest with the express goal of producing a set of next-generation utilities, including a build tool, that would be accessible not only to programmers but also to computer **users** such as physical scientists. * [Disclaimer: Ive not tried scons myself]
Re: [O] Not overwriting unchanged source code files when tangling
Sorry for the earlier mail: Send got pressed on a half-cooked mail :-) Heres a cleanup. -- Build-tools like scons rethink make's older timestamp = rebuild model. You may want to look at one such. From http://www.scons.org/architecture/index.html (S)Cons decides if a file was out-of-date by using MD5 checksums of the contents of files, not timestamps. SCons also comes to mind because you are using make for R not (traditional) programming [Again from the above link] Most of the build tools just mentioned were written by programmers and for programmers. (SCons arose from a) design contest with the express goal of producing a set of next-generation utilities, including a build tool, that would be accessible not only to programmers but also to computer users such as physical scientists. [Disclaimer: Ive not tried Scons myself]
Re: [O] Not overwriting unchanged source code files when tangling
Holger Hoefling hhoef...@gmail.com writes: Hi Carsten, thanks for the suggestion, but as I agree with Brian. If there is more than one source file in the org-file, then the whole project would still be recompiled, not just the updated file.To be more exact, I actually don#39;t want to compile things, but run R scripts using make. There are caching tools in R that might handle this. packages weaver and cacheSweave come to mind. For example, tangling your *.org file to produce a trivial Sweave document consisting of a single code chunk with the 'cache=T' argument, then Sweaving it using the 'weaver' driver from the weaver package will cache all the computations. Subsequent changes to the code followed by a new weave will result in cached values being used when possible and fresh computations being done as needed. I believe this is done on an expression by expression basis so adding a bit of whitespace or a comment will not trigger recomputation. So you can overwrite the document and making trivial changes to expressions without having to recompute expressions that have not changed (and have no dependencies that are changed). HTH, Chuck So the waiting time if a computationally intensive step is repeated although it is not necessary can be substantial. I wonder how difficult the following change would be (no emacs lisp experience, also do not know the org source code):- would it be possible to write out the source files when tangling into a temporary directory, then compare to the actual target files and overwrite only if something has changed? Then the time stamps would stay fixed. Hopefully, this would not involve too much work: - creating temporary files and remembering the mapping to true files- tangling out as usual into temporary files (so probably little change there)- compare temporary file to true file (does emacs already have a diff utility that could be used?) - overwrite true file if any changes- delete temporary filesEspecially, with this method no dependencies would be necessary and it would not be necessary to keep track in the org files which source blocks have been changed since the last tangling Thank you for your suggestionHolger On Fri, Nov 18, 2011 at 6:02 PM, Carsten Dominik span dir=ltrmailto:carsten.domi...@gmail.com/span wrote: On 18.11.2011, at 14:17, Holger Hoefling wrote: Hi, I have a problem/request for org-mode and was looking for help. I am using org-mode to write source code files and tangle them out. I want to compile them using make. My problem now is that org-mode overwrites the old files every time I tangle them out, therefore also updating the time stamp - even if nothing has changed. Subsequently, when I run make, everything gets recompiled, not just the changed source code files as all time stamps have changed. Is there an option for org-mode to only overwrite source code files that get tangled out if they have truly changed? How about changing the make file so that the dependence is on the Org file, not on the source file? You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it? Something along the lines of (untested, and probably wrong in this way...) file.o: http://somefile.org emacs -batch --eval #39;(org-babel-tangle-file http://somefile.org;)#39; cc file.o - Carsten -- Charles C. BerryDept of Family/Preventive Medicine cbe...@tajo.ucsd.eduUC San Diego http://famprevmed.ucsd.edu/faculty/cberry/ La Jolla, San Diego 92093-0901
Re: [O] Not overwriting unchanged source code files when tangling
Hi everyone, I wanted to thank everyone for their helpful suggestions and wanted to share the best solutions I heard of and found. One solution is to include a rule in the makefile for every sourcecode file that that copies it and only updates the copy if something has changed (see Nick's email below). Another one is the use of a non-standard make program like makepp, that allows for using md5 checksums for files instead of timestamp in order to derive which files have to be rebuild. Thanks again! Holger On Sat, Nov 19, 2011 at 7:58 AM, Holger Hoefling hhoef...@gmail.com wrote: Hey Nick, thank you very much. That sounds like a very good solution to my problem that does not require changes to org-mode. Best Holger On Sat, Nov 19, 2011 at 5:00 AM, Nick Dokos nicholas.do...@hp.com wrote: Holger Hoefling hhoef...@gmail.com wrote: I think you misunderstood me there - I am actually not worried about how computationally intensive the tangling process is. This always works very quickly, so even if they have to be copied around and take a bit longer, I would not mind. Ah, ok - so you are talking about tanglecompile org --- bunch of files - output The tangling step produces a bunch of files that are (re)compiled (or in any case require some sort of lengthy processing) to produce some output file. IMO, the best way to deal with it is still make: let's say foo.org a.x b.x c.x -- foo.out where the first arrow is the tangle and the second arrow is some processor, call it X. The standard way to set up a makefile is schematically: --8---cut here---start-8--- foo.out: a.x b.x c.x X a.x b.c c.x -o foo.out a.x b.x c.x: foo.org tangle foo.org --8---cut here---end---8--- Rewrite the make file as follows: --8---cut here---start-8--- foo.out: a.y b.y c.y X a.y b.y c.y -o foo.out a.y: a.x cmp --silent a.x a.y || cp a.x a.y b.y: b.x cmp --silent b.x b.y || cp b.x b.y c.y: c.x cmp --silent c.x c.y || cp c.x c.y a.x b.x c.x: foo.org tangle foo.org --8---cut here---end---8--- So if the *contents* of (say) a.x have not changed by the tangling, it compares equal to a.y and the copy is skipped. That leaves a.y untouched. OTOH, if the contents of a.x change (or a.y does not exist in the first place), the comparison fails and we copy a.x to a.y. That updates a.y and forces further updates on anything that depends on it. Using some make fu (works for GNU make, but not necessarily for other makes), you can write it more compactly: --8---cut here---start-8--- foo.out: a.y b.y c.y X a.y b.y c.y -o foo.out %.y: %.x -cmp --silent $ $@ || cp $ $@ a.x b.x c.x: foo.org tangle foo.org --8---cut here---end---8--- HTH, Nick
Re: [O] Not overwriting unchanged source code files when tangling
On Fri, Nov 18, 2011 at 7:17 AM, Holger Hoefling hhoef...@gmail.com wrote: I have a problem/request for org-mode and was looking for help. I am using org-mode to write source code files and tangle them out. I want to compile them using make. My problem now is that org-mode overwrites the old files every time I tangle them out, therefore also updating the time stamp - even if nothing has changed. Subsequently, when I run make, everything gets recompiled, not just the changed source code files as all time stamps have changed. Is there an option for org-mode to only overwrite source code files that get tangled out if they have truly changed? I believe that to do this, you would need to have a dependency tree of the nodes contributing to the output (perhaps already exists), and recursively mark any node that refers to a node that changed as dirty. You would also have to store last update times on each node so that they could be compared to each output file, contributing to the determination of needing a regeneration or not. From a make standpoint, if you were to have each node in a file (I am not recommending this), make already has the smarts to handle this. It just becomes unwieldy to manage from an editing perspective. Perhaps a way to deal with this would be to tangle to a different directory, and then sync any changes into your compilation source directory. If you would update the compilation directory only when something differs from the tangle directory, then make could handle it from that point on. Brian
Re: [O] Not overwriting unchanged source code files when tangling
Hi Holger, Holger Hoefling wrote: I have a problem/request for org-mode and was looking for help. I am using org-mode to write source code files and tangle them out. I want to compile them using make. My problem now is that org-mode overwrites the old files every time I tangle them out, therefore also updating the time stamp - even if nothing has changed. Subsequently, when I run make, everything gets recompiled, not just the changed source code files as all time stamps have changed. Is there an option for org-mode to only overwrite source code files that get tangled out if they have truly changed? Thank you very much for your help! I have absolutely no idea on how hard it could be, but this makes me think at some cache mechanism, like what currently exists for skipping evaluation of unchanged code blocks. The key could be to use the cache feature for tangling as well? Best regards, Seb -- Sebastien Vauban
Re: [O] Not overwriting unchanged source code files when tangling
On Fri, 18 Nov 2011 08:23:18 -0600, Brian Wightman midlife...@wightmanfam.org wrote: Perhaps a way to deal with this would be to tangle to a different directory, and then sync any changes into your compilation source directory. If you would update the compilation directory only when something differs from the tangle directory, then make could handle it from that point on. The tangle mechanism could probably handle this autoatically. i.e. not saving a file if the contents are identical. Tom
Re: [O] Not overwriting unchanged source code files when tangling
On Fri, Nov 18, 2011 at 10:46 AM, Tom Prince tom.pri...@ualberta.net wrote: On Fri, 18 Nov 2011 08:23:18 -0600, Brian Wightman midlife...@wightmanfam.org wrote: Perhaps a way to deal with this would be to tangle to a different directory, and then sync any changes into your compilation source directory. If you would update the compilation directory only when something differs from the tangle directory, then make could handle it from that point on. The tangle mechanism could probably handle this autoatically. i.e. not saving a file if the contents are identical. If there is not a lot of extra memory / time overhead associated with this, I could see this being a valid approach. I would request that, if implemented, this be placed behind an on/off switch. The makefile could also handle this with something along these lines (correct the leading space - tab conversion as well as proper macro definitions): tangleflag: totangle.org $(TANGLECOMMAND) totangle.org $(TOUCH) tangleflag syncflag: tangleflag $(SYNCCOMMAND) sourcedir tangledir $(TOUCH) syncflag --Brian
Re: [O] Not overwriting unchanged source code files when tangling
On 18.11.2011, at 14:17, Holger Hoefling wrote: Hi, I have a problem/request for org-mode and was looking for help. I am using org-mode to write source code files and tangle them out. I want to compile them using make. My problem now is that org-mode overwrites the old files every time I tangle them out, therefore also updating the time stamp - even if nothing has changed. Subsequently, when I run make, everything gets recompiled, not just the changed source code files as all time stamps have changed. Is there an option for org-mode to only overwrite source code files that get tangled out if they have truly changed? How about changing the make file so that the dependence is on the Org file, not on the source file? You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it? Something along the lines of (untested, and probably wrong in this way...) file.o: somefile.org emacs -batch --eval '(org-babel-tangle-file somefile.org)' cc file.o - Carsten
Re: [O] Not overwriting unchanged source code files when tangling
On Fri, Nov 18, 2011 at 11:02 AM, Carsten Dominik carsten.domi...@gmail.com wrote: How about changing the make file so that the dependence is on the Org file, not on the source file? You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it? The original question was trying to avoid recompiling everything generated from a tangle if the content didn't actually change. Because retangling the source rewrites /all/ of the files, and resets the dates, even if nothing has changed, make will then rebuild everything that was tangled, not just the partial set of tangled files that actually changed. Brian
Re: [O] Not overwriting unchanged source code files when tangling
Hi Carsten, thanks for the suggestion, but as I agree with Brian. If there is more than one source file in the org-file, then the whole project would still be recompiled, not just the updated file. To be more exact, I actually don't want to compile things, but run R scripts using make. So the waiting time if a computationally intensive step is repeated although it is not necessary can be substantial. I wonder how difficult the following change would be (no emacs lisp experience, also do not know the org source code): - would it be possible to write out the source files when tangling into a temporary directory, then compare to the actual target files and overwrite only if something has changed? Then the time stamps would stay fixed. Hopefully, this would not involve too much work: - creating temporary files and remembering the mapping to true files - tangling out as usual into temporary files (so probably little change there) - compare temporary file to true file (does emacs already have a diff utility that could be used?) - overwrite true file if any changes - delete temporary files Especially, with this method no dependencies would be necessary and it would not be necessary to keep track in the org files which source blocks have been changed since the last tangling Thank you for your suggestion Holger On Fri, Nov 18, 2011 at 6:02 PM, Carsten Dominik carsten.domi...@gmail.comwrote: On 18.11.2011, at 14:17, Holger Hoefling wrote: Hi, I have a problem/request for org-mode and was looking for help. I am using org-mode to write source code files and tangle them out. I want to compile them using make. My problem now is that org-mode overwrites the old files every time I tangle them out, therefore also updating the time stamp - even if nothing has changed. Subsequently, when I run make, everything gets recompiled, not just the changed source code files as all time stamps have changed. Is there an option for org-mode to only overwrite source code files that get tangled out if they have truly changed? How about changing the make file so that the dependence is on the Org file, not on the source file? You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it? Something along the lines of (untested, and probably wrong in this way...) file.o: somefile.org emacs -batch --eval '(org-babel-tangle-file somefile.org)' cc file.o - Carsten
Re: [O] Not overwriting unchanged source code files when tangling
Brian Wightman br...@wightmanfam.org writes: On Fri, Nov 18, 2011 at 11:02 AM, Carsten Dominik carsten.domi...@gmail.com wrote: How about changing the make file so that the dependence is on the Org file, not on the source file? You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it? The original question was trying to avoid recompiling everything generated from a tangle if the content didn't actually change. Because retangling the source rewrites /all/ of the files, and resets the dates, even if nothing has changed, make will then rebuild everything that was tangled, not just the partial set of tangled files that actually changed. I think the best approach in this case would be to tangle each file out to a temporary buffer, and then just before exiting the tangle function the content of these temporary buffers could be checked against the files on disk, and only those buffers which differ from disk would be written. See ob-tangle.el around line 240 for the relevant code. Unfortunately this would not be trivial, as currently content is written to the target files incrementally block by block. If the code in the .org file is grouped by subtree it may be possible to place calls to org-narrow-to-subtree in the Makefile before tangling, so that only part of the file is tangled. Finally, it may be easiest simply to play make's game as it were and break up the Org-mode file into multiple files. These multiple files could still be combined during export using #+INCLUDE lines from a single master Org-mode file. Best -- Eric Brian -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Not overwriting unchanged source code files when tangling
Holger Hoefling hhoef...@gmail.com writes: I have a problem/request for org-mode and was looking for help. I am using org-mode to write source code files and tangle them out. I want to compile them using make. My problem now is that org-mode overwrites the old files every time I tangle them out, therefore also updating the time stamp - even if nothing has changed. Subsequently, when I run make, everything gets recompiled, not just the changed source code files as all time stamps have changed. Make really doesn't have the smarts to deal with builds that it doesn't fully control. In your case, the actual source aren't the tangled files, but really (parts of) an orgmode file. With a bit of clever organization it should be possible to employ git to keep track of the sources for make. In a nutshell, you'd check in the sources into a repository and only compile those that are different from the last commit. After the successful compile you autocommit the tangled files. Since you will probably don't need version control of the tangled files (although that might be useful for debugging of the tangling process itself), you could either amend each commit or just periodically chop the repository and do a garbage collect. Putting the smarts into orgmode to keep track of each tangle block looks complicated in the general case; while possible is a bit of a stretch IMHO. It would also produce false positives for a lot of cases that I'd consider useful (splitting or fusioning tangle blocks without changing the result). But the solution outlined above could be used if the content of each file to be tangled was always available in full in a buffer before writing to the tangled file. The buffer could just be dropped if the SHA1 of the file on disk and the buffer content is identical. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves
Re: [O] Not overwriting unchanged source code files when tangling
Eric Schulte schulte.e...@gmail.com writes: I think the best approach in this case would be to tangle each file out to a temporary buffer, and then just before exiting the tangle function the content of these temporary buffers could be checked against the files on disk, and only those buffers which differ from disk would be written. See ob-tangle.el around line 240 for the relevant code. Unfortunately this would not be trivial, as currently content is written to the target files incrementally block by block. It would be wise to follow an age-old tradition and not clobber an existing file with before it is known that the process will finish without error. A temporary file is easily trashed when something goes wrong and the previous result still available. If all went well, the old file can be deleted (or renamed to a backup file) and the temporary file can be moved to where the old one was if the two SHA1 differ. If the code in the .org file is grouped by subtree it may be possible to place calls to org-narrow-to-subtree in the Makefile before tangling, so that only part of the file is tangled. I don't think that would work very well, see my other post. It is the content of the resulting file that needs to be compared, not the steps in which that result is produced. Finally, it may be easiest simply to play make's game as it were and break up the Org-mode file into multiple files. These multiple files could still be combined during export using #+INCLUDE lines from a single master Org-mode file. Well, I've been wondering about this for some time: can one make sort of an indirect buffer from the contents of multiple other buffers in Emacs? That would make such #+INCLUDEs much more seamless. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptation for Waldorf rackAttack V1.04R1: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] Not overwriting unchanged source code files when tangling
Holger Hoefling hhoef...@gmail.com wrote: Hi Carsten, thanks for the suggestion, but as I agree with Brian. If there is more than one source file in the org-file, then the whole project would still be recompiled, not just the updated file. To be more exact, I actually don't want to compile things, but run R scripts using make. So the waiting time if a computationally intensive step is repeated although it is not necessary can be substantial. I wonder how difficult the following change would be (no emacs lisp experience, also do not know the org source code): - would it be possible to write out the source files when tangling - into a temporary directory, then compare to the actual target files - and overwrite only if something has changed? Then the time stamps - would stay fixed. Hopefully, this would not involve too much work: You've lost right there unless there is a method to select *which* source blocks to tangle. IOW, the problem is not the *comparison* of the temp and actual target files, it is the *production* of the temp files themselves: that's the computationally expensive step and this method does nothing to alleviate that. Unless I'm missing something. Nick - creating temporary files and remembering the mapping to true files - tangling out as usual into temporary files (so probably little - change there) - compare temporary file to true file (does emacs already have a diff - utility that could be used?) - overwrite true file if any changes - delete temporary files
Re: [O] Not overwriting unchanged source code files when tangling
Achim Gratz strom...@nexgo.de writes: Eric Schulte schulte.e...@gmail.com writes: I think the best approach in this case would be to tangle each file out to a temporary buffer, and then just before exiting the tangle function the content of these temporary buffers could be checked against the files on disk, and only those buffers which differ from disk would be written. See ob-tangle.el around line 240 for the relevant code. Unfortunately this would not be trivial, as currently content is written to the target files incrementally block by block. It would be wise to follow an age-old tradition and not clobber an existing file with before it is known that the process will finish without error. A temporary file is easily trashed when something goes wrong and the previous result still available. If all went well, the old file can be deleted (or renamed to a backup file) and the temporary file can be moved to where the old one was if the two SHA1 differ. Agreed, it would be preferable to build up the tangled contents in memory and not write to the file system until the tangling process is complete. Finally, it may be easiest simply to play make's game as it were and break up the Org-mode file into multiple files. These multiple files could still be combined during export using #+INCLUDE lines from a single master Org-mode file. Well, I've been wondering about this for some time: can one make sort of an indirect buffer from the contents of multiple other buffers in Emacs? That would make such #+INCLUDEs much more seamless. Hmm, this could be useful both inside and outside of Org-mode. Emacs does have indirect buffers [1], however they are exact copies of another buffer, I don't know if it would be possible to combine multiple indirect buffers into a single buffer. Thanks -- Eric Regards, Achim. Footnotes: [1] (info (elisp) Indirect Buffers) -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Not overwriting unchanged source code files when tangling
Hi Eric, sounds like the problem may after all not be that simple.Could the code blocks be written incrementally to the buffer (or a temporary file on disk) and only after everything has been tangled out all temporary buffers or files checked against the ones on disk? Unfortunately, I do not think that breaking up the org-file into smaller org-files is a solution to my problem. I am running longer analyses (20 script files or more) each doing various things and files in parts depending on each other. If I break up the org-file into so many small parts, a large part of the advantage of using org (i.e. complete analysis in a single file with extensive documentation) will be lost. True, I can still include the subfiles during export - but i hardly ever export code blocks at all (I instead hide the code from export and produce a beamer presentation or latex article using the output from my earlier code). So overall, I would really like to keep everything together in one file. Someone else earlier suggested just tangling out to a temporary directory myself and synching in the makefile. This suggestion is close to what I would like to do and I may use it. However, it is still not quite satisfactory as this brings other problems. Source files sometimes call each other - when they are in a different directory after tangling than they are during the make file execution (due to synching), then either execution from within org may break, or the execution from the make file. If I knew more about e-lisp, I would just hack up a solution myself. I think org-mode is a really awesome tool. Thanks for your help and suggestions Holger On Fri, Nov 18, 2011 at 8:42 PM, Eric Schulte schulte.e...@gmail.comwrote: Brian Wightman br...@wightmanfam.org writes: On Fri, Nov 18, 2011 at 11:02 AM, Carsten Dominik carsten.domi...@gmail.com wrote: How about changing the make file so that the dependence is on the Org file, not on the source file? You could then arrange for make to call emacs in batch-mode to tangle the source file and then compile it? The original question was trying to avoid recompiling everything generated from a tangle if the content didn't actually change. Because retangling the source rewrites /all/ of the files, and resets the dates, even if nothing has changed, make will then rebuild everything that was tangled, not just the partial set of tangled files that actually changed. I think the best approach in this case would be to tangle each file out to a temporary buffer, and then just before exiting the tangle function the content of these temporary buffers could be checked against the files on disk, and only those buffers which differ from disk would be written. See ob-tangle.el around line 240 for the relevant code. Unfortunately this would not be trivial, as currently content is written to the target files incrementally block by block. If the code in the .org file is grouped by subtree it may be possible to place calls to org-narrow-to-subtree in the Makefile before tangling, so that only part of the file is tangled. Finally, it may be easiest simply to play make's game as it were and break up the Org-mode file into multiple files. These multiple files could still be combined during export using #+INCLUDE lines from a single master Org-mode file. Best -- Eric Brian -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] Not overwriting unchanged source code files when tangling
Hi Nick, I think you misunderstood me there - I am actually not worried about how computationally intensive the tangling process is. This always works very quickly, so even if they have to be copied around and take a bit longer, I would not mind. Thanks Holger On Fri, Nov 18, 2011 at 8:32 PM, Nick Dokos nicholas.do...@hp.com wrote: Holger Hoefling hhoef...@gmail.com wrote: Hi Carsten, thanks for the suggestion, but as I agree with Brian. If there is more than one source file in the org-file, then the whole project would still be recompiled, not just the updated file. To be more exact, I actually don't want to compile things, but run R scripts using make. So the waiting time if a computationally intensive step is repeated although it is not necessary can be substantial. I wonder how difficult the following change would be (no emacs lisp experience, also do not know the org source code): - would it be possible to write out the source files when tangling - into a temporary directory, then compare to the actual target files - and overwrite only if something has changed? Then the time stamps - would stay fixed. Hopefully, this would not involve too much work: You've lost right there unless there is a method to select *which* source blocks to tangle. IOW, the problem is not the *comparison* of the temp and actual target files, it is the *production* of the temp files themselves: that's the computationally expensive step and this method does nothing to alleviate that. Unless I'm missing something. Nick - creating temporary files and remembering the mapping to true files - tangling out as usual into temporary files (so probably little - change there) - compare temporary file to true file (does emacs already have a diff - utility that could be used?) - overwrite true file if any changes - delete temporary files
Re: [O] Not overwriting unchanged source code files when tangling
Holger Hoefling hhoef...@gmail.com wrote: I think you misunderstood me there - I am actually not worried about how computationally intensive the tangling process is. This always works very quickly, so even if they have to be copied around and take a bit longer, I would not mind. Ah, ok - so you are talking about tanglecompile org --- bunch of files - output The tangling step produces a bunch of files that are (re)compiled (or in any case require some sort of lengthy processing) to produce some output file. IMO, the best way to deal with it is still make: let's say foo.org a.x b.x c.x -- foo.out where the first arrow is the tangle and the second arrow is some processor, call it X. The standard way to set up a makefile is schematically: --8---cut here---start-8--- foo.out: a.x b.x c.x X a.x b.c c.x -o foo.out a.x b.x c.x: foo.org tangle foo.org --8---cut here---end---8--- Rewrite the make file as follows: --8---cut here---start-8--- foo.out: a.y b.y c.y X a.y b.y c.y -o foo.out a.y: a.x cmp --silent a.x a.y || cp a.x a.y b.y: b.x cmp --silent b.x b.y || cp b.x b.y c.y: c.x cmp --silent c.x c.y || cp c.x c.y a.x b.x c.x: foo.org tangle foo.org --8---cut here---end---8--- So if the *contents* of (say) a.x have not changed by the tangling, it compares equal to a.y and the copy is skipped. That leaves a.y untouched. OTOH, if the contents of a.x change (or a.y does not exist in the first place), the comparison fails and we copy a.x to a.y. That updates a.y and forces further updates on anything that depends on it. Using some make fu (works for GNU make, but not necessarily for other makes), you can write it more compactly: --8---cut here---start-8--- foo.out: a.y b.y c.y X a.y b.y c.y -o foo.out %.y: %.x -cmp --silent $ $@ || cp $ $@ a.x b.x c.x: foo.org tangle foo.org --8---cut here---end---8--- HTH, Nick
Re: [O] Not overwriting unchanged source code files when tangling
Hey Nick, thank you very much. That sounds like a very good solution to my problem that does not require changes to org-mode. Best Holger On Sat, Nov 19, 2011 at 5:00 AM, Nick Dokos nicholas.do...@hp.com wrote: Holger Hoefling hhoef...@gmail.com wrote: I think you misunderstood me there - I am actually not worried about how computationally intensive the tangling process is. This always works very quickly, so even if they have to be copied around and take a bit longer, I would not mind. Ah, ok - so you are talking about tanglecompile org --- bunch of files - output The tangling step produces a bunch of files that are (re)compiled (or in any case require some sort of lengthy processing) to produce some output file. IMO, the best way to deal with it is still make: let's say foo.org a.x b.x c.x -- foo.out where the first arrow is the tangle and the second arrow is some processor, call it X. The standard way to set up a makefile is schematically: --8---cut here---start-8--- foo.out: a.x b.x c.x X a.x b.c c.x -o foo.out a.x b.x c.x: foo.org tangle foo.org --8---cut here---end---8--- Rewrite the make file as follows: --8---cut here---start-8--- foo.out: a.y b.y c.y X a.y b.y c.y -o foo.out a.y: a.x cmp --silent a.x a.y || cp a.x a.y b.y: b.x cmp --silent b.x b.y || cp b.x b.y c.y: c.x cmp --silent c.x c.y || cp c.x c.y a.x b.x c.x: foo.org tangle foo.org --8---cut here---end---8--- So if the *contents* of (say) a.x have not changed by the tangling, it compares equal to a.y and the copy is skipped. That leaves a.y untouched. OTOH, if the contents of a.x change (or a.y does not exist in the first place), the comparison fails and we copy a.x to a.y. That updates a.y and forces further updates on anything that depends on it. Using some make fu (works for GNU make, but not necessarily for other makes), you can write it more compactly: --8---cut here---start-8--- foo.out: a.y b.y c.y X a.y b.y c.y -o foo.out %.y: %.x -cmp --silent $ $@ || cp $ $@ a.x b.x c.x: foo.org tangle foo.org --8---cut here---end---8--- HTH, Nick
