Re: LyX document-build process
John Pye wrote: More coffee will only heighten my paranoia! Seriously, LyX is a GUI application but it makes no effort at being GUI-friendly during the document build process. It wouldn't be that hard to add a 'progress' dialog box for this stuff? With current architecture yes, a bit hard. But I personally plan to add some more feedback (note that there is some feedback already in the status bar about LateX processing numbers). Run a separate thread for the build tools and update the screen as they go? A thread is an option. A "detached" process is cleaner though (IMO). Both options are very feasible and you are welcome to help us ;-) I personally plan to work on the second option and Peter K. has already worked on the first option IIRC. Helps to know when images are too large or too detailed, or if something is hung, etc? Everything's possible :-) I think that this is a bug in LyX. LyX *must* be able to do this type of dependency tracking 100% correctly if it is going to attempt it at all. It is no good that it outputs the wrong document in this case. Agreed. I think that good architecture going forward might be for LyX to divest all of the build tracking stuff to an external program that specialises in dependency tracking, caching, and parallel building, and to try to re-use what others have already done in this area. Thoughts? This makes sense IMO, but as usual nothing happens by magic. Abdel.
Re: LyX document-build process
John Pye wrote: As I argued above, LyX contains *its own* build system: it defines tools and file transformations, dependency tracking and caching. Because it contains all of these things internally it makes using it within an external build system much more difficult that it should be. I think that good architecture going forward might be for LyX to divest all of the build tracking stuff to an external program that specialises in dependency tracking, caching, and parallel building, and to try to re-use what others have already done in this area. Thoughts? LyX needs passionate people to drive forward its development. As you're clearly passionate about this, then you should consider contributing code to realize your vision. Angus
Re: LyX document-build process
Hi Andre, Thanks for these comments. More follows: Andre Poenitz wrote: > On Tue, Jun 05, 2007 at 09:52:49PM +1000, John Pye wrote: > >> Sorry for a bit of a delay on this one. I thought I would follow up with >> some comments though. >> >> Firstly, >> >> Jean-Marc Lasgouttes wrote: [...] > >> The concept I had here was that one might be writing some software with >> a tarball that contains both C code and supporting LyX documentation. >> The documentation ultimately gets embedded into the application file in >> the form of let's say a windows .CHM help file. Imagine that a developer >> want to make some changes to a small part of the program and perhaps >> some small change in the documentation. If the LyX build process were >> somehow integrated with scons, then developer could just call 'scons' >> and rebuild the necessary bits of the program code plus the missing bits >> of the documentation. The dependency checker would have everything it >> needed without launching the full-blown LyX application to build the >> documentation. In this case, LyX would just do the job of perhaps >> transforming .lyx to .tex; scons would do the job of compiling the .tex >> file together with .png images together into a set of .html files and >> then convert the .html files to .chm files... >> > > People have successfully used plain Makefile for this purpose. > Although I am sure it can be made work (inefficiently), it just doesn't seem like the architecture is right for this. More comment further below. > >> At present, because lyx uses its own build system, the efficiencies of >> the above approach are not possible. >> > > ? > > Which 'own' build system? > LyX *does* contain a build system. The following features: * LyX can work out what image files need to be converted (dependency tracking) * LyX knows how to convert image files (build rules) * The LyX GUI detects when an image file is changed and triggers an updated file conversion (source file modification tracking) * LyX instantiates tex in order to create a PDF file (multiple steps to achieve required output file) * LyX knows the order that things need to be built in (graph algorithms) * LyX keeps a cache of converted image files and doesn't re-convert them all every time (output caching) * Build rules and file types can be specified by the user (user-defined build rules) * LyX contains certain hard-wired default rules (default build rules) It may not be full-blown thing but I don't see any clear distinction between where LyX functionality stops and where a 'full blown' build system starts. > >> I think that perhaps it does need a build system. My thesis takes a good >> five minutes to build, and there is absolutely no indication of how far >> through it is, or what it's doing at any given point in time. >> > > You know that it takes five minutes, and if it takes an average of two > minutes to fill your coffee mug you'll know when 40% are done... > More coffee will only heighten my paranoia! Seriously, LyX is a GUI application but it makes no effort at being GUI-friendly during the document build process. It wouldn't be that hard to add a 'progress' dialog box for this stuff? Run a separate thread for the build tools and update the screen as they go? Helps to know when images are too large or too detailed, or if something is hung, etc? [...] > >> When I change things in 'sub-files' in my thesis (ie chapters) LyX >> doesn't always correctly detect that something's changed, and it ends up >> producing the same PDF file as before I made my changes. It seems that >> there could be a missing link in the dependency tracking. >> > > Makefile. > I think that this is a bug in LyX. LyX *must* be able to do this type of dependency tracking 100% correctly if it is going to attempt it at all. It is no good that it outputs the wrong document in this case. [...] > >> So you mentioned that the build process is trivial but I think that the >> above examples show that it's perhaps not so trivial, and perhaps if a >> full-scale build system were used it might allow the LyX development >> effort to reduce the amount of 'support code' that needs to be >> maintained, so that you can focus on some of the more core LyX >> functionality? >> > > The point is that LyX is not a build system, cannot be, and does not > want to be. If you need one (and you have shown good reasons for that) > just use one. If you prefer Scons over make, so be it, but in both cases > this is a solution _wrapping_ LyX, not a solution integrated in LyX. > My basic problem here is that LyX is not really wrap-able, because of the way that it fires off an indeterminate number of sub-processes to convert images and other source material. It works quite differently from the usual tools that are called from a build script such as CPP, GCC, flex, bison, SWIG, etc. As I argued above, LyX
Re: LyX document-build process
On Tue, Jun 05, 2007 at 09:52:49PM +1000, John Pye wrote: > Sorry for a bit of a delay on this one. I thought I would follow up with > some comments though. > > Firstly, > > Jean-Marc Lasgouttes wrote: > > John> The current system seems (from my rather superficial inspection) > > John> to revolve around lots of little script files written in Python; > > > > It produces these scripts actually. The logic is in C++ and needs some > > information from the data structures that may not be available outside. > > > > Perhaps a better approach would be for LyX to export an API (for example > via SWIG) that allowed *static* scripts to access the information they > need at run time. Added advantage, other than all the text file > generation, is that this would allow the scripts to be byte-compiled. This would mean we need to keep some kind of API stable over time in order not to break external scripts. This is close to impossible as it basically forbids core changbes without introduction of compatibility layers. This might work for things that have been designed as libraries and long life cycles, but it is not feasible for a project in constant flux as we have with LyX. > Making the scripts static (rather than generated from inside C++) would > make solving problems such as my DIA one more straightforward and obvious. This is true, however, the price is far too high. > The concept I had here was that one might be writing some software with > a tarball that contains both C code and supporting LyX documentation. > The documentation ultimately gets embedded into the application file in > the form of let's say a windows .CHM help file. Imagine that a developer > want to make some changes to a small part of the program and perhaps > some small change in the documentation. If the LyX build process were > somehow integrated with scons, then developer could just call 'scons' > and rebuild the necessary bits of the program code plus the missing bits > of the documentation. The dependency checker would have everything it > needed without launching the full-blown LyX application to build the > documentation. In this case, LyX would just do the job of perhaps > transforming .lyx to .tex; scons would do the job of compiling the .tex > file together with .png images together into a set of .html files and > then convert the .html files to .chm files... People have successfully used plain Makefile for this purpose. > At present, because lyx uses its own build system, the efficiencies of > the above approach are not possible. ? Which 'own' build system? > I think that perhaps it does need a build system. My thesis takes a good > five minutes to build, and there is absolutely no indication of how far > through it is, or what it's doing at any given point in time. You know that it takes five minutes, and if it takes an average of two minutes to fill your coffee mug you'll know when 40% are done... > Building of images is not necessarily a simple process. For example, > what happens if I have an Asymptote-to-EPS filter and I am trying to > build a PDF file? And what if I have an SVG-to-PNG file and I am > producing a DVI? A proper build system would allow the missing 'tools' > to be clearly identified. Put it in a Makefile. > When I change things in 'sub-files' in my thesis (ie chapters) LyX > doesn't always correctly detect that something's changed, and it ends up > producing the same PDF file as before I made my changes. It seems that > there could be a missing link in the dependency tracking. Makefile. > In response to the above, I usually re-start lyx when I want to > re-create the PDF. But this seems to trigger a complete rebuild of > everything, even the bits such as image conversions (SVG-to-EPS etc). > Surely there should be caching of these things, and it should be robust > enough that LyX doesn't have to start over again on each instance. There > should be some kind of database of file hashes to check that source > files have not been modified -- and this sort of thing is provided for > example by SCons. You can warm up any cache you wantm e.g, from within a Makefile. > Processors are tending to go parallel these days, so there is going to > be a greater need for parallel builds. That is something that SCons > would allow, and this would make a big difference particularly for > preparing images for embedding in large documents. make -j 8 > Another problem is that the current system copies 'source' files into a > temporary directory before running the script. This causes files with > relative file references within them to break, as for example with DIA > files as I noticed recently. SCons fixes this problem because, at your > preference, it can either build files in-place, or can generate a > parallel 'build directory' into which all necessary dependency files are > copied before they are needed. You can provide 'copiers' that do the Right Thing. > Finally you mentioned that the scripts are generate
Re: LyX document-build process
Sorry for a bit of a delay on this one. I thought I would follow up with some comments though. Firstly, Jean-Marc Lasgouttes wrote: > John> The current system seems (from my rather superficial inspection) > John> to revolve around lots of little script files written in Python; > > It produces these scripts actually. The logic is in C++ and needs some > information from the data structures that may not be available outside. > Perhaps a better approach would be for LyX to export an API (for example via SWIG) that allowed *static* scripts to access the information they need at run time. Added advantage, other than all the text file generation, is that this would allow the scripts to be byte-compiled. Making the scripts static (rather than generated from inside C++) would make solving problems such as my DIA one more straightforward and obvious. > John> it doesn't provide any GUI feedback on progress (perhaps this is > John> changing in the new version) and can't be interrupted (as far as > John> I know). > > Indeed, but I am not sure using scons would help us. > SCons 'builders' can be provided with an output routine that could be hooked back into the GUI. Or you could simply have scons output piped to a 'console' window (that could be hidden behind a 'expander' widget if desired). > John> If SCons were uses as the 'wrapper' for the LyX build process, > John> its dependency checking, parallel build features, caching and > John> hashing, interruptability, and platform independence would > John> reward the developers and the users too, quite a bit, I reckon. > > I think it would be a big hamme for our medium sized nail. > I've made some more comments to argue that it's definitely no thumb-tack... See below. > John> It would also reward people who wanted to use LyX as the > John> documentation format for there software development projects, as > John> they would immediately have access to a simple build platform > John> for transforming their documentations source files into compiled > John> form without needing to involve a GUI. > > There is no need for the gui to export files from the command line. > The concept I had here was that one might be writing some software with a tarball that contains both C code and supporting LyX documentation. The documentation ultimately gets embedded into the application file in the form of let's say a windows .CHM help file. Imagine that a developer want to make some changes to a small part of the program and perhaps some small change in the documentation. If the LyX build process were somehow integrated with scons, then developer could just call 'scons' and rebuild the necessary bits of the program code plus the missing bits of the documentation. The dependency checker would have everything it needed without launching the full-blown LyX application to build the documentation. In this case, LyX would just do the job of perhaps transforming .lyx to .tex; scons would do the job of compiling the .tex file together with .png images together into a set of .html files and then convert the .html files to .chm files... At present, because lyx uses its own build system, the efficiencies of the above approach are not possible. > John> With the stuff that LyX is already providing for building > John> documents from all those different source file formats, I wonder > John> where the limit is -- how much of a full-blow build system to > John> you develop as part of LyX before you stop and just reuse > John> something else that handles the whole thing? Or are there > John> problems with that? > > As often, 90% would be trivial, and the rest would be an horrible pain :) I'm sure you're right and there would be quite a bit of work in this, even though superficially it does sound fairly simple. Angus' comments now: Angus Leeming wrote: > Jean-Marc Lasgouttes <[EMAIL PROTECTED]> writes: > >> John> With the stuff that LyX is already providing for building >> John> documents from all those different source file formats, I wonder >> John> where the limit is -- how much of a full-blow build system to >> John> you develop as part of LyX before you stop and just reuse >> John> something else that handles the whole thing? Or are there >> John> problems with that? >> >> As often, 90% would be trivial, and the rest would be an horrible pain :) >> > > I don't think that LyX has any need of a build system. The dependencies are > trivial: the LyX document needs all its preconditions to be built (images > translated to some format) before it can be built itself. These image > transformations do not depend on one another, so no complex tracking is > needed. > I think that perhaps it does need a build system. My thesis takes a good five minutes to build, and there is absolutely no indication of how far through it is, or what it's doing at any given point in time. And... Building of images is not necessarily a simple process. For example, what happens if I have an As
Re: LyX document-build process
Jean-Marc Lasgouttes <[EMAIL PROTECTED]> writes: > John> With the stuff that LyX is already providing for building > John> documents from all those different source file formats, I wonder > John> where the limit is -- how much of a full-blow build system to > John> you develop as part of LyX before you stop and just reuse > John> something else that handles the whole thing? Or are there > John> problems with that? > > As often, 90% would be trivial, and the rest would be an horrible pain :) I don't think that LyX has any need of a build system. The dependencies are trivial: the LyX document needs all its preconditions to be built (images translated to some format) before it can be built itself. These image transformations do not depend on one another, so no complex tracking is needed. As Jean-Marc says, some scripts are generated dynamically (there's a script per image translation) by chaining together the meta-translation information in the "converters" concept. If you run LyX as "lyx -dbg graphics" from the command line you should (used to ;-)) get a dump to the console of exactly what these scripts contain. Other scripts exist already and are simply leveraged by the LyX C++ code. Have a look in the lib/scripts subdirectory. Kind regards (and thanks for the reality check ;-)), Angus
Re: LyX document-build process
> "John" == John Pye <[EMAIL PROTECTED]> writes: John> The current system seems (from my rather superficial inspection) John> to revolve around lots of little script files written in Python; It produces these scripts actually. The logic is in C++ and needs some information from the data structures that may not be available outside. John> it doesn't provide any GUI feedback on progress (perhaps this is John> changing in the new version) and can't be interrupted (as far as John> I know). Indeed, but I am not sure using scons would help us. John> If SCons were uses as the 'wrapper' for the LyX build process, John> its dependency checking, parallel build features, caching and John> hashing, interruptability, and platform independence would John> reward the developers and the users too, quite a bit, I reckon. I think it would be a big hamme for our medium sized nail. John> It would also reward people who wanted to use LyX as the John> documentation format for there software development projects, as John> they would immediately have access to a simple build platform John> for transforming their documentations source files into compiled John> form without needing to involve a GUI. There is no need for the gui to export files from the command line. John> With the stuff that LyX is already providing for building John> documents from all those different source file formats, I wonder John> where the limit is -- how much of a full-blow build system to John> you develop as part of LyX before you stop and just reuse John> something else that handles the whole thing? Or are there John> problems with that? As often, 90% would be trivial, and the rest would be an horrible pain :) JMarc
LyX document-build process (was Re: DIA diagram file with embedded (linked?) bitmap fails to convert)
Out of curiosity, has anyone ever considered embedding something like SCons into LyX? A python-based system for building 'targets' from disparate 'sources' -- in this case LyX documents, bitmaps, diagrams, SVGs, EPSs etc? The current system seems (from my rather superficial inspection) to revolve around lots of little script files written in Python; it doesn't provide any GUI feedback on progress (perhaps this is changing in the new version) and can't be interrupted (as far as I know). If SCons were uses as the 'wrapper' for the LyX build process, its dependency checking, parallel build features, caching and hashing, interruptability, and platform independence would reward the developers and the users too, quite a bit, I reckon. It would also reward people who wanted to use LyX as the documentation format for there software development projects, as they would immediately have access to a simple build platform for transforming their documentations source files into compiled form without needing to involve a GUI. With the stuff that LyX is already providing for building documents from all those different source file formats, I wonder where the limit is -- how much of a full-blow build system to you develop as part of LyX before you stop and just reuse something else that handles the whole thing? Or are there problems with that? In the context of the little script mentioned here by Angus, I was reminded of the SCons 'builder' and 'BuildDir' functionality. I believe Bo Peng will know about this: as I recall he (I assume 'he') did some work on a SCons build for the LyX app. SCons has all this stuff built it -- all you need to specify are a couple of regular expressions. I'm not at all familiar with the internals of LyX so perhaps this is already underway, or perhaps there are lots of good reasons for using some other approach. Just some thoughts. Cheers JP Angus Leeming wrote: > We have special logic to handle a similar situation with XFig files. You'll > need to write a little script to parse the Dia file to extract the referenced > image files so that they can be copied to the temporary directory. > > Angus > >
