Re: questions about using Doxygen and Python scripting

[Julien Rioux]

On 22/07/2013 4:10 AM, Guenter Milde wrote:

On 2013-07-22, Josh Hieronymus wrote:




Second, is there a tutorial
serves as a gentle introduction to Python scripting for LyX? I've not
worked with Python and C++ in the same project before, so I'm not really
sure how to get them to work together, especially when you consider issues
like type checking, namespaces, header files, and so on.


The Python scripts that come with LyX are all separate tools and pure
Python. There is no LyX Python module in C or C++.



Exactly, we just call the python interpreter on our scripts (or external 
scripts). They are run as separate processes, nothing tricky.



See the existing tools for examples how to configure a Python script as
exporter.



What Günter refers to is lib/configure.py where target export formats 
are defined, converters are being detected whther they are installed or 
not, etc. So e.g. ePub could be added as format, and configure could 
check if the lyx2epub.py converter (or whatever) is installed and define 
it as a converter. Then when LyX is requested to export to ePub, we call 
the python interpreter to run the python script ina temporary folder and 
move its output file(s) to the export folder.


Cheers,
Julien

Re: questions about using Doxygen and Python scripting

[Julien Rioux]

On 22/07/2013 4:10 AM, Guenter Milde wrote:

On 2013-07-22, Josh Hieronymus wrote:




Second, is there a tutorial
serves as a gentle introduction to Python scripting for LyX? I've not
worked with Python and C++ in the same project before, so I'm not really
sure how to get them to work together, especially when you consider issues
like type checking, namespaces, header files, and so on.


The Python scripts that come with LyX are all separate tools and pure
Python. There is no LyX Python module in C or C++.



Exactly, we just call the python interpreter on our scripts (or external 
scripts). They are run as separate processes, nothing tricky.



See the existing tools for examples how to configure a Python script as
exporter.



What Günter refers to is lib/configure.py where target export formats 
are defined, converters are being detected whther they are installed or 
not, etc. So e.g. ePub could be added as format, and configure could 
check if the lyx2epub.py converter (or whatever) is installed and define 
it as a converter. Then when LyX is requested to export to ePub, we call 
the python interpreter to run the python script ina temporary folder and 
move its output file(s) to the export folder.


Cheers,
Julien

Re: questions about using Doxygen and Python scripting

[Guenter Milde]

On 2013-07-22, Josh Hieronymus wrote:

...

 Second, is there a tutorial
 serves as a gentle introduction to Python scripting for LyX? I've not
 worked with Python and C++ in the same project before, so I'm not really
 sure how to get them to work together, especially when you consider issues
 like type checking, namespaces, header files, and so on.

The Python scripts that come with LyX are all separate tools and pure
Python. There is no LyX Python module in C or C++.

See the existing tools for examples how to configure a Python script as
exporter.

There is the generic lyxserver for sending communication between a
running LyX and an external program via pipes. I wrote a Python module to
get a more pythonic interface. It can be found at the LyX wiki.
http://wiki.lyx.org/Tools/PyClient

Günter

Re: questions about using Doxygen and Python scripting

[Guenter Milde]

On 2013-07-22, Josh Hieronymus wrote:

...

> Second, is there a tutorial
> serves as a gentle introduction to Python scripting for LyX? I've not
> worked with Python and C++ in the same project before, so I'm not really
> sure how to get them to work together, especially when you consider issues
> like type checking, namespaces, header files, and so on.

The Python scripts that come with LyX are all separate tools and pure
Python. There is no LyX Python module in C or C++.

See the existing tools for examples how to configure a Python script as
exporter.

There is the generic "lyxserver" for sending communication between a
running LyX and an external program via pipes. I wrote a Python module to
get a more "pythonic" interface. It can be found at the LyX wiki.
http://wiki.lyx.org/Tools/PyClient

Günter

questions about using Doxygen and Python scripting

[Josh Hieronymus]

Hi everyone,

I have two questions for everyone. First, is there a simple command to use
Doxygen to produce HTML documenting all of (or most of, or at least much
of) LyX? I think it would really help me have an easier time knowing how
the different parts of code are all related. Second, is there a tutorial
serves as a gentle introduction to Python scripting for LyX? I've not
worked with Python and C++ in the same project before, so I'm not really
sure how to get them to work together, especially when you consider issues
like type checking, namespaces, header files, and so on.

Thanks,
Josh

Re: questions about using Doxygen and Python scripting

[Pavel Sanda]

Josh Hieronymus wrote:
 I have two questions for everyone. First, is there a simple command to use
 Doxygen to produce HTML documenting all of (or most of, or at least much
 of) LyX? I think it would really help me have an easier time knowing how

make doxydoc, if you have graphviz, you will get nice pictures as well.

 the different parts of code are all related. Second, is there a tutorial
 serves as a gentle introduction to Python scripting for LyX? I've not

No. What do you need python for btw?
Pavel

Re: questions about using Doxygen and Python scripting

[Josh Hieronymus]

On Sun, Jul 21, 2013 at 8:58 PM, Pavel Sanda sa...@lyx.org wrote:

 Josh Hieronymus wrote:
  I have two questions for everyone. First, is there a simple command to
 use
  Doxygen to produce HTML documenting all of (or most of, or at least much
  of) LyX? I think it would really help me have an easier time knowing how

 make doxydoc, if you have graphviz, you will get nice pictures as well.

  the different parts of code are all related. Second, is there a tutorial
  serves as a gentle introduction to Python scripting for LyX? I've not

 No. What do you need python for btw?
 Pavel


Since an ePub file is pretty much a zipped file with XHTML and a few simple
files, I was thinking that it might be easiest to use Python to create
them. Steve Litt just recently posted a script to the LyX-users mailing
list that does much of the necessary work, and either extending it or using
it as inspiration for another script would likely make matters simpler.

- Josh

Re: questions about using Doxygen and Python scripting

[Pavel Sanda]

Josh Hieronymus wrote:
 Since an ePub file is pretty much a zipped file with XHTML and a few simple
 files, I was thinking that it might be easiest to use Python to create
 them.

If it comes to packing you might be interested to see our lyxpak.py script.
I'm not sure what you mean by 'create'; creating actual content of the file
should be done via c++ routines we already have for xhtml export.

Pavel

Re: questions about using Doxygen and Python scripting

[Josh Hieronymus]

On Sun, Jul 21, 2013 at 10:19 PM, Pavel Sanda sa...@lyx.org wrote:

 Josh Hieronymus wrote:
  Since an ePub file is pretty much a zipped file with XHTML and a few
 simple
  files, I was thinking that it might be easiest to use Python to create
  them.

 If it comes to packing you might be interested to see our lyxpak.py script.
 I'm not sure what you mean by 'create'; creating actual content of the file
 should be done via c++ routines we already have for xhtml export.

 Pavel



I was thinking of using the existing C++ routines to generate the XHTML,
and then using Python to generate the additional necessary files, which are
mostly XML files with metadata and manifest info, as well as to pack the
files.

Thanks for the doxygen help, by the way. That's really helping me to
understand the code even more.

Re: questions about using Doxygen and Python scripting

[Pavel Sanda]

Josh Hieronymus wrote:
 I was thinking of using the existing C++ routines to generate the XHTML,
 and then using Python to generate the additional necessary files, which are
 mostly XML files with metadata and manifest info, as well as to pack the
 files.

My point was just that you shouldn't use any python parsing of .lyx files
as does the library published by Steve, but all document data should by
fired out from our internal structures.

I can't tell how much easier is using python for packing things (need for some
3rd party library for python packing would be bad for example), just note that
we already have working library for zipping files as a part of LyX and commit
history might even contain much more generic solution written some time back by
Bo Peng as a part of bundled lyx format feature (and reverted later;).

Pavel

Re: questions about using Doxygen and Python scripting

[Pavel Sanda]

Pavel Sanda wrote:
 we already have working library for zipping files as a part of LyX and commit
 history might even contain much more generic solution written some time back 
 by
 Bo Peng as a part of bundled lyx format feature (and reverted later;).

You can check commits around
a095d0d785ca3f76f

questions about using Doxygen and Python scripting

[Josh Hieronymus]

Hi everyone,

I have two questions for everyone. First, is there a simple command to use
Doxygen to produce HTML documenting all of (or most of, or at least much
of) LyX? I think it would really help me have an easier time knowing how
the different parts of code are all related. Second, is there a tutorial
serves as a gentle introduction to Python scripting for LyX? I've not
worked with Python and C++ in the same project before, so I'm not really
sure how to get them to work together, especially when you consider issues
like type checking, namespaces, header files, and so on.

Thanks,
Josh

Re: questions about using Doxygen and Python scripting

[Pavel Sanda]

Josh Hieronymus wrote:
> I have two questions for everyone. First, is there a simple command to use
> Doxygen to produce HTML documenting all of (or most of, or at least much
> of) LyX? I think it would really help me have an easier time knowing how

make doxydoc, if you have graphviz, you will get nice pictures as well.

> the different parts of code are all related. Second, is there a tutorial
> serves as a gentle introduction to Python scripting for LyX? I've not

No. What do you need python for btw?
Pavel

Re: questions about using Doxygen and Python scripting

[Josh Hieronymus]

On Sun, Jul 21, 2013 at 8:58 PM, Pavel Sanda <sa...@lyx.org> wrote:

> Josh Hieronymus wrote:
> > I have two questions for everyone. First, is there a simple command to
> use
> > Doxygen to produce HTML documenting all of (or most of, or at least much
> > of) LyX? I think it would really help me have an easier time knowing how
>
> make doxydoc, if you have graphviz, you will get nice pictures as well.
>
> > the different parts of code are all related. Second, is there a tutorial
> > serves as a gentle introduction to Python scripting for LyX? I've not
>
> No. What do you need python for btw?
> Pavel
>

Since an ePub file is pretty much a zipped file with XHTML and a few simple
files, I was thinking that it might be easiest to use Python to create
them. Steve Litt just recently posted a script to the LyX-users mailing
list that does much of the necessary work, and either extending it or using
it as inspiration for another script would likely make matters simpler.

- Josh

Re: questions about using Doxygen and Python scripting

[Pavel Sanda]

Josh Hieronymus wrote:
> Since an ePub file is pretty much a zipped file with XHTML and a few simple
> files, I was thinking that it might be easiest to use Python to create
> them.

If it comes to packing you might be interested to see our lyxpak.py script.
I'm not sure what you mean by 'create'; creating actual content of the file
should be done via c++ routines we already have for xhtml export.

Pavel

Re: questions about using Doxygen and Python scripting

[Josh Hieronymus]

On Sun, Jul 21, 2013 at 10:19 PM, Pavel Sanda  wrote:

> Josh Hieronymus wrote:
> > Since an ePub file is pretty much a zipped file with XHTML and a few
> simple
> > files, I was thinking that it might be easiest to use Python to create
> > them.
>
> If it comes to packing you might be interested to see our lyxpak.py script.
> I'm not sure what you mean by 'create'; creating actual content of the file
> should be done via c++ routines we already have for xhtml export.
>
> Pavel
>


I was thinking of using the existing C++ routines to generate the XHTML,
and then using Python to generate the additional necessary files, which are
mostly XML files with metadata and manifest info, as well as to pack the
files.

Thanks for the doxygen help, by the way. That's really helping me to
understand the code even more.

Re: questions about using Doxygen and Python scripting

[Pavel Sanda]

Josh Hieronymus wrote:
> I was thinking of using the existing C++ routines to generate the XHTML,
> and then using Python to generate the additional necessary files, which are
> mostly XML files with metadata and manifest info, as well as to pack the
> files.

My point was just that you shouldn't use any python parsing of .lyx files
as does the library published by Steve, but all document data should by
fired out from our internal structures.

I can't tell how much easier is using python for packing things (need for some
3rd party library for python packing would be bad for example), just note that
we already have working library for zipping files as a part of LyX and commit
history might even contain much more generic solution written some time back by
Bo Peng as a part of bundled lyx format feature (and reverted later;).

Pavel

Re: questions about using Doxygen and Python scripting

[Pavel Sanda]

Pavel Sanda wrote:
> we already have working library for zipping files as a part of LyX and commit
> history might even contain much more generic solution written some time back 
> by
> Bo Peng as a part of bundled lyx format feature (and reverted later;).

You can check commits around
a095d0d785ca3f76f

Re: Scripting vs plugins (was RE: Goals for 2.1)

[Tommaso Cucinotta]

Il 05/05/2011 22:40, venom00 ha scritto:

My idea was to issue commands to LyX via LFUNs, which are

quite stable, even because they're associated with
customizable shortcuts. I think this is a very uninvasive approach.

For the language I prefer Python because _a lot_ of people

uses it and I think this is fundamental if we want to hope in
third party cooperation.

I had a similar idea to record/replay such LFUNs series,
especially for reproducing bug it could be useful.

Sure, scripting and macro are strictly related. Record a macro would be very 
simple as each user interaction with LyX goes through a LyX user command. We 
need several new LFUNs to have a useful and complete scripting envinronment but 
I think that's the way to follow.


I think I have quite an uncommon opinion (among LyX developers) about 
what LFUNs are causing (perhaps as a side effect) in the LyX code. 
Basically, many classes use this machinery to invoke operations, with 
the result that sometimes the class does not get a properly designed 
interface that users of that class can use. You may say you have the 
LFUN interface, however it's not a binary interface (it involves 
stringifying/parsing of the arguments), and it has no way to return 
anything whatsoever (except for a boolean, maybe).
I find recurrently myself in searching how to do something with an 
object, then I follow the LFUN path that achieves a similar/related 
effect, then I see there's no method call to do that, but the LFUN is at 
the same time tied to a specific GUI interaction, so it's just more 
complex that what I need. Normally, I copy some code from the LFUN 
implementation in other parts of the code, because I don't fell 
comfortable in breaking the no-interface pattern currently used in 
LyX. But I know copying code is bad as well.


I can still remember the time when I proposed first the Advanced FR 
patch: it used no LFUNs, but simply straightforward calls from the 
dialog to the objects I needed. Somebody told me in LyX, GUI-model 
interactions happen through this command pattern, so please use it, and 
I did. However, just digging into the past e-mail archive, you can 
easily find:
-) all the places where this LFUN creates me problem due to its inherent 
limitations;
-) my vein attempts to suggest a kind of binary LFUN interface, as 
opposed to the string-based current one (you can keep some string-based 
serialization of arguments, for when the user uses [A-x]whatever, but 
I see the stringification/parsing in the software components when 
talking to each other that resembles me some XML-RPC bloatware (but 
who cares on nowadays CPUs, right ?)

-) a few vein attempts to introduce LFUN return values.

As I said, probably just a side effect, and LFUNs (and string-based 
dispatching) and are certainly useful. However, let's pay attention when 
we say that we can do everything by LFUNs.


Comments are welcome of course.

T.

RE: Scripting vs plugins (was RE: Goals for 2.1)

[venom00]

 I think I have quite an uncommon opinion (among LyX developers) about 
 what LFUNs are causing (perhaps as a side effect) in the LyX code. 
 Basically, many classes use this machinery to invoke operations, with 
 the result that sometimes the class does not get a properly designed 
 interface that users of that class can use. You may say you have the 
 LFUN interface, however it's not a binary interface (it involves 
 stringifying/parsing of the arguments), and it has no way to return 
 anything whatsoever (except for a boolean, maybe).
 I find recurrently myself in searching how to do something with an 
 object, then I follow the LFUN path that achieves a similar/related 
 effect, then I see there's no method call to do that, but the 
 LFUN is at 
 the same time tied to a specific GUI interaction, so it's just more 
 complex that what I need. Normally, I copy some code from the LFUN 
 implementation in other parts of the code, because I don't fell 
 comfortable in breaking the no-interface pattern currently used in 
 LyX. But I know copying code is bad as well.

I don't a very deep knowledge of the LyX code but I agree with you that using 
LFUNs in the code should be avoided, if possible, but...

 Somebody told me in LyX, GUI-model 
 interactions happen through this command pattern, so please 
 use it

LFUNs are great for user interaction as they are a strict but human-friendly 
interface. Automate them in macros should be simple.

 However, just digging into the past e-mail archive, you can 
 easily find:
 -) all the places where this LFUN creates me problem due to 
 its inherent 
 limitations;
 -) my vein attempts to suggest a kind of binary LFUN interface, as 
 opposed to the string-based current one (you can keep some 
 string-based 
 serialization of arguments, for when the user uses 
 [A-x]whatever, but 
 I see the stringification/parsing in the software components when 
 talking to each other that resembles me some XML-RPC bloatware (but 
 who cares on nowadays CPUs, right ?)

Well, XML-RPC has a certain amount of advantages :P

Remember however that only arguments are parsed, the LFUN code is numerical. 
Moreover most of the LFUNs don't take arguments or frequently they're text. Do 
we really want make such a big change just for LFUNs taking numbers?
Consider the following LFUN:

case LFUN_SET_COLOR: {
string lyx_name;
string const x11_name = split(to_utf8(cmd.argument()), lyx_name, ' ');
...

LFUN arguments are used as a unique string by LFUN implementation itself, it 
would require a lot of rewrite to change this. 

In my idea the scripting won't be something like:
lfun(lyx-command argument 123)

But:
lyxInterface.lyxCommand(argument, 123)

Which internally will be:
def lyxCommand:
lfun(LFUN_ID, args.join( ))

In this way we could make a binary implementation of LFUNs in the future and 
just update the lyxInterface internals.

 -) a few vein attempts to introduce LFUN return values.

That's fundamental for the scripting engine.

venom00

RE: Scripting vs plugins (was RE: Goals for 2.1)

[venom00]

  Lua
 + small and fast,
 + used in LuaTeX, so it will become more common and known in the
   TeX community,
 + a Lua interpreter can be embedded in LyX with minimal 
 impact on
   the binary size.
 
  Wasn't there another thread with the result that LyX is not 
 bloated enough?
 
 According to http://www.lua.org/about.html
 
   The tarball for Lua 5.1.4, which contains source code, 
 documentation,
   and examples, takes 212K compressed and 860K uncompressed. 
 The source
   contains around 17000 lines of C. Under Linux, the Lua interpreter
   built with all standard Lua libraries takes 153K and the Lua library
   takes 203K. 
 
 Compare this to a minimal Python installation or the size of the LyX
 binary.

We already require Python, that's a point in its favour.

  Or did I get it wrong? ;)
 
 The idea is to outsource¹ tasks. This would reduce the binary size and
 memory footprint even more. An example would be the parsing of the
 unicodesymbols file.

Weren't we talking about scripting? However seems hard that making something in 
an interpreted language could improve performances compared with (good) C++.

venom00

P.S. Please use this thread.

Re: Scripting vs plugins

[Guenter Milde]

On 2011-05-06, venom00 wrote:
  Lua
 + small and fast,
 + used in LuaTeX, so it will become more common and known in the
   TeX community,
 + a Lua interpreter can be embedded in LyX with minimal 
 impact on
   the binary size.

  Wasn't there another thread with the result that LyX is not 
  bloated enough?

Actually this was a response to the allow plug-ins sub-thread of
the goals for 2.1. The idea was to reduce the memory footprint by a
plugin system.

 According to http://www.lua.org/about.html

   The tarball for Lua 5.1.4, which contains source code, 
 documentation,
   and examples, takes 212K compressed and 860K uncompressed. 
 The source
   contains around 17000 lines of C. Under Linux, the Lua interpreter
   built with all standard Lua libraries takes 153K and the Lua library
   takes 203K. 

 Compare this to a minimal Python installation or the size of the LyX
 binary.

 We already require Python, that's a point in its favour.

If we are talking about external tools, yes. 
But it is also about requiring vs. embedding.

For Python scripting, I have written a PyClient package
http://wiki.lyx.org/Tools/PyClient that I would welcome to see adopted.


 The idea is to outsource¹ tasks. This would reduce the binary size and
 memory footprint even more. An example would be the parsing of the
 unicodesymbols file.

 Weren't we talking about scripting? However seems hard that making
 something in an interpreted language could improve performances
 compared with (good) C++.

With an embedded Lua interpreter, OTOH the impact on performance will be
small for tasks like writing/parsing config files, lyx2lyx, tex2lyx etc.
The intelligence can be stored in scripts that do not bloat the main
binary.

Günter

Re: Scripting vs plugins

[Andre Poenitz]

On Fri, May 06, 2011 at 07:33:35PM +, Guenter Milde wrote:
 On 2011-05-06, venom00 wrote:
   Lua
  + small and fast,
  + used in LuaTeX, so it will become more common and known in the
TeX community,
  + a Lua interpreter can be embedded in LyX with minimal 
  impact on
the binary size.
 
   Wasn't there another thread with the result that LyX is not 
   bloated enough?
 
 Actually this was a response to the allow plug-ins sub-thread of
 the goals for 2.1. The idea was to reduce the memory footprint by a
 plugin system.

But honestly, the memory footprint of LyX is really not a problem.
We are talking about a 8.5 stripped binary. Some screensavers have
more fat. It's really not worth even _thinking_ about a plugin system
to reduce binary size.

 [...]
 With an embedded Lua interpreter, OTOH the impact on performance will be
 small for tasks like writing/parsing config files, lyx2lyx, tex2lyx etc.
 The intelligence can be stored in scripts that do not bloat the main
 binary.

What bloat, please?

Andre'

Re: Scripting vs plugins (was RE: Goals for 2.1)

[Tommaso Cucinotta]

Il 05/05/2011 22:40, venom00 ha scritto:

My idea was to issue commands to LyX via LFUNs, which are

quite stable, even because they're associated with
customizable shortcuts. I think this is a very uninvasive approach.

For the language I prefer Python because _a lot_ of people

uses it and I think this is fundamental if we want to hope in
third party cooperation.

I had a similar idea to record/replay such LFUNs series,
especially for reproducing bug it could be useful.

Sure, scripting and macro are strictly related. Record a macro would be very 
simple as each user interaction with LyX goes through a LyX user command. We 
need several new LFUNs to have a useful and complete scripting envinronment but 
I think that's the way to follow.


I think I have quite an uncommon opinion (among LyX developers) about 
what LFUNs are causing (perhaps as a side effect) in the LyX code. 
Basically, many classes use this machinery to invoke operations, with 
the result that sometimes the class does not get a properly designed 
interface that users of that class can use. You may say "you have the 
LFUN interface", however it's not a "binary interface" (it involves 
stringifying/parsing of the arguments), and it has no way to return 
anything whatsoever (except for a boolean, maybe).
I find recurrently myself in searching how to do something with an 
object, then I follow the LFUN path that achieves a similar/related 
effect, then I see there's no method call to do that, but the LFUN is at 
the same time tied to a specific GUI interaction, so it's just more 
complex that what I need. Normally, I copy some code from the LFUN 
implementation in other parts of the code, because I don't fell 
comfortable in breaking the "no-interface" pattern currently used in 
LyX. But I know copying code is bad as well.


I can still remember the time when I proposed first the Advanced F 
patch: it used no LFUNs, but simply straightforward calls from the 
dialog to the objects I needed. Somebody told me "in LyX, GUI<->model 
interactions happen through this command pattern, so please use it", and 
I did. However, just digging into the past e-mail archive, you can 
easily find:
-) all the places where this LFUN creates me problem due to its inherent 
limitations;
-) my vein attempts to suggest a kind of "binary" LFUN interface, as 
opposed to the string-based current one (you can keep some string-based 
serialization of arguments, for when the user uses "[A-x]whatever", but 
I see the stringification/parsing in the software components when 
talking to each other that resembles me some XML-RPC "bloatware" (but 
who cares on nowadays CPUs, right ?)

-) a few vein attempts to introduce LFUN return values.

As I said, probably just a side effect, and LFUNs (and string-based 
dispatching) and are certainly useful. However, let's pay attention when 
we say that we can do everything by LFUNs.


Comments are welcome of course.

T.

RE: Scripting vs plugins (was RE: Goals for 2.1)

[venom00]

> I think I have quite an uncommon opinion (among LyX developers) about 
> what LFUNs are causing (perhaps as a side effect) in the LyX code. 
> Basically, many classes use this machinery to invoke operations, with 
> the result that sometimes the class does not get a properly designed 
> interface that users of that class can use. You may say "you have the 
> LFUN interface", however it's not a "binary interface" (it involves 
> stringifying/parsing of the arguments), and it has no way to return 
> anything whatsoever (except for a boolean, maybe).
> I find recurrently myself in searching how to do something with an 
> object, then I follow the LFUN path that achieves a similar/related 
> effect, then I see there's no method call to do that, but the 
> LFUN is at 
> the same time tied to a specific GUI interaction, so it's just more 
> complex that what I need. Normally, I copy some code from the LFUN 
> implementation in other parts of the code, because I don't fell 
> comfortable in breaking the "no-interface" pattern currently used in 
> LyX. But I know copying code is bad as well.

I don't a very deep knowledge of the LyX code but I agree with you that using 
LFUNs in the code should be avoided, if possible, but...

> Somebody told me "in LyX, GUI<->model 
> interactions happen through this command pattern, so please 
> use it"

LFUNs are great for user interaction as they are a strict but human-friendly 
interface. Automate them in macros should be simple.

> However, just digging into the past e-mail archive, you can 
> easily find:
> -) all the places where this LFUN creates me problem due to 
> its inherent 
> limitations;
> -) my vein attempts to suggest a kind of "binary" LFUN interface, as 
> opposed to the string-based current one (you can keep some 
> string-based 
> serialization of arguments, for when the user uses 
> "[A-x]whatever", but 
> I see the stringification/parsing in the software components when 
> talking to each other that resembles me some XML-RPC "bloatware" (but 
> who cares on nowadays CPUs, right ?)

Well, XML-RPC has a certain amount of advantages :P

Remember however that only arguments are parsed, the LFUN code is numerical. 
Moreover most of the LFUNs don't take arguments or frequently they're text. Do 
we really want make such a big change just for LFUNs taking numbers?
Consider the following LFUN:

case LFUN_SET_COLOR: {
string lyx_name;
string const x11_name = split(to_utf8(cmd.argument()), lyx_name, ' ');
...

LFUN arguments are used as a unique string by LFUN implementation itself, it 
would require a lot of rewrite to change this. 

In my idea the scripting won't be something like:
lfun("lyx-command argument 123")

But:
lyxInterface.lyxCommand("argument", 123)

Which internally will be:
def lyxCommand:
lfun(LFUN_ID, args.join(" "))

In this way we could make a "binary" implementation of LFUNs in the future and 
just update the lyxInterface internals.

> -) a few vein attempts to introduce LFUN return values.

That's fundamental for the scripting engine.

venom00

RE: Scripting vs plugins (was RE: Goals for 2.1)

[venom00]

> >> Lua
> >>+ small and fast,
> >>+ used in LuaTeX, so it will become more common and known in the
> >>  TeX community,
> >>+ a Lua interpreter can be embedded in LyX with minimal 
> impact on
> >>  the binary size.
> 
> > Wasn't there another thread with the result that LyX is not 
> bloated enough?
> 
> According to http://www.lua.org/about.html
> 
>   The tarball for Lua 5.1.4, which contains source code, 
> documentation,
>   and examples, takes 212K compressed and 860K uncompressed. 
> The source
>   contains around 17000 lines of C. Under Linux, the Lua interpreter
>   built with all standard Lua libraries takes 153K and the Lua library
>   takes 203K. 
> 
> Compare this to a minimal Python installation or the size of the LyX
> binary.

We already require Python, that's a point in its favour.

> > Or did I get it wrong? ;)
> 
> The idea is to outsource¹ tasks. This would reduce the binary size and
> memory footprint even more. An example would be the parsing of the
> unicodesymbols file.

Weren't we talking about scripting? However seems hard that making something in 
an interpreted language could improve performances compared with (good) C++.

venom00

P.S. Please use this thread.

Re: Scripting vs plugins

[Guenter Milde]

On 2011-05-06, venom00 wrote:
>> >> Lua
>> >>+ small and fast,
>> >>+ used in LuaTeX, so it will become more common and known in the
>> >>  TeX community,
>> >>+ a Lua interpreter can be embedded in LyX with minimal 
>> impact on
>> >>  the binary size.

>> > Wasn't there another thread with the result that LyX is not 
>> > bloated enough?

Actually this was a response to the "allow plug-ins" sub-thread of
the "goals for 2.1". The idea was to reduce the memory footprint by a
plugin system.

>> According to http://www.lua.org/about.html

>>   The tarball for Lua 5.1.4, which contains source code, 
>> documentation,
>>   and examples, takes 212K compressed and 860K uncompressed. 
>> The source
>>   contains around 17000 lines of C. Under Linux, the Lua interpreter
>>   built with all standard Lua libraries takes 153K and the Lua library
>>   takes 203K. 

>> Compare this to a minimal Python installation or the size of the LyX
>> binary.

> We already require Python, that's a point in its favour.

If we are talking about external tools, yes. 
But it is also about requiring vs. embedding.

For Python scripting, I have written a PyClient package
http://wiki.lyx.org/Tools/PyClient that I would welcome to see adopted.


>> The idea is to outsource¹ tasks. This would reduce the binary size and
>> memory footprint even more. An example would be the parsing of the
>> unicodesymbols file.

> Weren't we talking about scripting? However seems hard that making
> something in an interpreted language could improve performances
> compared with (good) C++.

With an embedded Lua interpreter, OTOH the impact on performance will be
small for tasks like writing/parsing config files, lyx2lyx, tex2lyx etc.
The "intelligence" can be stored in scripts that do not bloat the main
binary.

Günter

Re: Scripting vs plugins

[Andre Poenitz]

On Fri, May 06, 2011 at 07:33:35PM +, Guenter Milde wrote:
> On 2011-05-06, venom00 wrote:
> >> >> Lua
> >> >>+ small and fast,
> >> >>+ used in LuaTeX, so it will become more common and known in the
> >> >>  TeX community,
> >> >>+ a Lua interpreter can be embedded in LyX with minimal 
> >> impact on
> >> >>  the binary size.
> 
> >> > Wasn't there another thread with the result that LyX is not 
> >> > bloated enough?
> 
> Actually this was a response to the "allow plug-ins" sub-thread of
> the "goals for 2.1". The idea was to reduce the memory footprint by a
> plugin system.

But honestly, the memory footprint of LyX is really not a problem.
We are talking about a 8.5 stripped binary. Some screensavers have
more fat. It's really not worth even _thinking_ about a plugin system
to reduce binary size.

> [...]
> With an embedded Lua interpreter, OTOH the impact on performance will be
> small for tasks like writing/parsing config files, lyx2lyx, tex2lyx etc.
> The "intelligence" can be stored in scripts that do not bloat the main
> binary.

What bloat, please?

Andre'

Scripting vs plugins (was RE: Goals for 2.1)

[venom00]

Jean Marc said:
 The problem with script plugins is that people seem to expect that by 
 linking LyX to python everybody will be able to write python scritps 
 that can manipulate LyX objects natively.
 
 I may be missing most of current advancement in programming 
 tools, but I 
 do not see how that can happen. This is doable, but requires 
 a lot of work.
 
 And worse, people will expect that their scripts will 
 continue to work 
 with future versions. This means that we mostly stop code cleanups.

My idea was to issue commands to LyX via LFUNs, which are quite stable, even 
because they're associated with customizable shortcuts. I think this is a very 
uninvasive approach.
For the language I prefer Python because _a lot_ of people uses it and I think 
this is fundamental if we want to hope in third party cooperation.

Andre said:
 Plugin systemis only (or rather, at most) make sense if there's
 a decent chance to have third parties creating plugins that are
 not maintained/distributed with the main product.
 For LyX, assuming a probability of zero for that case seems to be
 a good first approximation.

I agree, if we're talking about plugins in C++, but with Python we have a lot 
more changes.

venom00

Re: Scripting vs plugins (was RE: Goals for 2.1)

[Peter Kümmel]

On 05.05.2011 19:26, venom00 wrote:

Jean Marc said:

The problem with script plugins is that people seem to expect that by
linking LyX to python everybody will be able to write python scritps
that can manipulate LyX objects natively.

I may be missing most of current advancement in programming
tools, but I
do not see how that can happen. This is doable, but requires
a lot of work.

And worse, people will expect that their scripts will
continue to work
with future versions. This means that we mostly stop code cleanups.


My idea was to issue commands to LyX via LFUNs, which are quite stable, even 
because they're associated with customizable shortcuts. I think this is a very 
uninvasive approach.
For the language I prefer Python because _a lot_ of people uses it and I think 
this is fundamental if we want to hope in third party cooperation.


I had a similar idea to record/replay such LFUNs series, especially for 
reproducing bug it could be useful.



Andre said:

Plugin systemis only (or rather, at most) make sense if there's
a decent chance to have third parties creating plugins that are
not maintained/distributed with the main product.
For LyX, assuming a probability of zero for that case seems to be
a good first approximation.


I agree, if we're talking about plugins in C++, but with Python we have a lot 
more changes.

venom00

RE: Scripting vs plugins (was RE: Goals for 2.1)

[venom00]

  My idea was to issue commands to LyX via LFUNs, which are 
 quite stable, even because they're associated with 
 customizable shortcuts. I think this is a very uninvasive approach.
  For the language I prefer Python because _a lot_ of people 
 uses it and I think this is fundamental if we want to hope in 
 third party cooperation.
 
 I had a similar idea to record/replay such LFUNs series, 
 especially for reproducing bug it could be useful.

Sure, scripting and macro are strictly related. Record a macro would be very 
simple as each user interaction with LyX goes through a LyX user command. We 
need several new LFUNs to have a useful and complete scripting envinronment but 
I think that's the way to follow.

venom00

Scripting vs plugins (was RE: Goals for 2.1)

[venom00]

Jean Marc said:
> The problem with script plugins is that people seem to expect that by 
> linking LyX to python everybody will be able to write python scritps 
> that can manipulate LyX objects natively.
> 
> I may be missing most of current advancement in programming 
> tools, but I 
> do not see how that can happen. This is doable, but requires 
> a lot of work.
> 
> And worse, people will expect that their scripts will 
> continue to work 
> with future versions. This means that we mostly stop code cleanups.

My idea was to issue commands to LyX via LFUNs, which are quite stable, even 
because they're associated with customizable shortcuts. I think this is a very 
uninvasive approach.
For the language I prefer Python because _a lot_ of people uses it and I think 
this is fundamental if we want to hope in third party cooperation.

Andre said:
> Plugin systemis only (or rather, "at most") make sense if there's
> a decent chance to have third parties creating plugins that are
> not maintained/distributed with the main product.
> For LyX, assuming a probability of zero for that case seems to be
> a good first approximation.

I agree, if we're talking about plugins in C++, but with Python we have a lot 
more changes.

venom00

Re: Scripting vs plugins (was RE: Goals for 2.1)

[Peter Kümmel]

On 05.05.2011 19:26, venom00 wrote:

Jean Marc said:

The problem with script plugins is that people seem to expect that by
linking LyX to python everybody will be able to write python scritps
that can manipulate LyX objects natively.

I may be missing most of current advancement in programming
tools, but I
do not see how that can happen. This is doable, but requires
a lot of work.

And worse, people will expect that their scripts will
continue to work
with future versions. This means that we mostly stop code cleanups.


My idea was to issue commands to LyX via LFUNs, which are quite stable, even 
because they're associated with customizable shortcuts. I think this is a very 
uninvasive approach.
For the language I prefer Python because _a lot_ of people uses it and I think 
this is fundamental if we want to hope in third party cooperation.


I had a similar idea to record/replay such LFUNs series, especially for 
reproducing bug it could be useful.



Andre said:

Plugin systemis only (or rather, "at most") make sense if there's
a decent chance to have third parties creating plugins that are
not maintained/distributed with the main product.
For LyX, assuming a probability of zero for that case seems to be
a good first approximation.


I agree, if we're talking about plugins in C++, but with Python we have a lot 
more changes.

venom00

RE: Scripting vs plugins (was RE: Goals for 2.1)

[venom00]

> > My idea was to issue commands to LyX via LFUNs, which are 
> quite stable, even because they're associated with 
> customizable shortcuts. I think this is a very uninvasive approach.
> > For the language I prefer Python because _a lot_ of people 
> uses it and I think this is fundamental if we want to hope in 
> third party cooperation.
> 
> I had a similar idea to record/replay such LFUNs series, 
> especially for reproducing bug it could be useful.

Sure, scripting and macro are strictly related. Record a macro would be very 
simple as each user interaction with LyX goes through a LyX user command. We 
need several new LFUNs to have a useful and complete scripting envinronment but 
I think that's the way to follow.

venom00

RE: Python scripting

[venom00]

 to give you wider overview, i have been asking about must-have new features
for 2.0 each month from early spring and formally closed the gates by beta1, so
little bit late...

I always arrive too late in this sort of things :D Anyway it's not too urgent,
maybe it could be ready for the next release.

 who told you? iirc adv search and adv math macros appeared this way...

I read somewhere on the wiki that new contributors have to stay in a corner
fixing bugs for a pair of years before they can implement new features. :D

@Guenter: your work is interesting, and will probably be helpful!

So, my idea was to add python scripting in-process. The script should be able to
register to an event, call LFUN and receive responses, nothing more. Maybe we'll
need to add some LFUN to get the selected text, or the whole document source,
because I couldn't find such functions in LyXAction.cpp [1]. I don't think
there's the need to wrap commands to a more OO structure, they change too
frequently IMHO, and the Python library wouldn't be up-to-date.
Events should be things like specific key-press and click on a custom toolbar
button, but these are things one could achieve just with an LFUN (e.g. associate
Ctrl + Shift + R to execute-python-command-from-scripts nameOfTheFunction): the
things that would require massive modifications in the code is signaling
events like startup, quit, new document loaded, document closed and things like
that.
What do you think?

Ale

[1] http://www.lyx.org/trac/browser/lyx-devel/trunk/src/LyXAction.cpp

Re: Python scripting

[Pavel Sanda]

veno...@arcadiaclub.com wrote:
  who told you? iirc adv search and adv math macros appeared this way...
 
 I read somewhere on the wiki that new contributors have to stay in a corner
 fixing bugs for a pair of years before they can implement new features. :D

can't remember such kind of rules. we just suggest that people hang around
for a while with small things to get idea about the code, otherwise big
parts of their code would need rewrite...

 So, my idea was to add python scripting in-process. The script should be able 
 to
 register to an event, call LFUN and receive responses, nothing more. Maybe 
 we'll

 things that would require massive modifications in the code is signaling
 events like startup, quit, new document loaded, document closed and things 
 like
 that.

it means untrivial ditching into the dispatch mechanism, so this is for sure not
2.0 business.

pavel

RE: Python scripting

[venom00]

 it means untrivial ditching into the dispatch mechanism, so this is for sure
not
 2.0 business.

OK, then I'll just start thinking about it. Maybe a first release of the
scripting support could not have such events, but just keypresses as
notification in LyX server.
I don't know how these things work, do you consider this a so big feature to
require a tag, or a branch?

BTW I'm reading the following docs to better understand and edit LyX code:
http://www.lyx.org/trac/browser/lyx-devel/trunk/development/coding/Rules
http://www.lyx.org/trac/browser/lyx-devel/trunk/development/coding/Recommendatio
ns

Any other useful link?

Ale

RE: Python scripting

[venom00]

> to give you wider overview, i have been asking about must-have new features
for 2.0 each month from early spring and formally closed the gates by beta1, so
little bit late...

I always arrive too late in this sort of things :D Anyway it's not too urgent,
maybe it could be ready for the next release.

> who told you? iirc adv search and adv math macros appeared this way...

I read somewhere on the wiki that new contributors have to stay in a corner
fixing bugs for a pair of years before they can implement new features. :D

@Guenter: your work is interesting, and will probably be helpful!

So, my idea was to add python scripting in-process. The script should be able to
register to an event, call LFUN and receive responses, nothing more. Maybe we'll
need to add some LFUN to get the selected text, or the whole document source,
because I couldn't find such functions in LyXAction.cpp [1]. I don't think
there's the need to wrap commands to a more OO structure, they change too
frequently IMHO, and the Python library wouldn't be up-to-date.
Events should be things like specific key-press and click on a custom toolbar
button, but these are things one could achieve just with an LFUN (e.g. associate
Ctrl + Shift + R to execute-python-command-from-scripts nameOfTheFunction): the
things that would require "massive" modifications in the code is signaling
events like startup, quit, new document loaded, document closed and things like
that.
What do you think?

Ale

[1] http://www.lyx.org/trac/browser/lyx-devel/trunk/src/LyXAction.cpp

Re: Python scripting

[Pavel Sanda]

veno...@arcadiaclub.com wrote:
> > who told you? iirc adv search and adv math macros appeared this way...
> 
> I read somewhere on the wiki that new contributors have to stay in a corner
> fixing bugs for a pair of years before they can implement new features. :D

can't remember such kind of rules. we just suggest that people hang around
for a while with small things to get idea about the code, otherwise big
parts of their code would need rewrite...

> So, my idea was to add python scripting in-process. The script should be able 
> to
> register to an event, call LFUN and receive responses, nothing more. Maybe 
> we'll

> things that would require "massive" modifications in the code is signaling
> events like startup, quit, new document loaded, document closed and things 
> like
> that.

it means untrivial ditching into the dispatch mechanism, so this is for sure not
2.0 business.

pavel

RE: Python scripting

[venom00]

> it means untrivial ditching into the dispatch mechanism, so this is for sure
not
> 2.0 business.

OK, then I'll just start thinking about it. Maybe a first release of the
scripting support could not have such events, but just keypresses as
notification in LyX server.
I don't know how these things work, do you consider this a so big feature to
require a tag, or a branch?

BTW I'm reading the following docs to better understand and edit LyX code:
http://www.lyx.org/trac/browser/lyx-devel/trunk/development/coding/Rules
http://www.lyx.org/trac/browser/lyx-devel/trunk/development/coding/Recommendatio
ns

Any other useful link?

Ale

Python scripting

[venom00]

Hello, LyX 2.0 is getting awesome, but I think there's just a last must-have 
feature missing: support for scripting. I think implementing Python scripting 
[1] wouldn't be that difficult, as all the functionalities (event notification, 
commands, response and errors) are already implemented in the LyX server [2].
Maybe I can take care of this, if you're interested, even if I know new users 
shouldn't try to implemnt new features, but I've been walking around LyX code 
for a while and recently (today) I wrote a patch 
(http://www.lyx.org/trac/ticket/7042#comment:7).
I hope in your support because this is probably one of those features which 
could attract a lot of interest in LyX. Imagine that one can be able to write a 
quick plugin in Python to use a specific LaTeX package? Wouldn't that be 
interesting?

Ale

[1] http://docs.python.org/extending/embedding.html
[2] http://wiki.lyx.org/LyX/LyXServer

Re: Python scripting

[Richard Heck]

On 11/26/2010 02:22 PM, veno...@arcadiaclub.com wrote:

Hello, LyX 2.0 is getting awesome, but I think there's just a last must-have 
feature missing: support for scripting. I think implementing Python scripting 
[1] wouldn't be that difficult, as all the functionalities (event notification, 
commands, response and errors) are already implemented in the LyX server [2].
Maybe I can take care of this, if you're interested, even if I know new users 
shouldn't try to implemnt new features, but I've been walking around LyX code 
for a while and recently (today) I wrote a patch 
(http://www.lyx.org/trac/ticket/7042#comment:7).
I hope in your support because this is probably one of those features which 
could attract a lot of interest in LyX. Imagine that one can be able to write a 
quick plugin in Python to use a specific LaTeX package? Wouldn't that be 
interesting?

Whether such a thing could make it into 2.0, I'm doubtful, as trunk is 
pretty much closed to enhancements at this point (though this is 
ultimately up to Pavel). Still, this kind of change is not a format 
change, i.e., it doesn't affect file structure, so it could in 
principle make it into the 2.0.x series.


That said, I'm not entirely sure what you want to do here. What sort of 
support for scripting would you like to have that can't be implemented 
though the existing LyX server?


I'm intending to look at your other patch, by the way, but am busy with 
work right now. I don't use PSTricks, etc, though, so the testing part I 
can't do.


Richard

Re: Python scripting

[Guenter Milde]

On 2010-11-26, veno...@arcadiaclub.com wrote:

 Hello, LyX 2.0 is getting awesome, but I think there's just a last
 must-have feature missing: support for scripting.

I wrote a Python package that provides an interface via the LyXserver
pipes. It's available for download via the LyX wiki
http://wiki.lyx.org/Tools/PyClient

Maybe you can build on this?

Günter

Re: Python scripting

[Pavel Sanda]

veno...@arcadiaclub.com wrote:
 Hello, LyX 2.0 is getting awesome, but I think there's just a last must-have 
 feature missing: support for scripting.

to give you wider overview, i have been asking about must-have new features for 
2.0 each month from early spring and formally closed the gates by beta1, so 
little bit late...

 I think implementing Python scripting [1] wouldn't be that difficult, as all 
 the functionalities (event notification, commands, response and errors) are 
 already implemented in the LyX server [2].

hard to say unless one sees the code. there is often quite a gap between 
something which works and something which is accepted as a sane code included 
into the tree, lets see the other patch ;)

 Maybe I can take care of this, if you're interested,

this is a long time request for which you could even find bug number, so even 
if this is not part of 2.0 it has good potential to be included later.
as a first step i would recommend to discuss your vision of 'scripting support' 
here on the list.

even if I know new users shouldn't try to implemnt new features,

who told you? iirc adv search and adv math macros appeared this way...

 I hope in your support because this is probably one of those features which 
 could attract a lot of interest in LyX.

as said this has pretty good chance to get support, but for 2.0 i would like 
that our effort now goes mainly on fixing bugs.

and dont get me wrong, you are welcomed here! not so often somebody asks for 
something and provides patch ;)
pavel

Python scripting

[venom00]

Hello, LyX 2.0 is getting awesome, but I think there's just a last must-have 
feature missing: support for scripting. I think implementing Python scripting 
[1] wouldn't be that difficult, as all the functionalities (event notification, 
commands, response and errors) are already implemented in the LyX server [2].
Maybe I can take care of this, if you're interested, even if I know new users 
shouldn't try to implemnt new features, but I've been walking around LyX code 
for a while and recently (today) I wrote a patch 
(http://www.lyx.org/trac/ticket/7042#comment:7).
I hope in your support because this is probably one of those features which 
could attract a lot of interest in LyX. Imagine that one can be able to write a 
quick plugin in Python to use a specific LaTeX package? Wouldn't that be 
interesting?

Ale

[1] http://docs.python.org/extending/embedding.html
[2] http://wiki.lyx.org/LyX/LyXServer

Re: Python scripting

[Richard Heck]

On 11/26/2010 02:22 PM, veno...@arcadiaclub.com wrote:

Hello, LyX 2.0 is getting awesome, but I think there's just a last must-have 
feature missing: support for scripting. I think implementing Python scripting 
[1] wouldn't be that difficult, as all the functionalities (event notification, 
commands, response and errors) are already implemented in the LyX server [2].
Maybe I can take care of this, if you're interested, even if I know new users 
shouldn't try to implemnt new features, but I've been walking around LyX code 
for a while and recently (today) I wrote a patch 
(http://www.lyx.org/trac/ticket/7042#comment:7).
I hope in your support because this is probably one of those features which 
could attract a lot of interest in LyX. Imagine that one can be able to write a 
quick plugin in Python to use a specific LaTeX package? Wouldn't that be 
interesting?

Whether such a thing could make it into 2.0, I'm doubtful, as trunk is 
pretty much closed to enhancements at this point (though this is 
ultimately up to Pavel). Still, this kind of change is not a "format 
change", i.e., it doesn't affect file structure, so it could in 
principle make it into the 2.0.x series.


That said, I'm not entirely sure what you want to do here. What sort of 
support for scripting would you like to have that can't be implemented 
though the existing LyX server?


I'm intending to look at your other patch, by the way, but am busy with 
work right now. I don't use PSTricks, etc, though, so the testing part I 
can't do.


Richard

Re: Python scripting

[Guenter Milde]

On 2010-11-26, <veno...@arcadiaclub.com> wrote:

> Hello, LyX 2.0 is getting awesome, but I think there's just a last
> must-have feature missing: support for scripting.

I wrote a Python package that provides an interface via the LyXserver
pipes. It's available for download via the LyX wiki
http://wiki.lyx.org/Tools/PyClient

Maybe you can build on this?

Günter

Re: Python scripting

[Pavel Sanda]

veno...@arcadiaclub.com wrote:
> Hello, LyX 2.0 is getting awesome, but I think there's just a last must-have 
> feature missing: support for scripting.

to give you wider overview, i have been asking about must-have new features for 
2.0 each month from early spring and formally closed the gates by beta1, so 
little bit late...

> I think implementing Python scripting [1] wouldn't be that difficult, as all 
> the functionalities (event notification, commands, response and errors) are 
> already implemented in the LyX server [2].

hard to say unless one sees the code. there is often quite a gap between 
"something which works" and something which is accepted as a sane code included 
into the tree, lets see the other patch ;)

> Maybe I can take care of this, if you're interested,

this is a long time request for which you could even find bug number, so even 
if this is not part of 2.0 it has good potential to be included later.
as a first step i would recommend to discuss your vision of 'scripting support' 
here on the list.

>even if I know new users shouldn't try to implemnt new features,

who told you? iirc adv search and adv math macros appeared this way...

> I hope in your support because this is probably one of those features which 
> could attract a lot of interest in LyX.

as said this has pretty good chance to get support, but for 2.0 i would like 
that our effort now goes mainly on fixing bugs.

and dont get me wrong, you are welcomed here! not so often somebody asks for 
something and provides patch ;)
pavel

RE: Scripting fun - removing unneeded headers from source code

[Sam Liddicott]

I think only a reasoned approach will produce a result that compils and makes 
sense.

A file my compile because a header it needs is wrongly included in a header it 
uses.

Each header fle must be exained in conjunction with an accepted coding policy 
to see if the headers it includes can be justified.

Then the cpp files can be examined to see if the intended functionailty they 
import is also justified.

This process answers the question: Does the code express rational intent? 
rather than the question: dos it compile faster?

Sam



-Original Message-
From: Steve Litt sl...@troubleshooters.com
Sent: 07 November 2009 19:19
To: LyX devel lyx-devel@lists.lyx.org
Subject: Re: Scripting fun - removing unneeded headers from source code


On Saturday 07 November 2009 08:37:30 Pavel Sanda wrote:
 hi,

 if there is somebody who would like to contribute to lyx and like to tackle
 with some python/bash scripting, consider fixing the bug
 http://www.lyx.org/trac/ticket/6305 .

 pavel


Hi Pavel,

The part where you remove .h includes from .cpp  is dead bang easy if rather 
slow. Not so with the .h includes inside other .h files. For each #include 
removal from a .h file, you'd have to compile every .cpp that includes the 
including .h.

1.5.4 has 539 .h files and 463 .cpp files. That's roughly a quarter million 
combinations. Multiply that by each #include within each .h file. Ugh! Check 
this out:

sl...@mydesk:~/junk/lyx-1.5.4$ find . | grep \.h$ | xargs grep \#include | wc
   16943604   90852
sl...@mydesk:~/junk/lyx-1.5.4$

On the positive side, short circuit logic means that on non-extraneous 
includes, the error will probably appear before all .cpp files have been 
compiled.

Also, I'm pretty sure that some #includes aren't necessary for compile, but 
WILL eliminate some warnings. So the test would need to fail if any output 
occurs on the gcc -Wall command.

I'm wondering if there's a less compute intensive method than the brute force 
method of removing a #include from a .h, and then recompiling every .cpp file.

SteveT

Steve Litt
Recession Relief Package
http://www.recession-relief.US
Twitter: http://www.twitter.com/stevelitt

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

John McCabe-Dansted wrote:
 I have heard that some .h files include other .h files, but are not
 required to do so by POSIX etc. Thus removing them may harm
 portability. I am not sure how this could be detected automatically

we have to be careful about few headers like cstdlib. then i remember
one include needed in our sources which was not needed for compilation,
but for proper run. if only i can remember which one was it... i rememeber
some weird bug, which JMarc revealed as missing include, probably from
support or so?

other things will be quickly tested by people around, no automatical
magic needed.

pavel

Re: Scripting fun - removing unneeded headers from source code

[José Matos]

On Saturday 07 November 2009 13:37:30 Pavel Sanda wrote:
 hi,
 
 if there is somebody who would like to contribute to lyx and like to tackle
  with some python/bash scripting, consider fixing the bug
  http://www.lyx.org/trac/ticket/6305 .
 
 pavel

Angus has tackled this task years ago. Some lessons from is job,

a) as John McCabe-Dansted stated for portability reasons you should run this 
on the different platforms we support or else the compilation will run nicely 
in linux and it will fail on windows or on mac or on solaris... (rinse and 
repeat for other combinations).

b) sometimes the order of removal matters because of the multiple dependencies 
between header files.

c) this problem is more general than just lyx, it would be nice to see what 
other are doing on this issue.

Regards,
-- 
José Abílio

Re: Scripting fun - removing unneeded headers from source code

[Andre Poenitz]

On Sat, Nov 07, 2009 at 11:12:06PM +0100, Alex Fernandez wrote:
 On Sat, Nov 7, 2009 at 8:19 PM, Steve Litt sl...@troubleshooters.com wrote:
  I'm wondering if there's a less compute intensive method than the brute 
  force
  method of removing a #include from a .h, and then recompiling every .cpp 
  file.
 
 I thought about semantic processing: read all the prototypes in the
 .h, then see if they are actually used in the .cpp files. Remove those
 which aren't and try compiling; if the job has been well done then
 compilation should proceed without a hitch.
 
 Depending on how convoluted the LyX code actually is (things like
 #defines might screw any semantic processing) it might work reasonably
 well.

It looks like a job for a somewhat more generic enviroment like an IDE
that already has a preprocessor or model of the code than for an
application specific script...

Andre'

Re: Scripting fun - removing unneeded headers from source code

[Andre Poenitz]

On Sun, Nov 08, 2009 at 12:55:15AM +0100, Vincent van Ravesteijn wrote:
 Pavel Sanda schreef:
 Vincent van Ravesteijn wrote:
   
 my draft implied one compilation per one #include in our sources, no
 combinations. the only tweaking part was that detection in .h files 
 - one has
 to distinguish whether the compilation fails because of header  
 insuficiency in
 .h or in consequent .cpp which includes it.

 pavel
 
 Yes, but if you include ten  .h files, and each one includes ten 
 other .h files, then it is very difficult to find the least amount of 
 includes that are needed. And in the end, it might not even have any 
 influence on building times as one .h file is often included through 
 some detours.
 

 i didn't claim we will find optimal solution. but if this script is going to
 happen i believe it will speed up the compilation.

 pavel

 I think you'll mostly remove includes that are redundant in some sense.  
 It's less likely there are huge amounts of costly includes.

A while ago there was quite some effort to remove unnecessary includes.
I don't think that there was much avoidable left at that time.

The next step (that was also partially taken) was to avoid stuff that
was really expensive to use. There's still  development/tools/numbers
that gives a rough idea what the costs of individual #includes are.

Andre'

Re: Scripting fun - removing unneeded headers from source code

[Andre Poenitz]

On Sun, Nov 08, 2009 at 11:19:55AM +0800, John McCabe-Dansted wrote:
 I have heard that some .h files include other .h files, but are not
 required to do so by POSIX etc. Thus removing them may harm
 portability.

This discussion is not much about C headers. They are comparatively
short and above that quicker to process than C++ headers. 

Andre'

RE: Scripting fun - removing unneeded headers from source code

[Sam Liddicott]

I think only a reasoned approach will produce a result that compils and makes 
sense.

A file my compile because a header it needs is wrongly included in a header it 
uses.

Each header fle must be exained in conjunction with an accepted coding policy 
to see if the headers it includes can be justified.

Then the cpp files can be examined to see if the intended functionailty they 
import is also justified.

This process answers the question: Does the code express rational intent? 
rather than the question: dos it compile faster?

Sam



-Original Message-
From: Steve Litt <sl...@troubleshooters.com>
Sent: 07 November 2009 19:19
To: LyX devel <lyx-devel@lists.lyx.org>
Subject: Re: Scripting fun - removing unneeded headers from source code


On Saturday 07 November 2009 08:37:30 Pavel Sanda wrote:
> hi,
>
> if there is somebody who would like to contribute to lyx and like to tackle
> with some python/bash scripting, consider fixing the bug
> http://www.lyx.org/trac/ticket/6305 .
>
> pavel


Hi Pavel,

The part where you remove .h includes from .cpp  is dead bang easy if rather 
slow. Not so with the .h includes inside other .h files. For each #include 
removal from a .h file, you'd have to compile every .cpp that includes the 
including .h.

1.5.4 has 539 .h files and 463 .cpp files. That's roughly a quarter million 
combinations. Multiply that by each #include within each .h file. Ugh! Check 
this out:

sl...@mydesk:~/junk/lyx-1.5.4$ find . | grep \.h$ | xargs grep \#include | wc
   16943604   90852
sl...@mydesk:~/junk/lyx-1.5.4$

On the positive side, short circuit logic means that on non-extraneous 
includes, the error will probably appear before all .cpp files have been 
compiled.

Also, I'm pretty sure that some #includes aren't necessary for compile, but 
WILL eliminate some warnings. So the test would need to fail if any output 
occurs on the gcc -Wall command.

I'm wondering if there's a less compute intensive method than the brute force 
method of removing a #include from a .h, and then recompiling every .cpp file.

SteveT

Steve Litt
Recession Relief Package
http://www.recession-relief.US
Twitter: http://www.twitter.com/stevelitt

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

John McCabe-Dansted wrote:
> I have heard that some .h files include other .h files, but are not
> required to do so by POSIX etc. Thus removing them may harm
> portability. I am not sure how this could be detected automatically

we have to be careful about few headers like cstdlib. then i remember
one include needed in our sources which was not needed for compilation,
but for proper run. if only i can remember which one was it... i rememeber
some weird bug, which JMarc revealed as missing include, probably from
support or so?

other things will be quickly tested by people around, no automatical
magic needed.

pavel

Re: Scripting fun - removing unneeded headers from source code

[José Matos]

On Saturday 07 November 2009 13:37:30 Pavel Sanda wrote:
> hi,
> 
> if there is somebody who would like to contribute to lyx and like to tackle
>  with some python/bash scripting, consider fixing the bug
>  http://www.lyx.org/trac/ticket/6305 .
> 
> pavel

Angus has tackled this task years ago. Some lessons from is job,

a) as John McCabe-Dansted stated for portability reasons you should run this 
on the different platforms we support or else the compilation will run nicely 
in linux and it will fail on windows or on mac or on solaris... (rinse and 
repeat for other combinations).

b) sometimes the order of removal matters because of the multiple dependencies 
between header files.

c) this problem is more general than just lyx, it would be nice to see what 
other are doing on this issue.

Regards,
-- 
José Abílio

Re: Scripting fun - removing unneeded headers from source code

[Andre Poenitz]

On Sat, Nov 07, 2009 at 11:12:06PM +0100, Alex Fernandez wrote:
> On Sat, Nov 7, 2009 at 8:19 PM, Steve Litt  wrote:
> > I'm wondering if there's a less compute intensive method than the brute 
> > force
> > method of removing a #include from a .h, and then recompiling every .cpp 
> > file.
> 
> I thought about semantic processing: read all the prototypes in the
> .h, then see if they are actually used in the .cpp files. Remove those
> which aren't and try compiling; if the job has been well done then
> compilation should proceed without a hitch.
> 
> Depending on how convoluted the LyX code actually is (things like
> #defines might screw any semantic processing) it might work reasonably
> well.

It looks like a job for a somewhat more generic enviroment like an IDE
that already has a preprocessor or "model" of the code than for an
application specific script...

Andre'

Re: Scripting fun - removing unneeded headers from source code

[Andre Poenitz]

On Sun, Nov 08, 2009 at 12:55:15AM +0100, Vincent van Ravesteijn wrote:
> Pavel Sanda schreef:
>> Vincent van Ravesteijn wrote:
>>   
 my draft implied one compilation per one #include in our sources, no
 combinations. the only tweaking part was that detection in .h files 
 - one has
 to distinguish whether the compilation fails because of header  
 insuficiency in
 .h or in consequent .cpp which includes it.

 pavel
 
>>> Yes, but if you include ten  .h files, and each one includes ten 
>>> other .h files, then it is very difficult to find the least amount of 
>>> includes that are needed. And in the end, it might not even have any 
>>> influence on building times as one .h file is often included through 
>>> some detours.
>>> 
>>
>> i didn't claim we will find optimal solution. but if this script is going to
>> happen i believe it will speed up the compilation.
>>
>> pavel
>
> I think you'll mostly remove includes that are redundant in some sense.  
> It's less likely there are huge amounts of costly includes.

A while ago there was quite some effort to remove unnecessary includes.
I don't think that there was much avoidable left at that time.

The next step (that was also partially taken) was to avoid stuff that
was really expensive to use. There's still  development/tools/numbers
that gives a rough idea what the costs of individual #includes are.

Andre'

Re: Scripting fun - removing unneeded headers from source code

[Andre Poenitz]

On Sun, Nov 08, 2009 at 11:19:55AM +0800, John McCabe-Dansted wrote:
> I have heard that some .h files include other .h files, but are not
> required to do so by POSIX etc. Thus removing them may harm
> portability.

This discussion is not much about C headers. They are comparatively
short and above that quicker to process than C++ headers. 

Andre'

Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

hi,

if there is somebody who would like to contribute to lyx and like to tackle 
with some
python/bash scripting, consider fixing the bug 
http://www.lyx.org/trac/ticket/6305 .

pavel

Re: Scripting fun - removing unneeded headers from source code

[Steve Litt]

On Saturday 07 November 2009 08:37:30 Pavel Sanda wrote:
 hi,

 if there is somebody who would like to contribute to lyx and like to tackle
 with some python/bash scripting, consider fixing the bug
 http://www.lyx.org/trac/ticket/6305 .

 pavel


Hi Pavel,

The part where you remove .h includes from .cpp  is dead bang easy if rather 
slow. Not so with the .h includes inside other .h files. For each #include 
removal from a .h file, you'd have to compile every .cpp that includes the 
including .h.

1.5.4 has 539 .h files and 463 .cpp files. That's roughly a quarter million 
combinations. Multiply that by each #include within each .h file. Ugh! Check 
this out:

sl...@mydesk:~/junk/lyx-1.5.4$ find . | grep \.h$ | xargs grep \#include | wc
   16943604   90852
sl...@mydesk:~/junk/lyx-1.5.4$

On the positive side, short circuit logic means that on non-extraneous 
includes, the error will probably appear before all .cpp files have been 
compiled.

Also, I'm pretty sure that some #includes aren't necessary for compile, but 
WILL eliminate some warnings. So the test would need to fail if any output 
occurs on the gcc -Wall command.

I'm wondering if there's a less compute intensive method than the brute force 
method of removing a #include from a .h, and then recompiling every .cpp file.

SteveT

Steve Litt
Recession Relief Package
http://www.recession-relief.US
Twitter: http://www.twitter.com/stevelitt

Re: Scripting fun - removing unneeded headers from source code

[Alex Fernandez]

On Sat, Nov 7, 2009 at 8:19 PM, Steve Litt sl...@troubleshooters.com wrote:
 I'm wondering if there's a less compute intensive method than the brute force
 method of removing a #include from a .h, and then recompiling every .cpp file.

I thought about semantic processing: read all the prototypes in the
.h, then see if they are actually used in the .cpp files. Remove those
which aren't and try compiling; if the job has been well done then
compilation should proceed without a hitch.

Depending on how convoluted the LyX code actually is (things like
#defines might screw any semantic processing) it might work reasonably
well.

Alex.

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

Steve Litt wrote:
 The part where you remove .h includes from .cpp  is dead bang easy if rather 
 slow. Not so with the .h includes inside other .h files. For each #include 
 removal from a .h file, you'd have to compile every .cpp that includes the 
 including .h.

my draft implied one compilation per one #include in our sources, no
combinations. the only tweaking part was that detection in .h files - one has
to distinguish whether the compilation fails because of header insuficiency in
.h or in consequent .cpp which includes it.

pavel

Re: Scripting fun - removing unneeded headers from source code

[Vincent van Ravesteijn]

Pavel Sanda schreef:

Steve Litt wrote:
  
The part where you remove .h includes from .cpp  is dead bang easy if rather 
slow. Not so with the .h includes inside other .h files. For each #include 
removal from a .h file, you'd have to compile every .cpp that includes the 
including .h.



my draft implied one compilation per one #include in our sources, no
combinations. the only tweaking part was that detection in .h files - one has
to distinguish whether the compilation fails because of header insuficiency in
.h or in consequent .cpp which includes it.

pavel
  
Yes, but if you include ten  .h files, and each one includes ten other 
.h files, then it is very difficult to find the least amount of includes 
that are needed. And in the end, it might not even have any influence on 
building times as one .h file is often included through some detours.


Vincent

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

Vincent van Ravesteijn wrote:
 my draft implied one compilation per one #include in our sources, no
 combinations. the only tweaking part was that detection in .h files - one 
 has
 to distinguish whether the compilation fails because of header 
 insuficiency in
 .h or in consequent .cpp which includes it.

 pavel
   
 Yes, but if you include ten  .h files, and each one includes ten other .h 
 files, then it is very difficult to find the least amount of includes that 
 are needed. And in the end, it might not even have any influence on 
 building times as one .h file is often included through some detours.

i didn't claim we will find optimal solution. but if this script is going to
happen i believe it will speed up the compilation.

pavel

Re: Scripting fun - removing unneeded headers from source code

[Vincent van Ravesteijn]

Pavel Sanda schreef:

Vincent van Ravesteijn wrote:
  

my draft implied one compilation per one #include in our sources, no
combinations. the only tweaking part was that detection in .h files - one 
has
to distinguish whether the compilation fails because of header 
insuficiency in

.h or in consequent .cpp which includes it.

pavel
  
  
Yes, but if you include ten  .h files, and each one includes ten other .h 
files, then it is very difficult to find the least amount of includes that 
are needed. And in the end, it might not even have any influence on 
building times as one .h file is often included through some detours.



i didn't claim we will find optimal solution. but if this script is going to
happen i believe it will speed up the compilation.

pavel
  


I think you'll mostly remove includes that are redundant in some sense. 
It's less likely there are huge amounts of costly includes.


Vincent

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

Vincent van Ravesteijn wrote:
 I think you'll mostly remove includes that are redundant in some sense. 
 It's less likely there are huge amounts of costly includes.

who knows. i believe that those 13 includes i removed from insetmathhull
today were not just redundant of other headers.

pavel

Re: Scripting fun - removing unneeded headers from source code

[Alex Fernandez]

On Sat, Nov 7, 2009 at 11:12 PM, Alex Fernandez alejandro...@gmail.com wrote:
 I thought about semantic processing: read all the prototypes in the
 .h, then see if they are actually used in the .cpp files. Remove those
 which aren't and try compiling; if the job has been well done then
 compilation should proceed without a hitch.

After some testing, semantic processing is too complex: basically you
have to write a C++ parser in Python.

Alex.

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

Pavel Sanda wrote:
 Vincent van Ravesteijn wrote:
  I think you'll mostly remove includes that are redundant in some sense. 
  It's less likely there are huge amounts of costly includes.
 
 who knows. i believe that those 13 includes i removed from insetmathhull
 today were not just redundant of other headers.

compile time gives 0.1s time difference on 2.7 mhz cpu. not much i agree.
pavel

Re: Scripting fun - removing unneeded headers from source code

[John McCabe-Dansted]

I have heard that some .h files include other .h files, but are not
required to do so by POSIX etc. Thus removing them may harm
portability. I am not sure how this could be detected automatically

That said I don't know much about this. And even if this is a problem,
it presumably would help to know which can be removed on a particular
platform, and use judgement to decide which to actually remove.

On Sun, Nov 8, 2009 at 3:19 AM, Steve Litt sl...@troubleshooters.com wrote:
 The part where you remove .h includes from .cpp  is dead bang easy if rather
 slow. Not so with the .h includes inside other .h files. For each #include
 removal from a .h file, you'd have to compile every .cpp that includes the
 including .h.

 1.5.4 has 539 .h files and 463 .cpp files. That's roughly a quarter million
 combinations. Multiply that by each #include within each .h file. Ugh! Check
 this out:

 sl...@mydesk:~/junk/lyx-1.5.4$ find . | grep \.h$ | xargs grep \#include | wc
   1694    3604   90852
 sl...@mydesk:~/junk/lyx-1.5.4$

 On the positive side, short circuit logic means that on non-extraneous
 includes, the error will probably appear before all .cpp files have been
 compiled.

 Also, I'm pretty sure that some #includes aren't necessary for compile, but
 WILL eliminate some warnings. So the test would need to fail if any output
 occurs on the gcc -Wall command.

 I'm wondering if there's a less compute intensive method than the brute force
 method of removing a #include from a .h, and then recompiling every .cpp file.

 SteveT

 Steve Litt
 Recession Relief Package
 http://www.recession-relief.US
 Twitter: http://www.twitter.com/stevelitt






-- 
John C. McCabe-Dansted

Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

hi,

if there is somebody who would like to contribute to lyx and like to tackle 
with some
python/bash scripting, consider fixing the bug 
http://www.lyx.org/trac/ticket/6305 .

pavel

Re: Scripting fun - removing unneeded headers from source code

[Steve Litt]

On Saturday 07 November 2009 08:37:30 Pavel Sanda wrote:
> hi,
>
> if there is somebody who would like to contribute to lyx and like to tackle
> with some python/bash scripting, consider fixing the bug
> http://www.lyx.org/trac/ticket/6305 .
>
> pavel


Hi Pavel,

The part where you remove .h includes from .cpp  is dead bang easy if rather 
slow. Not so with the .h includes inside other .h files. For each #include 
removal from a .h file, you'd have to compile every .cpp that includes the 
including .h.

1.5.4 has 539 .h files and 463 .cpp files. That's roughly a quarter million 
combinations. Multiply that by each #include within each .h file. Ugh! Check 
this out:

sl...@mydesk:~/junk/lyx-1.5.4$ find . | grep \.h$ | xargs grep \#include | wc
   16943604   90852
sl...@mydesk:~/junk/lyx-1.5.4$

On the positive side, short circuit logic means that on non-extraneous 
includes, the error will probably appear before all .cpp files have been 
compiled.

Also, I'm pretty sure that some #includes aren't necessary for compile, but 
WILL eliminate some warnings. So the test would need to fail if any output 
occurs on the gcc -Wall command.

I'm wondering if there's a less compute intensive method than the brute force 
method of removing a #include from a .h, and then recompiling every .cpp file.

SteveT

Steve Litt
Recession Relief Package
http://www.recession-relief.US
Twitter: http://www.twitter.com/stevelitt

Re: Scripting fun - removing unneeded headers from source code

[Alex Fernandez]

On Sat, Nov 7, 2009 at 8:19 PM, Steve Litt  wrote:
> I'm wondering if there's a less compute intensive method than the brute force
> method of removing a #include from a .h, and then recompiling every .cpp file.

I thought about semantic processing: read all the prototypes in the
.h, then see if they are actually used in the .cpp files. Remove those
which aren't and try compiling; if the job has been well done then
compilation should proceed without a hitch.

Depending on how convoluted the LyX code actually is (things like
#defines might screw any semantic processing) it might work reasonably
well.

Alex.

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

Steve Litt wrote:
> The part where you remove .h includes from .cpp  is dead bang easy if rather 
> slow. Not so with the .h includes inside other .h files. For each #include 
> removal from a .h file, you'd have to compile every .cpp that includes the 
> including .h.

my draft implied one compilation per one #include in our sources, no
combinations. the only tweaking part was that detection in .h files - one has
to distinguish whether the compilation fails because of header insuficiency in
.h or in consequent .cpp which includes it.

pavel

Re: Scripting fun - removing unneeded headers from source code

[Vincent van Ravesteijn]

Pavel Sanda schreef:

Steve Litt wrote:
  
The part where you remove .h includes from .cpp  is dead bang easy if rather 
slow. Not so with the .h includes inside other .h files. For each #include 
removal from a .h file, you'd have to compile every .cpp that includes the 
including .h.



my draft implied one compilation per one #include in our sources, no
combinations. the only tweaking part was that detection in .h files - one has
to distinguish whether the compilation fails because of header insuficiency in
.h or in consequent .cpp which includes it.

pavel
  
Yes, but if you include ten  .h files, and each one includes ten other 
.h files, then it is very difficult to find the least amount of includes 
that are needed. And in the end, it might not even have any influence on 
building times as one .h file is often included through some detours.


Vincent

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

Vincent van Ravesteijn wrote:
>> my draft implied one compilation per one #include in our sources, no
>> combinations. the only tweaking part was that detection in .h files - one 
>> has
>> to distinguish whether the compilation fails because of header 
>> insuficiency in
>> .h or in consequent .cpp which includes it.
>>
>> pavel
>>   
> Yes, but if you include ten  .h files, and each one includes ten other .h 
> files, then it is very difficult to find the least amount of includes that 
> are needed. And in the end, it might not even have any influence on 
> building times as one .h file is often included through some detours.

i didn't claim we will find optimal solution. but if this script is going to
happen i believe it will speed up the compilation.

pavel

Re: Scripting fun - removing unneeded headers from source code

[Vincent van Ravesteijn]

Pavel Sanda schreef:

Vincent van Ravesteijn wrote:
  

my draft implied one compilation per one #include in our sources, no
combinations. the only tweaking part was that detection in .h files - one 
has
to distinguish whether the compilation fails because of header 
insuficiency in

.h or in consequent .cpp which includes it.

pavel
  
  
Yes, but if you include ten  .h files, and each one includes ten other .h 
files, then it is very difficult to find the least amount of includes that 
are needed. And in the end, it might not even have any influence on 
building times as one .h file is often included through some detours.



i didn't claim we will find optimal solution. but if this script is going to
happen i believe it will speed up the compilation.

pavel
  


I think you'll mostly remove includes that are redundant in some sense. 
It's less likely there are huge amounts of costly includes.


Vincent

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

Vincent van Ravesteijn wrote:
> I think you'll mostly remove includes that are redundant in some sense. 
> It's less likely there are huge amounts of costly includes.

who knows. i believe that those 13 includes i removed from insetmathhull
today were not just redundant of other headers.

pavel

Re: Scripting fun - removing unneeded headers from source code

[Alex Fernandez]

On Sat, Nov 7, 2009 at 11:12 PM, Alex Fernandez  wrote:
> I thought about semantic processing: read all the prototypes in the
> .h, then see if they are actually used in the .cpp files. Remove those
> which aren't and try compiling; if the job has been well done then
> compilation should proceed without a hitch.

After some testing, semantic processing is too complex: basically you
have to write a C++ parser in Python.

Alex.

Re: Scripting fun - removing unneeded headers from source code

[Pavel Sanda]

Pavel Sanda wrote:
> Vincent van Ravesteijn wrote:
> > I think you'll mostly remove includes that are redundant in some sense. 
> > It's less likely there are huge amounts of costly includes.
> 
> who knows. i believe that those 13 includes i removed from insetmathhull
> today were not just redundant of other headers.

compile time gives 0.1s time difference on 2.7 mhz cpu. not much i agree.
pavel

Re: Scripting fun - removing unneeded headers from source code

[John McCabe-Dansted]

I have heard that some .h files include other .h files, but are not
required to do so by POSIX etc. Thus removing them may harm
portability. I am not sure how this could be detected automatically

That said I don't know much about this. And even if this is a problem,
it presumably would help to know which can be removed on a particular
platform, and use judgement to decide which to actually remove.

On Sun, Nov 8, 2009 at 3:19 AM, Steve Litt  wrote:
> The part where you remove .h includes from .cpp  is dead bang easy if rather
> slow. Not so with the .h includes inside other .h files. For each #include
> removal from a .h file, you'd have to compile every .cpp that includes the
> including .h.
>
> 1.5.4 has 539 .h files and 463 .cpp files. That's roughly a quarter million
> combinations. Multiply that by each #include within each .h file. Ugh! Check
> this out:
>
> sl...@mydesk:~/junk/lyx-1.5.4$ find . | grep \.h$ | xargs grep \#include | wc
>   1694    3604   90852
> sl...@mydesk:~/junk/lyx-1.5.4$
>
> On the positive side, short circuit logic means that on non-extraneous
> includes, the error will probably appear before all .cpp files have been
> compiled.
>
> Also, I'm pretty sure that some #includes aren't necessary for compile, but
> WILL eliminate some warnings. So the test would need to fail if any output
> occurs on the gcc -Wall command.
>
> I'm wondering if there's a less compute intensive method than the brute force
> method of removing a #include from a .h, and then recompiling every .cpp file.
>
> SteveT
>
> Steve Litt
> Recession Relief Package
> http://www.recession-relief.US
> Twitter: http://www.twitter.com/stevelitt
>
>
>



-- 
John C. McCabe-Dansted

Re: scripting troubles

[Georg Baum]

Am Freitag, 30. Juni 2006 20:21 schrieb Enrico Forestieri:
 What I mean is that I don't think that the renaming work I did can be
 responsible for this one.

IIRC it was more than renaming, otherwise I would not have mentioned it.


Georg

Re: scripting troubles

[Enrico Forestieri]

On Sat, Jul 01, 2006 at 12:01:16PM +0200, Georg Baum wrote:

 Am Freitag, 30. Juni 2006 20:21 schrieb Enrico Forestieri:
  What I mean is that I don't think that the renaming work I did can be
  responsible for this one.
 
 IIRC it was more than renaming, otherwise I would not have mentioned it.

For Win32 I simply activated the checkbutton in preferences which allows
to subvert the configure.py findings about \tex_expects_windows_paths.

For Cygwin I made the path returned by external_path() independent of
the above checkbutton status.

So, you are correct, it was a bit more than renaming, but this bit is
strictly related to the renaming and doesn't touch the path handling
stuff in Win32.

Then, I am also correct stating that it cannot be related to the problem
reported by Edwin.

-- 
Enrico

Re: scripting troubles

[Georg Baum]

Am Freitag, 30. Juni 2006 20:21 schrieb Enrico Forestieri:
> What I mean is that I don't think that the renaming work I did can be
> responsible for this one.

IIRC it was more than renaming, otherwise I would not have mentioned it.


Georg

Re: scripting troubles

[Enrico Forestieri]

On Sat, Jul 01, 2006 at 12:01:16PM +0200, Georg Baum wrote:

> Am Freitag, 30. Juni 2006 20:21 schrieb Enrico Forestieri:
> > What I mean is that I don't think that the renaming work I did can be
> > responsible for this one.
> 
> IIRC it was more than renaming, otherwise I would not have mentioned it.

For Win32 I simply activated the checkbutton in preferences which allows
to subvert the configure.py findings about \tex_expects_windows_paths.

For Cygwin I made the path returned by external_path() independent of
the above checkbutton status.

So, you are correct, it was a bit more than renaming, but this bit is
strictly related to the renaming and doesn't touch the path handling
stuff in Win32.

Then, I am also correct stating that it cannot be related to the problem
reported by Edwin.

-- 
Enrico

scripting troubles

[Edwin Leuven]

atm saved preferences end up in my lyx file!

is this related to your recent changes bo?

here is a snippet:

#LyX 1.5.0svn created this file. For more info see http://www.lyx.org/
\lyxformat 247
\begin_document
\begin_header
\textclass article
\language english
\inputencoding auto
\font_roman default
\font_sans default
\font_typewriter default
\font_default_family default
\font_sc false
\font_osf false
\font_sf_scale 100
\font_tt_scale 100
\graphics default
\paperfontsize default
\papersize default
\use_geometry false
\use_amsmath 1
\cite_engine basic
\use_bibtopic false
\paperorientation portrait
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\defskip medskip
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\end_header

Re: scripting troubles

[Juergen Spitzmueller]

Edwin Leuven wrote:
 here is a snippet:

What's wrong with this?

Jürgen

Re: scripting troubles

[Georg Baum]

Edwin Leuven wrote:

 atm saved preferences end up in my lyx file!

I don't see any preferences there. If you meant the \font... settings: Have
a look at development/FORMAT.


Georg

Re: scripting troubles

[Lars Gullik Bjønnes]

Edwin Leuven [EMAIL PROTECTED] writes:

| atm saved preferences end up in my lyx file!
| 
| is this related to your recent changes bo?
| 
| here is a snippet:
| 
| #LyX 1.5.0svn created this file. For more info see http://www.lyx.org/
| \lyxformat 247
| \begin_document
| \begin_header
| \textclass article
| \language english
| \inputencoding auto
| \font_roman default

This looks like regular document info to me.

What part of this do you see as preference variables?

-- 
Lgb

Re: scripting troubles

[Edwin Leuven]

Juergen Spitzmueller wrote:

Edwin Leuven wrote:

here is a snippet:


What's wrong with this?


mmm, it seems i need another coffee

didn't look close enough

atm i have troubles previewing the document and i somehow connected it 
to changing preferences (but i was wrong)


get an error message:

file does not exist:
c:/tmp/lyx_tmpdir ... /C:_data_newfile8.dvi

it doesn't like the semicolon i guess, probably path conversion going 
wrong somewhere...

Re: scripting troubles

[Georg Baum]

Edwin Leuven wrote:

 atm i have troubles previewing the document and i somehow connected it
 to changing preferences (but i was wrong)
 
 get an error message:
 
 file does not exist:
 c:/tmp/lyx_tmpdir ... /C:_data_newfile8.dvi
 
 it doesn't like the semicolon i guess, probably path conversion going
 wrong somewhere...

Could that be Enricos recent path handling changes?

You could undo them with

svn merge -r 14242:14241


Georg

Re: scripting troubles

[Enrico Forestieri]

On Fri, Jun 30, 2006 at 03:56:37PM +0200, Georg Baum wrote:

 Edwin Leuven wrote:
 
  atm i have troubles previewing the document and i somehow connected it
  to changing preferences (but i was wrong)
  
  get an error message:
  
  file does not exist:
  c:/tmp/lyx_tmpdir ... /C:_data_newfile8.dvi
  
  it doesn't like the semicolon i guess, probably path conversion going
  wrong somewhere...
 
 Could that be Enricos recent path handling changes?

Or, perhaps, the hole in the ozone layer...

-- 
Enrico

Re: scripting troubles

[Enrico Forestieri]

On Fri, Jun 30, 2006 at 06:51:32PM +0200, Enrico Forestieri wrote:

 On Fri, Jun 30, 2006 at 03:56:37PM +0200, Georg Baum wrote:
 
  Edwin Leuven wrote:
  
   atm i have troubles previewing the document and i somehow connected it
   to changing preferences (but i was wrong)
   
   get an error message:
   
   file does not exist:
   c:/tmp/lyx_tmpdir ... /C:_data_newfile8.dvi
   
   it doesn't like the semicolon i guess, probably path conversion going
   wrong somewhere...
  
  Could that be Enricos recent path handling changes?
 
 Or, perhaps, the hole in the ozone layer...

What I mean is that I don't think that the renaming work I did can be
responsible for this one.

-- 
Enrico

scripting troubles

[Edwin Leuven]

atm saved preferences end up in my lyx file!

is this related to your recent changes bo?

here is a snippet:

#LyX 1.5.0svn created this file. For more info see http://www.lyx.org/
\lyxformat 247
\begin_document
\begin_header
\textclass article
\language english
\inputencoding auto
\font_roman default
\font_sans default
\font_typewriter default
\font_default_family default
\font_sc false
\font_osf false
\font_sf_scale 100
\font_tt_scale 100
\graphics default
\paperfontsize default
\papersize default
\use_geometry false
\use_amsmath 1
\cite_engine basic
\use_bibtopic false
\paperorientation portrait
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\defskip medskip
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\end_header

Re: scripting troubles

[Juergen Spitzmueller]

Edwin Leuven wrote:
> here is a snippet:

What's wrong with this?

Jürgen

Re: scripting troubles

[Georg Baum]

Edwin Leuven wrote:

> atm saved preferences end up in my lyx file!

I don't see any preferences there. If you meant the \font... settings: Have
a look at development/FORMAT.


Georg

Re: scripting troubles

[Lars Gullik Bjønnes]

Edwin Leuven <[EMAIL PROTECTED]> writes:

| atm saved preferences end up in my lyx file!
| 
| is this related to your recent changes bo?
| 
| here is a snippet:
| 
| #LyX 1.5.0svn created this file. For more info see http://www.lyx.org/
| \lyxformat 247
| \begin_document
| \begin_header
| \textclass article
| \language english
| \inputencoding auto
| \font_roman default

This looks like regular document info to me.

What part of this do you see as preference variables?

-- 
Lgb

Re: scripting troubles

[Edwin Leuven]

Juergen Spitzmueller wrote:

Edwin Leuven wrote:

here is a snippet:


What's wrong with this?


mmm, it seems i need another coffee

didn't look close enough

atm i have troubles previewing the document and i somehow connected it 
to changing preferences (but i was wrong)


get an error message:

file does not exist:
c:/tmp/lyx_tmpdir ... /C:_data_newfile8.dvi

it doesn't like the semicolon i guess, probably path conversion going 
wrong somewhere...

Re: scripting troubles

[Georg Baum]

Edwin Leuven wrote:

> atm i have troubles previewing the document and i somehow connected it
> to changing preferences (but i was wrong)
> 
> get an error message:
> 
> file does not exist:
> c:/tmp/lyx_tmpdir ... /C:_data_newfile8.dvi
> 
> it doesn't like the semicolon i guess, probably path conversion going
> wrong somewhere...

Could that be Enricos recent path handling changes?

You could undo them with

svn merge -r 14242:14241


Georg