Re: [Scons-dev] Extensions to the Tool subsystem...

[William Deegan]

Evan,

You make some good points.  
As I see it, there are 3 types of SCOns users:
1) open source building software for the world, should build in many different 
environments
2) Hobbyist/individual - Should build on my machine. don't care about any other
3) Commercial - It should build on all the machines at the company (and maybe 
at a partners), but the versions of the tools and the location of the tools 
needs to be tightly controlled such that a user's PATH doesn't cause the build 
system to pick up the "wrong" tool(s).

Even on #1 it may be reasonable/important to use the system install gcc vs 
another version the user has built as it may not work with other system 
installed libraries. (gcc's ABI does change from time to time).

For #1 though if it's important to just work, then the author should import the 
users PATH if it's reasonable to them.

-Bill
On Dec 28, 2012, at 7:32 PM, Evan Driscoll  wrote:

> On 12/20/2012 12:09 PM, William Deegan wrote:
>> On Dec 20, 2012, at 9:51 AM, "Managan, Rob"  wrote:
>>> My question is where does Scons stand these days on the issue of paths and
>>> not using the whole user environment by default?
>> 
>> From my perspective NOT propagating the users environment by default is an 
>> important feature of any sane build system.
>> You want to have a repeatable build. ...
>> 
>> That said, there's always been a (documented) way for developers of build 
>> systems using SCons as their build tool to propagate the users environment 
>> if it's desirable for them.
> 
> I know this email was a bit ago but I thought I'd put in my two cents
> anyway.
> 
> Take the Tar builder for instance. From my perspective, one of two
> things has to be true for a sane build system: either it should be able
> to figure out where that tool lives automatically even if it is located
> in some crazy place (for instance, my main Windows desktop has Cygwin
> installed to p:\programs\cygwin, and Latex is installed to
> p:\programs\miktex), or there needs to be a STANDARD, CONSISTENT way for
> me to tell SCons where it is located.
> 
> [Disclaimer before I continue: I don't think I've ever used Tar(), and
> I'm not sure I've used the Latex tools on Windows, so it's possible
> everything currently just works. I can't test it now. If so: great!]
> 
> 
> On autodetection:
> 
> For some tools like 'tar', it's my opinion that the ONLY correct way to
> autodetect where it is located is to look at the PATH environment
> variable, and if you try any other approach (e.g. look for Cygwin or
> Msys) then you have to provide a standard configuration option. For
> instance, maybe I just want to download the gnuwin32 version which just
> gives you a zip file you extract somewhere.
> 
> So if the tar tool basically looks through the invocation $PATH to find
> the tar executable and, say, uses the full path to it if it's in some
> weird location; that's fine. (One way of looking at this is that the
> PATH variable provides a way
> 
> I'm not sure how many tools this applies too. Maybe it's just simple
> things like 'tar', or maybe it's more complex stuff. I'm not sure.
> 
> 
> On path selection:
> 
> You state that it's possible for the SConstruct writer to import the
> environment -- but I think this is far from sufficient. What happens if
> I want to build some piece of software where the writer DIDN'T do this?
> If it uses .Tar() and my 'tar' is in some location it can't autodetect,
> how can I fix it?
> 
> If the answer is "go edit SConstruct", I think that there's a lot of
> work to be done. IMO, if you want to go this route, the 'tar' tool
> should listen to some 'scons TAR=xxx' command line option.
> 
> Alternatively, there should be a standard, easily-findable place where I
> can tell SCons "here's where my stuff is located!" (for instance to put
> some function call at the top of SConstruct that says "add this to
> env['ENV']['PATH'] for *all* environments you ever make"). It doesn't
> have to be foolproof, just hard to get wrong.
> 
> Sure, maybe the author did things nicely and it's pretty easy to find
> what to edit, but maybe it isn't. I could even see it being somewhat
> obnoxious to find out what file even needs changed, let alone what line
> and how to do it.
> 
> 
> Evan
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> http://two.pairlist.net/mailman/listinfo/scons-dev

___
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Extensions to the Tool subsystem...

[Evan Driscoll]

On 12/20/2012 12:09 PM, William Deegan wrote:
> On Dec 20, 2012, at 9:51 AM, "Managan, Rob"  wrote:
>> My question is where does Scons stand these days on the issue of paths and
>> not using the whole user environment by default?
> 
> From my perspective NOT propagating the users environment by default is an 
> important feature of any sane build system.
> You want to have a repeatable build. ...
> 
> That said, there's always been a (documented) way for developers of build 
> systems using SCons as their build tool to propagate the users environment if 
> it's desirable for them.

I know this email was a bit ago but I thought I'd put in my two cents
anyway.

Take the Tar builder for instance. From my perspective, one of two
things has to be true for a sane build system: either it should be able
to figure out where that tool lives automatically even if it is located
in some crazy place (for instance, my main Windows desktop has Cygwin
installed to p:\programs\cygwin, and Latex is installed to
p:\programs\miktex), or there needs to be a STANDARD, CONSISTENT way for
me to tell SCons where it is located.

[Disclaimer before I continue: I don't think I've ever used Tar(), and
I'm not sure I've used the Latex tools on Windows, so it's possible
everything currently just works. I can't test it now. If so: great!]


On autodetection:

For some tools like 'tar', it's my opinion that the ONLY correct way to
autodetect where it is located is to look at the PATH environment
variable, and if you try any other approach (e.g. look for Cygwin or
Msys) then you have to provide a standard configuration option. For
instance, maybe I just want to download the gnuwin32 version which just
gives you a zip file you extract somewhere.

So if the tar tool basically looks through the invocation $PATH to find
the tar executable and, say, uses the full path to it if it's in some
weird location; that's fine. (One way of looking at this is that the
PATH variable provides a way

I'm not sure how many tools this applies too. Maybe it's just simple
things like 'tar', or maybe it's more complex stuff. I'm not sure.


On path selection:

You state that it's possible for the SConstruct writer to import the
environment -- but I think this is far from sufficient. What happens if
I want to build some piece of software where the writer DIDN'T do this?
If it uses .Tar() and my 'tar' is in some location it can't autodetect,
how can I fix it?

If the answer is "go edit SConstruct", I think that there's a lot of
work to be done. IMO, if you want to go this route, the 'tar' tool
should listen to some 'scons TAR=xxx' command line option.

Alternatively, there should be a standard, easily-findable place where I
can tell SCons "here's where my stuff is located!" (for instance to put
some function call at the top of SConstruct that says "add this to
env['ENV']['PATH'] for *all* environments you ever make"). It doesn't
have to be foolproof, just hard to get wrong.

Sure, maybe the author did things nicely and it's pretty easy to find
what to edit, but maybe it isn't. I could even see it being somewhat
obnoxious to find out what file even needs changed, let alone what line
and how to do it.


Evan
___
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Extensions to the Tool subsystem...

[Gary Oberbrunner]

Yeah, this is a general problem with Windows.  You can't assume "C:\Program
Files (x86)" because of different drive names, and localizations.  MSVC
does a lot of registry lookups so it's not all that hard if it goes in a
standard place like HKLM\Software\.

I think TeXlive just goes in c:\texlive\ though, and doesn't write to
the registry.  So maybe a simple loop through drive letters and years would
do it.


On Thu, Dec 20, 2012 at 1:45 PM, Dirk Bächle  wrote:

> On 20.12.2012 18:51, Managan, Rob wrote:
>
>> [...]
>>
>>
>>
>> Is there a similar set of paths on Windows that we should add to the
>> default? Or is there a place to look for a path when initializing a
>> specific tool that needs a given executable or set of executables?
>>
>
> I'd say that, apart from guessing and probing standard paths like
> "C:\miktex", our only chance is accessing the Windows registry and trying
> to find an entry in the "AppPath" section for the executable in question.
>
> Dirk
>
>
> __**_
> Scons-dev mailing list
> Scons-dev@scons.org
> http://two.pairlist.net/**mailman/listinfo/scons-dev
>



-- 
Gary
___
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Extensions to the Tool subsystem...

[Dirk Bächle]

On 20.12.2012 18:51, Managan, Rob wrote:

[...]


Is there a similar set of paths on Windows that we should add to the
default? Or is there a place to look for a path when initializing a
specific tool that needs a given executable or set of executables?


I'd say that, apart from guessing and probing standard paths like 
"C:\miktex", our only chance is accessing the Windows registry and 
trying to find an entry in the "AppPath" section for the executable in 
question.


Dirk

___
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Extensions to the Tool subsystem...

[William Deegan]

Greetings,


On Dec 20, 2012, at 9:51 AM, "Managan, Rob"  wrote:

> I wanted to weigh in with a path related issue that came up in the LaTeX
> tools.
> 
> My question is where does Scons stand these days on the issue of paths and
> not using the whole user environment by default?


>From my perspective NOT propagating the users environment by default is an 
>important feature of any sane build system.
You want to have a repeatable build. Thus if a users environment can change the 
tools used by the build, you may end up (And I have) spending days trying to 
figure out why user A's build doesn't quite work the way User B's build does 
due to some bug in a specific version of the compiler..

That said, there's always been a (documented) way for developers of build 
systems using SCons as their build tool to propagate the users environment if 
it's desirable for them.


> 
> For Macs, the Latex tools were rarely on the default paths that Scons
> searched and therefore you had to create an Environment that included
> os.environ's path.  This was a real pain since most of the test/TEX/*
> files did not do thatŠ So what I came up with and is in the code base now
> is to add to the path the directories that the system adds to its path for
> installed features like Latex and X11. We did this because Mac OSX has a
> standard file/directory (/etc/paths and /etc/paths.d/*) that lists
> directories that get added to the system path.
> 
> Is there a similar set of paths on Windows that we should add to the
> default? Or is there a place to look for a path when initializing a
> specific tool that needs a given executable or set of executables?

Each tool (assuming it's installed by the tools standard windows installer) has 
a default install path.
If the user installs into a separate path, then in some (most?) cases (when the 
installer doesn't store the install info in the registry) it's o.k. for the 
build to fail to find the tool.
I think the main problem we have now is that we don't do a good job notifying 
the user when we fail to find a tool, especially important if it was explicitly 
requested.

I think this discussion actually has two parts:
1) Should SCons pull paths from the users environment when it is run by 
default? (I believe no)
2) Should the test infrastructure provide a way to specify alternative path in 
general, and for specific tools such that tests machines could have multiple 
versions/competing tools installed and get the test infrastructure to 
include/exclude them from a given test run. (I believe this may resolve your 
issue above)

-Bill

> 
> I think is one issue that needs work as we discuss tool initialization.
> 
> *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
> Rob Managan email managan at llnl.gov
> LLNL phone: 925-423-0903
> P.O. Box 808, L-095   FAX:   925-422-3389
> Livermore, CA  94551-0808
> 
> 
> 
> 
> 
> On 12/20/12 9:09 AM, "Dirk Bächle"  wrote:
> 
>> Hello developers,
>> 
>> based on my proposed changes to the current tests in src/test there has
>> been some discussion about how a Tool should work. Especially in
>> connection with the LaTeX Tool, questions like:
>> 
>>  - Do we want to have one "latex" Tool for all, or separate ones for
>> "miktex", "texlive"...?
>>  - Should a Tool try to find a "Miktex" installation in the current
>> system, or simply search for
>>the "latex" executable while relying on a correct setup of the PATH
>> variable?
>> 
>> showed up.
>> 
> 
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> http://two.pairlist.net/mailman/listinfo/scons-dev

___
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Extensions to the Tool subsystem...

[Managan, Rob]

I wanted to weigh in with a path related issue that came up in the LaTeX
tools.

My question is where does Scons stand these days on the issue of paths and
not using the whole user environment by default?

For Macs, the Latex tools were rarely on the default paths that Scons
searched and therefore you had to create an Environment that included
os.environ's path.  This was a real pain since most of the test/TEX/*
files did not do thatŠ So what I came up with and is in the code base now
is to add to the path the directories that the system adds to its path for
installed features like Latex and X11. We did this because Mac OSX has a
standard file/directory (/etc/paths and /etc/paths.d/*) that lists
directories that get added to the system path.

Is there a similar set of paths on Windows that we should add to the
default? Or is there a place to look for a path when initializing a
specific tool that needs a given executable or set of executables?

I think is one issue that needs work as we discuss tool initialization.

*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
Rob Managan email managan at llnl.gov
LLNL phone: 925-423-0903
P.O. Box 808, L-095   FAX:   925-422-3389
Livermore, CA  94551-0808





On 12/20/12 9:09 AM, "Dirk Bächle"  wrote:

>Hello developers,
>
>based on my proposed changes to the current tests in src/test there has
>been some discussion about how a Tool should work. Especially in
>connection with the LaTeX Tool, questions like:
>
>   - Do we want to have one "latex" Tool for all, or separate ones for
>"miktex", "texlive"...?
>   - Should a Tool try to find a "Miktex" installation in the current
>system, or simply search for
> the "latex" executable while relying on a correct setup of the PATH
>variable?
>
>showed up.
>

___
Scons-dev mailing list
Scons-dev@scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev

[Scons-dev] Extensions to the Tool subsystem...

[Dirk Bächle]

Hello developers,

based on my proposed changes to the current tests in src/test there has 
been some discussion about how a Tool should work. Especially in 
connection with the LaTeX Tool, questions like:


  - Do we want to have one "latex" Tool for all, or separate ones for 
"miktex", "texlive"...?
  - Should a Tool try to find a "Miktex" installation in the current 
system, or simply search for
the "latex" executable while relying on a correct setup of the PATH 
variable?


showed up.

Before simply repeating my pull request of the changes mentioned above, 
I'd like to reach a consensus about some basic guidelines and design 
decisions for the Tool subsystem. I reread the PlatformToolConfig page 
in the Wiki, describing the design of the long-planned IAPAT extension. 
Its requirements are partly taken into account, but I'd like
to keep the "GnuBS-like configure context" out of this discussion. My 
idea is to come up with a plan for our Tools, that hopefully makes it 
easy to add all the "configure" stuff later.


I also want to make a distinction between:

  - defining the Tool (and its supporting classes) as basic class in 
the framework and
  - how we use this Tool module to organize the build/configure/test 
workflows in our

default implementation of SCons.

On the PlatformToolConfig page, the following remark can be found:


(Comment: As I was writing this page, I found myself flipping back and
forth as to whether a Tool module configured a tool (that is, a single
command) or a toolchain (that is, a series of commands). The current Tool
modules actually implement toolchains (e.g., the gcc.py module provides
the environment variables for the compiler, the linker, the static archiver,
the shared archiver, and the bundle archiver). This isn't good
modularization, which suggests that there should be a higher-level
module explicitly for toolchains that can invoke one or more tool
modules as building blocks. That isn't in this proposal (should it be?),
but it's something that should be kept in mind for the future.)


Since we try to provide a framework for build systems, our Tool shouldn't
care. Both should be possible ways for a user to extend the build engine.

With this being said...

What does a Tool do?
=

A Tool has the task of changing an existing Environment. It can alter 
construction variables like CC or the

ENV['PATH'] and is most often used to add Builders or modify existing ones.



What can the Tools subsystem do?
=
(My christmas wishlist ;) for the SCons Tools)

PlatformInfo


I'd like to have a common module for detecting, storing and retrieving 
infos about the current host architecture, (platform, processor type, 
vendor, kernel, OS,...). It should be used throughout the whole source 
tree, including the test framework and the tests themselves.


Toolchains/Tools
##

A Toolchain class should get added, as an abstraction for a series 
(=list) of tools being initialized by a single keyword.

It should be possible to:

+ Check whether we can load/use a toolchain or single tool in our 
current environment (as given by os.environ['PATH']).
+ Check whether we can load/use a toolchain or single tool in a 
special environment.
+ Get a list of possible toolchains for a "Tooltopic" (can't think 
of another name right now). A "Tooltopic" would be "C++", and possible 
toolchains include "mingw" and "msvs". This selection would probably 
depend on the PlatformInfo.
+ Get a default toolchain for a topic, that is actually installed 
in the current system.

+ Dynamically add new toolchains.
+ Dynamically add new tools to a toolchain.
+ Dynamically change the preferred order in which toolchains are tried.
+ Add new Toolchains system-wide, that contain parameterized 
versions of existing tools, e.g. a "cxx-embedded" for a cross-compiling 
gcc that requires special options.


The single Tool support as it is now should still be available and work 
as expected.


External Tools
#

For better support of the external tools we could use:

+ A way to install a Tool in the local SCons distribution and
+ to deinstall it again.
+ SCons should be able to display (--version) that there are 
external tools installed and, on request (--list-external) which Tools 
exactly (and in which version!).

+ For this, single tools should support a version string.


Questions!
===

- How do we want our Tools to be organized for the standard SCons 
implementation? Especially, when we have a Tool like "latex" that 
basically looks the same for all distributions (miktex, texlive, ...) in 
terms of command calls.
  Do we want a "miktex" Tool under the covers, that gets automatically 
selected by the "latex" Toolchain? Should there be a LatexCommon.py in 
cases like this? And how do we go about tests for these toolchains? Do 
we stop using live tests and always provide a fake LaTeX distribution, 
like for the C comp