Re: [Scons-dev] SCons tools refactoring progress

[anatoly techtonik]

On Wed, Oct 21, 2015 at 3:59 AM, Gary Oberbrunner 
wrote:

> Comments inline.
>
> On Tue, Oct 20, 2015 at 12:52 AM, anatoly techtonik 
> wrote:
>
>> On Tue, Oct 20, 2015 at 3:57 AM, Gary Oberbrunner 
>> wrote:
>>>
>>> On Sun, Oct 18, 2015 at 7:52 AM, anatoly techtonik 
>>> wrote:
>>>

 I see the implementation, but I don't see any use cases. I know it
 sounds too formal, but I can't validate the assumptions we had towards the
 new toolchain without a formal list. Do you have some notes or maybe BDD
 tests for that?

>>>
>>> A very good question. I don't have enough design notes in there; I'll
>>> write up some of my motivations and design goals here and add them to the
>>> README.rst.
>>>
>>> The basic idea is that the current concept of a Tool is too low level;
>>> the primary motivating use case is that users (SConscript authors) should
>>> be able to select groups of related tools, _as_ a group, with fallbacks. A
>>> Toolchain is that abstraction.
>>>
>>
>> The use case is already good to be recorded. But there are two cases:
>> 1. Select a group of related tools
>> 2. Fallback
>> Do we have a concrete groups of related tools already?
>>
>
> Yes, I described a few. (Compiler and linker, maybe assembler is the
> obvious one.)
>

By concrete I mean "g++ binary version 4.3" + "gnulink version 4.x" + "as
v4+". The actual toolchains that would be used, for example when SCons
tries to compile Wesnoth on Ubuntu. Like how you'd dump the information
about toolchain into log file.


> The fallback story needs to be expanded. Are there fallback choices inside
>> of one group (with priority or other preference strategy) or the fallback
>> means "fallback to another group"? Maybe I am too detailed, but also - what
>> are the cases when fallbacks occur?
>>
>
> This is all implemented. Check the code and test cases
> (Toolchain-test.py). I think it's pretty solid but please review. Basically
> it's recursive AND and OR trees with optional or required elements.
>

I will try to, but I can't read Python code as well as read plain text.
Before I can look at how it works de-facto I need to get understanding of
how it should work when complete.

Also I try to look at this stuff as an end user, and by looking at the code
I might get diverted from my point by some coding details / implementation
optimizations that I don't want to deal with right now.


> It it not necessary to design everything now upfront - the primary goal of
>> my questions is to find *real world* stories (not even use cases) for which
>> new mechanism is needed. I am afraid to design a system that will be too
>> perfect for implementation in a free time.
>>
>
> Well, at least for the toolchain, I think it's mostly there
> implementation-wise (check it out). The interesting stuff is improving the
> Tool ideas (there's a sort of registry so you can give concrete tools names
> and look them up that way), adding Finders, and all the other stuff that
> we've been discussing. But if you have some real-world stories you'd like
> to see covered, this is a good time to get them out there.
>

Ok. I need a tool to create a shared library. This tool is external to
SCons. How do I install it, specify and use? Current way is:

env.Environment(tools=['someshlibtool'])
env.SomeSharedLib(source)

And I want to get help where this SomeSharedLib comes from, how I can
configure it and what are its options?


> All the rest is just there to make that idea work well. The secondary
>>> goal, in service of the main one, is that Tools need to obey their
>>> abstraction, for instance always calling exists().
>>>
>>
>> What is the story behind this? Because I know the opposite story -
>> exists() for default tools is called even if I don't build anything with
>> those tools - this delays the build start and produces messages about
>> missing compiler to the screen.
>>
>
> exists() is how a tool knows whether its binaries (or whatever it needs to
> run) exist or not, so toolchains can't work without it. As for default
> tools, the idea here is that SConscript writers will (finally!) be able to
> specify exactly which tools they want. I hope to do away with the current
> default tool initialization system, though some of that still needs to be
> thought out. The current design is much "lazier" but still needs work.
>

So what exactly should they need to write so that they use 'gcc' on Windows
to compile env.Program('hello.c')?


> The new system also creates a distinction between an abstract tool, such
>>> as intelc, and a concrete instance of it, such as intelc v12 x86. This is
>>> needed so the user can create chains of either specific or general tools.
>>>
>>
>> I understand where this might be useful, but still - is there a real
>> world story where this was needed?
>>
>
> Absolutely - in my day job we specify very carefully what tool chain is
> used to build 

Re: [Scons-dev] SCons tools refactoring progress

[anatoly techtonik]

On Tue, Oct 20, 2015 at 6:39 PM, Jason Kenny  wrote:
>
> I would hope that the concepts I talked about here are something we can
> find common ground on and talk about defining some common name for. I know
> this is a large e-mail, However I like very much some thoughts about what I
> suggest at high level to form a base of communication for use on the tool
> design.
>
I am for the opposite approach - find place in SCons for components already
present in Parts ASAP. Choose the ones that will be reused anyway. Like if
there is a code that does lookup in path on Linux, registry on Windows etc.
I would like to see a Finder module for that. Maybe even `finder` module
that is independent of SCons.

Then I'd like to have a "tool discovery debugger" that will invoke
SCons/Parts API to just do tool bootstrapping part on tool discovery and
print output to the screen.

Then I'd like to see how do we actually specify the tools. Is it the
current Environment(tools=['list', 'of', 'names']) or there are some use
cases that are not covered by this call. Maybe there are some tales that
tell that this is inconvenient.

And the thing that I feel is needed that for all these new objects -
Platform, ToolInfo etc. it to have static state that could be dumped into
JSON/YAML and restored from. For debugging, caching, for sharing settings,
whatever.
-- 
anatoly t.
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] SCons tools refactoring progress

[anatoly techtonik]

On Thu, Oct 22, 2015 at 6:57 PM, anatoly techtonik 
wrote:

> On Wed, Oct 21, 2015 at 3:48 PM, Bill Deegan 
> wrote:
>
>> I'm thinking that some combination of bugs in the bug tracking system +
>> wiki to capture user stories, etc would be better than an email thread?
>>
>> Here:
>> https://bitbucket.org/scons/scons/wiki/browse/ToolchainRevamp
>>
>
> No BB wiki please. I don't get notifications and don't see history for it.
> Maintaining stuff in a separate repository on Github with free commmit
> policy would be an advantage personally for me.
>

Don't get me wrong - I don't mind if somebody maintains a page on BB. As
long as the page has maintainer and contains latest information, I don't
care what will be used. But tracking progress as diffs is essential.

-- 
anatoly t.
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] SCons tools refactoring progress

[Bill Deegan]

I'm thinking that some combination of bugs in the bug tracking system +
wiki to capture user stories, etc would be better than an email thread?

Here:
https://bitbucket.org/scons/scons/wiki/browse/ToolchainRevamp



On Tue, Oct 20, 2015 at 6:13 PM, Gary Oberbrunner 
wrote:

> You talk a lot about what I consider "policy": specific types of args that
> would be used to configure specific types of Tools, like compilers. This is
> all fine, but my goal is to build a framework where such policies can be
> applied, and then get into the specifics of platforms, cross-compilers,
> versions, targets and all that. Building a game asset manager, or a
> visual-effects pipeline, or a document-preparation system, or a
> scientific-data analysis pipeline, will have very different _specifics_ but
> the same general mechanism should be usable for all, or at least that's my
> goal.
>
> Jason, since you've already been down this path once, I'd very much
> appreciate a code and design review of what I have so far. It's only 700
> lines or so, including tests. I'm especially interested in use cases Parts
> can handle that my proposed system wouldn't be able to. (Again, from a
> mechanism or base framework standpoint; not that I don't specifically
> define a target architecture argument for instance.)
>
> One use case I see from your notes below is this one, which I'll capture:
>   As a SConscript writer, I'd like to be able to enumerate all versions of
> a tool on the system so I can choose which one to use.
> If there are others, please send them along.
>
> -- Gary
>
> On Tue, Oct 20, 2015 at 11:39 AM, Jason Kenny  wrote:
>
>> Hi Gary,
>>
>>
>>
>> From looking at the below this seems to be in line with what is going on
>> in Parts.  I had hoped that what I did here would be used /improved on in
>> Scons. That is the a goal of Parts.. to improve Scons.
>>
>>
>>
>> From what you talk about below I think we are in general on the same
>> page. I know when I started this I was also of the view that the “args”
>> should be external. At the time Steve Knight release pushed hard that all
>> value could be defined in the environment. I see some value in this,
>> however I was like you in which I wanted to have something separate to
>> define arguments to setup a tool. I also wanted to have a standard set of
>> values. For example we should use the same values to talk about archecture,
>> version, etc.. when we can as this makes it easier to use and develop new
>> tools. Given what I have seen in practice, I was moving towards the IATAP
>> idea from Greg. This is what I call Settings in Parts. While I have not
>> finished this object yet I think it goes toward this idea better. Here we
>> have an object that given values creates an environment to be used to build
>> stuff. I think at a high level we want to move this way.
>>
>>
>>
>> For a design of the tools I think these are items we want to understand
>> and define some way.
>>
>>
>>
>> *Platform*- Parts define a system platform object, Scons defines a
>> string, for the OS, it has nothing standard for the architecture. I think
>> we should agree we need to define this better in Scons. I would like to see
>> something in Scons defining this. I suggest what I have in Parts as a way
>> to start. I define in a given environment a HOST_PLATFORM object and a
>> TARGET_PLATFORM Object. This is to extend the PLATFORM string we have now.
>> The way I did in in Parts allows these items to be synced and to extended
>> with new os or archecture value at run time. I think having something like
>> this in SCons will allow for a better more reusable tool design.
>>
>>
>>
>> *Finders*- In Parts I have Finders and Scanners. These are different in
>> what they return, but in effect are the same thing. I believe in the case
>> of what Gary is talking about in Scons they serve the same purpose. The
>> goal is it to return if we found something at a given resource location,
>> given that being a path, registry entries, or environment variable. I
>> should point out in someway this is like a autoconf test to find a
>> compiler, etc. I would like to think this way is better however. I Parts I
>> have finders and scanners. The difference in Parts is that a scanner
>> returns back a set of value with version information, where a finder is
>> much more simple “We have a match as resource X”. The reason for the
>> difference is that some tools are easier to find as a set vs defining every
>> known case. For example when I worked at Intel, the Intel compiler would
>> make lots of internal drops as one would expect, these drops would have
>> different versions. People would want to test the new drops out. Instead of
>> me making a new entry for each new drop, I scanned the disk for example for
>> what was installed. And returned all versions installed on disk. This
>> scaled better for this tool set for users. Another example ( which I should
>> improve a bit more) 

Re: [Scons-dev] SCons tools refactoring progress

[Dirk Baechle]

Hi there, 

just wanted to say that I second Gary in all his points...especially the last 
one. ;)

Dirk

Am 21. Oktober 2015 02:59:02 MESZ, schrieb Gary Oberbrunner 
:
>Comments inline.
>
>On Tue, Oct 20, 2015 at 12:52 AM, anatoly techtonik
>
>wrote:
>
>> On Tue, Oct 20, 2015 at 3:57 AM, Gary Oberbrunner
>
>> wrote:
>>>
>>> On Sun, Oct 18, 2015 at 7:52 AM, anatoly techtonik
>
>>> wrote:
>>>

 I see the implementation, but I don't see any use cases. I know it
 sounds too formal, but I can't validate the assumptions we had
>towards the
 new toolchain without a formal list. Do you have some notes or
>maybe BDD
 tests for that?

>>>
>>> A very good question. I don't have enough design notes in there;
>I'll
>>> write up some of my motivations and design goals here and add them
>to the
>>> README.rst.
>>>
>>> The basic idea is that the current concept of a Tool is too low
>level;
>>> the primary motivating use case is that users (SConscript authors)
>should
>>> be able to select groups of related tools, _as_ a group, with
>fallbacks. A
>>> Toolchain is that abstraction.
>>>
>>
>> The use case is already good to be recorded. But there are two cases:
>> 1. Select a group of related tools
>> 2. Fallback
>> Do we have a concrete groups of related tools already?
>>
>
>Yes, I described a few. (Compiler and linker, maybe assembler is the
>obvious one.)
>
>
>> The fallback story needs to be expanded. Are there fallback choices
>inside
>> of one group (with priority or other preference strategy) or the
>fallback
>> means "fallback to another group"? Maybe I am too detailed, but also
>- what
>> are the cases when fallbacks occur?
>>
>
>This is all implemented. Check the code and test cases
>(Toolchain-test.py).
>I think it's pretty solid but please review. Basically it's recursive
>AND
>and OR trees with optional or required elements.
>
>
>> It it not necessary to design everything now upfront - the primary
>goal of
>> my questions is to find *real world* stories (not even use cases) for
>which
>> new mechanism is needed. I am afraid to design a system that will be
>too
>> perfect for implementation in a free time.
>>
>
>Well, at least for the toolchain, I think it's mostly there
>implementation-wise (check it out). The interesting stuff is improving
>the
>Tool ideas (there's a sort of registry so you can give concrete tools
>names
>and look them up that way), adding Finders, and all the other stuff
>that
>we've been discussing. But if you have some real-world stories you'd
>like
>to see covered, this is a good time to get them out there.
>
>All the rest is just there to make that idea work well. The secondary
>goal,
>>> in service of the main one, is that Tools need to obey their
>abstraction,
>>> for instance always calling exists().
>>>
>>
>> What is the story behind this? Because I know the opposite story -
>> exists() for default tools is called even if I don't build anything
>with
>> those tools - this delays the build start and produces messages about
>> missing compiler to the screen.
>>
>
>exists() is how a tool knows whether its binaries (or whatever it needs
>to
>run) exist or not, so toolchains can't work without it. As for default
>tools, the idea here is that SConscript writers will (finally!) be able
>to
>specify exactly which tools they want. I hope to do away with the
>current
>default tool initialization system, though some of that still needs to
>be
>thought out. The current design is much "lazier" but still needs work.
>
>The new system also creates a distinction between an abstract tool,
>such as
>>> intelc, and a concrete instance of it, such as intelc v12 x86. This
>is
>>> needed so the user can create chains of either specific or general
>tools.
>>>
>>
>> I understand where this might be useful, but still - is there a real
>world
>> story where this was needed?
>>
>
>Absolutely - in my day job we specify very carefully what tool chain is
>used to build any given product. If the machine doesn't have the right
>version of the Intel compiler the build should fail. For open-source
>projects, on the other hand, they should try to build on as many
>configurations as possible so they want to keep things general -- use
>any
>Intel compiler, or any gcc, or any cc (for instance).
>
>
>> One restriction I'm imposing in the new system is that Tools have to
>be
>>> configurable outside of any Environment; I don't like the current
>system
>>> where tool-configuration variables ("tool args" basically) are mixed
>into
>>> the Environment. This poses some challenges for a fully
>generalizable
>>> system but I think I have a decent handle on that. The current Intel
>>> compiler has an initial attempt at such tool args.
>>>
>>
>> How to handle "tools args" is a question for a separate thread. We
>will
>> need to have a standard set for every *type* of tool and specific for
>every
>> tool supported. For 

Re: [Scons-dev] SCons tools refactoring progress

[Jason Kenny]

Hi Gary,

>From looking at the below this seems to be in line with what is going on in 
>Parts.  I had hoped that what I did here would be used /improved on in Scons. 
>That is the a goal of Parts.. to improve Scons.

>From what you talk about below I think we are in general on the same page. I 
>know when I started this I was also of the view that the “args” should be 
>external. At the time Steve Knight release pushed hard that all value could be 
>defined in the environment. I see some value in this, however I was like you 
>in which I wanted to have something separate to define arguments to setup a 
>tool. I also wanted to have a standard set of values. For example we should 
>use the same values to talk about archecture, version, etc.. when we can as 
>this makes it easier to use and develop new tools. Given what I have seen in 
>practice, I was moving towards the IATAP idea from Greg. This is what I call 
>Settings in Parts. While I have not finished this object yet I think it goes 
>toward this idea better. Here we have an object that given values creates an 
>environment to be used to build stuff. I think at a high level we want to move 
>this way.

For a design of the tools I think these are items we want to understand and 
define some way.

Platform- Parts define a system platform object, Scons defines a string, for 
the OS, it has nothing standard for the architecture. I think we should agree 
we need to define this better in Scons. I would like to see something in Scons 
defining this. I suggest what I have in Parts as a way to start. I define in a 
given environment a HOST_PLATFORM object and a TARGET_PLATFORM Object. This is 
to extend the PLATFORM string we have now. The way I did in in Parts allows 
these items to be synced and to extended with new os or archecture value at run 
time. I think having something like this in SCons will allow for a better more 
reusable tool design.

Finders- In Parts I have Finders and Scanners. These are different in what they 
return, but in effect are the same thing. I believe in the case of what Gary is 
talking about in Scons they serve the same purpose. The goal is it to return if 
we found something at a given resource location, given that being a path, 
registry entries, or environment variable. I should point out in someway this 
is like a autoconf test to find a compiler, etc. I would like to think this way 
is better however. I Parts I have finders and scanners. The difference in Parts 
is that a scanner returns back a set of value with version information, where a 
finder is much more simple “We have a match as resource X”. The reason for the 
difference is that some tools are easier to find as a set vs defining every 
known case. For example when I worked at Intel, the Intel compiler would make 
lots of internal drops as one would expect, these drops would have different 
versions. People would want to test the new drops out. Instead of me making a 
new entry for each new drop, I scanned the disk for example for what was 
installed. And returned all versions installed on disk. This scaled better for 
this tool set for users. Another example ( which I should improve a bit more) 
is the GNU tool change. People can install different version of the compiler on 
a Linux system and they often install them in different ways, such as under 
/opt or in there home directory in a standard way. I tried to have the Gnu 
logic in Parts look for these common places and use these version of requested. 
The use of a scanner logic was a lot easier here than making lots of custom 
finders. The main value of a finder vs scanner for me however is that a finder 
can be more declarative. This makes it easier to read and understand. For a 
system like SCons this is important for people to add new tools in a common way 
to be reused.

Tool args- I don’t define this at the moment as an object. I define some common 
values, but not an object. Originally I was going to define an object. However 
Steve knight convinced me otherwise. There is some value in that choice. Making 
it an object has some different value. I not against or for making a tools arg 
like object. What I am against if a notions of a **kw object which was 
something Steve Knight basically had and what seems to be defined here in the 
new tool form Gary. The reason I have issue with this is that this allows the 
utter chaos we have in the current tools.. ie there is no standard way to set a 
given tool up. I agree a tool should be able to define unique items if it makes 
sense, but we should strive for a common set of basic values that we reuse in 
all tools. Some value like a version may be ignored as it does not make sense 
for it. We still should define a common set as most tool have a common ground 
and people need a common way to set them up. Sure advance cases will exists, we 
have a way with **kw to add advance stuff as it happens. For me value that are 
useful here are an idea of ( This is just a 

Re: [Scons-dev] SCons tools refactoring progress

[Gary Oberbrunner]

Comments inline.

On Tue, Oct 20, 2015 at 12:52 AM, anatoly techtonik 
wrote:

> On Tue, Oct 20, 2015 at 3:57 AM, Gary Oberbrunner 
> wrote:
>>
>> On Sun, Oct 18, 2015 at 7:52 AM, anatoly techtonik 
>> wrote:
>>
>>>
>>> I see the implementation, but I don't see any use cases. I know it
>>> sounds too formal, but I can't validate the assumptions we had towards the
>>> new toolchain without a formal list. Do you have some notes or maybe BDD
>>> tests for that?
>>>
>>
>> A very good question. I don't have enough design notes in there; I'll
>> write up some of my motivations and design goals here and add them to the
>> README.rst.
>>
>> The basic idea is that the current concept of a Tool is too low level;
>> the primary motivating use case is that users (SConscript authors) should
>> be able to select groups of related tools, _as_ a group, with fallbacks. A
>> Toolchain is that abstraction.
>>
>
> The use case is already good to be recorded. But there are two cases:
> 1. Select a group of related tools
> 2. Fallback
> Do we have a concrete groups of related tools already?
>

Yes, I described a few. (Compiler and linker, maybe assembler is the
obvious one.)


> The fallback story needs to be expanded. Are there fallback choices inside
> of one group (with priority or other preference strategy) or the fallback
> means "fallback to another group"? Maybe I am too detailed, but also - what
> are the cases when fallbacks occur?
>

This is all implemented. Check the code and test cases (Toolchain-test.py).
I think it's pretty solid but please review. Basically it's recursive AND
and OR trees with optional or required elements.


> It it not necessary to design everything now upfront - the primary goal of
> my questions is to find *real world* stories (not even use cases) for which
> new mechanism is needed. I am afraid to design a system that will be too
> perfect for implementation in a free time.
>

Well, at least for the toolchain, I think it's mostly there
implementation-wise (check it out). The interesting stuff is improving the
Tool ideas (there's a sort of registry so you can give concrete tools names
and look them up that way), adding Finders, and all the other stuff that
we've been discussing. But if you have some real-world stories you'd like
to see covered, this is a good time to get them out there.

All the rest is just there to make that idea work well. The secondary goal,
>> in service of the main one, is that Tools need to obey their abstraction,
>> for instance always calling exists().
>>
>
> What is the story behind this? Because I know the opposite story -
> exists() for default tools is called even if I don't build anything with
> those tools - this delays the build start and produces messages about
> missing compiler to the screen.
>

exists() is how a tool knows whether its binaries (or whatever it needs to
run) exist or not, so toolchains can't work without it. As for default
tools, the idea here is that SConscript writers will (finally!) be able to
specify exactly which tools they want. I hope to do away with the current
default tool initialization system, though some of that still needs to be
thought out. The current design is much "lazier" but still needs work.

The new system also creates a distinction between an abstract tool, such as
>> intelc, and a concrete instance of it, such as intelc v12 x86. This is
>> needed so the user can create chains of either specific or general tools.
>>
>
> I understand where this might be useful, but still - is there a real world
> story where this was needed?
>

Absolutely - in my day job we specify very carefully what tool chain is
used to build any given product. If the machine doesn't have the right
version of the Intel compiler the build should fail. For open-source
projects, on the other hand, they should try to build on as many
configurations as possible so they want to keep things general -- use any
Intel compiler, or any gcc, or any cc (for instance).


> One restriction I'm imposing in the new system is that Tools have to be
>> configurable outside of any Environment; I don't like the current system
>> where tool-configuration variables ("tool args" basically) are mixed into
>> the Environment. This poses some challenges for a fully generalizable
>> system but I think I have a decent handle on that. The current Intel
>> compiler has an initial attempt at such tool args.
>>
>
> How to handle "tools args" is a question for a separate thread. We will
> need to have a standard set for every *type* of tool and specific for every
> tool supported. For the maintenance that means tables, and perhaps tests
> for those args. And neither unit tests, nor BB wiki are good things for
> human readable tests and human writable tables.
>

Maybe. I think it is _vital_ at this stage to separate mechanism from
policy. We need a mechanism that can support many policies; table-driven
args or standard 

Re: [Scons-dev] SCons tools refactoring progress

[anatoly techtonik]

On Tue, Oct 20, 2015 at 3:57 AM, Gary Oberbrunner 
wrote:
>
> On Sun, Oct 18, 2015 at 7:52 AM, anatoly techtonik 
> wrote:
>
>>
>> I see the implementation, but I don't see any use cases. I know it sounds
>> too formal, but I can't validate the assumptions we had towards the new
>> toolchain without a formal list. Do you have some notes or maybe BDD tests
>> for that?
>>
>
> A very good question. I don't have enough design notes in there; I'll
> write up some of my motivations and design goals here and add them to the
> README.rst.
>
> The basic idea is that the current concept of a Tool is too low level; the
> primary motivating use case is that users (SConscript authors) should be
> able to select groups of related tools, _as_ a group, with fallbacks. A
> Toolchain is that abstraction.
>

The use case is already good to be recorded. But there are two cases:
1. Select a group of related tools
2. Fallback
Do we have a concrete groups of related tools already?
The fallback story needs to be expanded. Are there fallback choices inside
of one group (with priority or other preference strategy) or the fallback
means "fallback to another group"? Maybe I am too detailed, but also - what
are the cases when fallbacks occur?

It it not necessary to design everything now upfront - the primary goal of
my questions is to find *real world* stories (not even use cases) for which
new mechanism is needed. I am afraid to design a system that will be too
perfect for implementation in a free time.


> All the rest is just there to make that idea work well. The secondary
> goal, in service of the main one, is that Tools need to obey their
> abstraction, for instance always calling exists().
>

What is the story behind this? Because I know the opposite story - exists()
for default tools is called even if I don't build anything with those tools
- this delays the build start and produces messages about missing compiler
to the screen.


> The new system also creates a distinction between an abstract tool, such
> as intelc, and a concrete instance of it, such as intelc v12 x86. This is
> needed so the user can create chains of either specific or general tools.
>

I understand where this might be useful, but still - is there a real world
story where this was needed?


> One restriction I'm imposing in the new system is that Tools have to be
> configurable outside of any Environment; I don't like the current system
> where tool-configuration variables ("tool args" basically) are mixed into
> the Environment. This poses some challenges for a fully generalizable
> system but I think I have a decent handle on that. The current Intel
> compiler has an initial attempt at such tool args.
>

How to handle "tools args" is a question for a separate thread. We will
need to have a standard set for every *type* of tool and specific for every
tool supported. For the maintenance that means tables, and perhaps tests
for those args. And neither unit tests, nor BB wiki are good things for
human readable tests and human writable tables.

Again, for the design it needs concrete stories. Like "I want to use intelc
v12 x86 which uses its own args on Windows". My goal with all these stories
is to dig up the cases where new more complicated system just don't worth
it and simple Environment variable hack is good.


> Some use cases:
>  * a simple SConstruct should automatically select the "best"
> compiler/linker
>

What is the best in real world projects? For example, pre-configured chain
for specific combination of OS, environment, flags, source file types? How
many criteria we may (which arguments should we) pass to
get_the_best_tool() function?

Also, I see that it will be good to make it independent of SCons codebase
for now.


>  * a non-C-related SConstruct (e.g. doc prep, asset management, scientific
> data handling) shouldn't have to care about compilers, linkers or other
> unrelated tools
>

Good. Now we need to expand what this means and record into separate story.
"When I invoke Sphinx and SCSS tool, SCons initializes compilers, linkers
and other related tools and complains about missing Visual Studio
compliler. It also takes a long time."


>  * a SConstruct author should be able to specify toolchains and
> alternatives, and handle failures gracefully
>

What is the failure? Only "tool does not exist"? Where it is needed right
now?


>  * it should be possible to write a SConstruct where the tools and
> toolchains can be manipulated by cmd line args if desired
>  * it should be possible to specify desired tools generally
> ("gcc/gnulink") or very specifically ("gcc 4.3, x86 cross compiler, with
> gnulink 4.3 - and TeXLive")
>

I don't see how that's better than just passing the absolute path for
specific binary. Making specific restriction for "gcc" and "4.3" in SCons
defined classes may make it harder to build stuff with compatible
toolchains that don't have SCons definitions.


> There's a bunch of 

Re: [Scons-dev] SCons tools refactoring progress

[Gary Oberbrunner]

On Sun, Oct 18, 2015 at 7:52 AM, anatoly techtonik 
wrote:

>
> I see the implementation, but I don't see any use cases. I know it sounds
> too formal, but I can't validate the assumptions we had towards the new
> toolchain without a formal list. Do you have some notes or maybe BDD tests
> for that?
>

A very good question. I don't have enough design notes in there; I'll write
up some of my motivations and design goals here and add them to the
README.rst.

The basic idea is that the current concept of a Tool is too low level; the
primary motivating use case is that users (SConscript authors) should be
able to select groups of related tools, _as_ a group, with fallbacks. A
Toolchain is that abstraction. All the rest is just there to make that idea
work well. The secondary goal, in service of the main one, is that Tools
need to obey their abstraction, for instance always calling exists(). The
new system also creates a distinction between an abstract tool, such as
intelc, and a concrete instance of it, such as intelc v12 x86. This is
needed so the user can create chains of either specific or general tools.
One restriction I'm imposing in the new system is that Tools have to be
configurable outside of any Environment; I don't like the current system
where tool-configuration variables ("tool args" basically) are mixed into
the Environment. This poses some challenges for a fully generalizable
system but I think I have a decent handle on that. The current Intel
compiler has an initial attempt at such tool args.

Some use cases:
 * a simple SConstruct should automatically select the "best"
compiler/linker
 * a non-C-related SConstruct (e.g. doc prep, asset management, scientific
data handling) shouldn't have to care about compilers, linkers or other
unrelated tools
 * a SConstruct author should be able to specify toolchains and
alternatives, and handle failures gracefully
 * it should be possible to write a SConstruct where the tools and
toolchains can be manipulated by cmd line args if desired
 * it should be possible to specify desired tools generally ("gcc/gnulink")
or very specifically ("gcc 4.3, x86 cross compiler, with gnulink 4.3 - and
TeXLive")

There's a bunch of tests in that dir; they're mostly unit-test level at
this point. There are also some examples of the new subsystem in use (see
the README) which are closer to actual use cases.

My current task is to design a good way for Tools to find their executables
and configure themselves. The current ad-hoc method is not consistent and
doesn't encourage reuse. Jason has some ideas in Parts (see its concept of
Finders); I don't intend to reuse that directly but at least take some
inspiration from it. My current design I'm working on is something like
this: ... a good architecture might be to have a list of finders, to be
tried in order; each one can be any type (env, reg, path, fixed path,
function, whatever); build a list of all the results, and then have a
finder-picker that picks the best from that list (the simplest
finder-picker might be return paths[0]). But where that list comes from and
how it's manipulated is still TBD.

Additionally I'd like to see how far it makes sense to go in having
standard args to configure Tools; one way is just to leave args up to each
Tool (but maybe have the Finders have some high-level control); the other
is to define many standard ones (like ARCH, search in PATH or some custom
path, and so on) - the latter will make it easier someday to pass these all
the way down from the cmd line. I want Tools (and their Toolchains) to
stand alone easily, but also have some way to override their decisions at a
high level (globally, from cmd line, etc.)

And yes, with this system it is a goal that there would be no more "missing
Visual Studio" errors on Windows. :-)

-- 
Gary
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] SCons tools refactoring progress

[anatoly techtonik]

On Fri, Oct 16, 2015 at 10:41 AM, Vasily  wrote:

> so I wanted to see if there was something on the SCons side of things I
> can look at for inspiration.


Well, the current message about missing Visual Studio on Windows is a great
source of inspiration to me. =)

-- 
anatoly t.
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] SCons tools refactoring progress

[Vasily]

Hi all,

Unfortunately I don't think I can devote enough time to get to this right
now as I do not know all the gory details, so I'd need to understand them
first.
My question was actually inspired by the though of slightly re-desigining
tooling concept that wraps SCons Tools (stuff that Jason talked above,
actually :) ), so I wanted to see if there was something on the SCons side
of things I can look at for inspiration.

Thanks for the link, I'll poke around that.


Thanks,
Vasily

2015-10-14 3:54 GMT+03:00 Jason Kenny <dragon...@live.com>:

> Hi Gary,
>
> Glad to see this getting back in gear again. What are the thoughts about
> the finders. I know you stated some time ago about using the finder idea I
> had in Parts. I was wondering had you updated the design you had to define
> sets of finders in some order. Something like what I had in Parts called
> ToolInfo. That object for me took a number of finders/scanners to find a
> needed path, which the toolinfo used to validate the tool existed and to
> provide a set of variable that needed to be setup in the environment, such
> as include/bin/lib/etc.. paths. I am not clear on what your intent is with
> the current design.
>
> Jason
>
> *From:* Gary Oberbrunner <ga...@oberbrunner.com>
> *Sent:* Tuesday, October 13, 2015 7:14 PM
> *To:* SCons developer list <scons-dev@scons.org>
> *Subject:* Re: [Scons-dev] SCons tools refactoring progress
>
> I'm just starting to get back into it.
> If you'd like to take a look at the current state of things, check out
> h...@bitbucket.org/garyo/scons-newtool and look in
> src/engine/SCons/ToolchainDesign.
> The next thing on my list for it is a nice generalized executable-finder
> (PATH, registry, env, hard-coded, etc.)
> Any feedback much appreciated!
>
> On Tue, Oct 13, 2015 at 6:09 PM, Bill Deegan <b...@baddogconsulting.com>
> wrote:
>
>> Vasily,
>>
>> I think Gary was working on that. Work's ongoing.
>> It's not imminent to be merged into main.
>>
>> Are you volunteering to help?
>> ;)
>>
>> -Bill
>>
>> On Tue, Oct 13, 2015 at 2:29 PM, Vasily <just.one@yandex.ru> wrote:
>>
>>> Hi all!
>>>
>>> I recall there were some plans on refactoring how Tools work in SCons.
>>> Are there any news on that?
>>>
>>> Thanks,
>>> Vasily
>>>
>>> ___
>>> Scons-dev mailing list
>>> Scons-dev@scons.org
>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>
>>>
>>
>> ___
>> Scons-dev mailing list
>> Scons-dev@scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
>
>
> --
> Gary
>
> --
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] SCons tools refactoring progress

[Bill Deegan]

Vasily,

I think Gary was working on that. Work's ongoing.
It's not imminent to be merged into main.

Are you volunteering to help?
;)

-Bill

On Tue, Oct 13, 2015 at 2:29 PM, Vasily  wrote:

> Hi all!
>
> I recall there were some plans on refactoring how Tools work in SCons.
> Are there any news on that?
>
> Thanks,
> Vasily
>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] SCons tools refactoring progress

[Gary Oberbrunner]

I'm just starting to get back into it.
If you'd like to take a look at the current state of things, check out
h...@bitbucket.org/garyo/scons-newtool and look in
src/engine/SCons/ToolchainDesign.
The next thing on my list for it is a nice generalized executable-finder
(PATH, registry, env, hard-coded, etc.)
Any feedback much appreciated!

On Tue, Oct 13, 2015 at 6:09 PM, Bill Deegan 
wrote:

> Vasily,
>
> I think Gary was working on that. Work's ongoing.
> It's not imminent to be merged into main.
>
> Are you volunteering to help?
> ;)
>
> -Bill
>
> On Tue, Oct 13, 2015 at 2:29 PM, Vasily  wrote:
>
>> Hi all!
>>
>> I recall there were some plans on refactoring how Tools work in SCons.
>> Are there any news on that?
>>
>> Thanks,
>> Vasily
>>
>> ___
>> Scons-dev mailing list
>> Scons-dev@scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>


-- 
Gary
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev