Re: [Scons-dev] Tool initialization is called twice
On Thu, May 29, 2014 at 12:43 AM, Bill Deegan b...@baddogconsulting.comwrote: Gary, I was under the impression (need to check the code), that you could do: DefaultEnvironment(tools=[]) in SConstruct and it would prevent any tools from being initialized and speed SCons startup. Did I misunderstand? No, that's correct. https://stackoverflow.com/questions/15205210/prevent-scons-from-looking-for-standard-tools#comment21499852_15206709 -Bill On Wed, May 28, 2014 at 5:40 AM, Gary Oberbrunner ga...@oberbrunner.comwrote: On Wed, May 28, 2014 at 3:39 AM, anatoly techtonik techto...@gmail.comwrote: Hi, I found that SCons calls tool initialization twice for this simple SConstruct: env = Environment() env.Program(target='flite', source=[]) Right, this is one of the things I want to address with the toolchain revamp. DefaultEnvironment is constructed before the SConstruct is read, and all the default tools are detected there. Environment() is distinct from DefaultEnvironment(); DefaultEnvironment is _only_ used for plain Object(), Program() etc. calls. This is by design. Environment() does not inherit changes in DefaultEnvironment(). -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Toolchain requirements
Hi Gary, here are my comments: On 25.05.2014 19:14, Gary Oberbrunner wrote: I'd like to kick off a round of discussion about toolchains, so can make some progress toward a design. I have some preliminary thoughts. Please comment. Apologies for the HTML mail, let me know if this isn't readable for you. * Allow installing external tools (pip install or ...) * o scons --version (or similar) should list installed tools and toolchains o missing external tools should give sensible errors I'm not sure I understand what you mean by missing here. An external Tool that isn't able to find the executables or packages it depends on in the current system? Like an overview, where you get an Available/NotFound for each Tool? What I wouldn't like is, if we'd define a set of external Tools that the user would *have* to install. We can provide a package of external tools in a bundled fashion, something like our best of. But each user should be free to install it, or only the single Tools he needs, without getting confusing messages about a Tool not being found, just because we think he needs to have it. * Tool setup must happen before reading SConstruct somehow I don't know if this is really required, see my comment below about lazy initialization of Tools. It would be good to have a pool of Platform-related info (which OS/Python/Kernel are we running on?) at this early stage, that all environments/tools can access directly. * o DefaultEnvironment and all new Environments should know about all tools o alternative: lazy-construct DefaultEnvironment o user-specified tools and toolchains need to be specifiable at beginning of build * User should be able to set default tools and toolchains ...and the order in which they are probed, if possible this specification should be per OS. * o unused tools shouldn't take any startup time * Lazy init of tools and chains * o This is faster because unused tools don't matter o It allows missing unused tools to not give errors, but missing used tools can (and should) o But it makes configuring environments much harder for users, because they can't override or append to tool-provided variables until those exist. This would break a lot of existing SConstructs. o We need to find some kind of compromise here: o + Explicitly list tools required by build (where?): this should work well because only the needed tools will be initialized + if nothing explicitly specified, fall back to current method We should also take the case into account where a user might set up a dummy environment. Depending on which Toolchain seems to be available, he then wants to setup his actual environments, or install additional packages first, or... I, personally, think that we don't need lazy initialization here. If we give a user more option to control which Tools/Toolchains are probed in which order, he can size down the effort for detecting stuff nicely. At least, the latter (=toolchain control) should have higher priority over the certainly nice and cool lazy init feature. Doing things lazy, also makes things like debugging, writing test cases or creating issue reports a little harder... * Within a tool: * o specify dependencies on other tools What kind of dependencies are you thinking about? Module-wise, method-wise or both? Is it more about data dependencies, or order dependencies for the probing phase? * o detect existence on system reliably, and without modifying env o + need better error messaging: ability to probe silently, but also give sensible errors when needed o constructor needs to allow args: version, path, ABI, etc. (this is important) o allow for common setup (all C compilers, etc.) as now o tools should be versioned so user can check if up to date, etc. * Tool chains: * o either-or o and o collections * Platform * o How much do we need to know about the platform, for tools to initialize themselves? o Cross-compilation comes into this, but may be too much to include as a general part of this project. It's really hard to draw the line here. For my personal taste, we should have a PlatformInfo with only the basic data about OS/CPU/RAM/Kernel and the Python version we're running under. Everything else, like latest Java version or available cross-compilers, should be handled by the rest of the toolchain/tools stuff somehow. In general, I just don't want to have things getting all muddy and confused to fast. This will happen anyway down the road of implementation, faster than we can think. ;) So let's try to be as clear and concise as possible right now at the beginning, and establish clear visions and borders about what we want to do, and what
Re: [Scons-dev] Toolchain requirements
I'd really like to see some easier way of configuring things. Our sysadmins in
their wisdom install things somewhere where scons can't find them (with the
rather fun result that on solaris, it finds gcc and tries to pass it the
parameters for CC which merely adds to the fun).
I'd like to say - I want to use xlC and *here is where to find it*, then let
scons set up all the flags and things it thinks I should use. I'd be happy
enough to create a wrapper tool which sets up the required path, then delegates
most of the setup to the provided tool, then adds in whatever specific useful
stuff we might require.
I'd like a better way of specifying the site dir to find tools. Though you can
do this on the command line, you can't do it inside the SConstruct file which
makes for grief (this is sort of counter to the tool setup happening before
SConstruct is read though). I've seen more than 1 place where
SCons.Script.Main._load_site_scons_dir(Dir('#'), site_dir)
is called.
If you are going to make lazy initialisation of work, it sort of requires
clients to be able to wrap the tool initialisation with their own, which I
don't think makes things more complex, it would clean things up a bit, so that
if you say used CC and scons looked for CC in your site tools directory, and
found it, as long as your site tools could hand off to the original site tools
after preconfiguring, then add additional stuff, then you'd get a cleaneier and
not too hard to understand system. On the bad side, it would break things.
$CC being left blank would presumably not be an issue, assuming that the
toolchain setup did the right thing and either succeeded with all the variables
set up or failed properly.
Now I need to go read the documentation...
- Original Message -
From: scons-dev@scons.org
To: scons-dev@scons.org
At: May 25 2014 18:15:09
I'd like to kick off a round of discussion about toolchains, so can make some
progress toward a design. I have some preliminary thoughts. Please comment.
Apologies for the HTML mail, let me know if this isn't readable for you.
*Allow installing external tools (pip install or ...)
*
*scons --version (or similar) should list installed tools and toolchains
*missing external tools should give sensible errors
*Tool setup must happen before reading SConstruct somehow
*
*DefaultEnvironment and all new Environments should know about all tools
*alternative: lazy-construct DefaultEnvironment
*user-specified tools and toolchains need to be specifiable at beginning
of build
*User should be able to set default tools and toolchains
*
*unused tools shouldn't take any startup time
*Lazy init of tools and chains
*
*This is faster because unused tools don't matter
*It allows missing unused tools to not give errors, but missing used tools
can (and should)
*But it makes configuring environments much harder for users, because they
can't override or append to tool-provided variables until those exist. This
would break a lot of existing SConstructs.
*We need to find some kind of compromise here:
*
*Explicitly list tools required by build (where?): this should work
well because only the needed tools will be initialized
*if nothing explicitly specified, fall back to current method
*Within a tool:
*
*specify dependencies on other tools
*detect existence on system reliably, and without modifying env
*
*need better error messaging: ability to probe silently, but also give
sensible errors when needed
*constructor needs to allow args: version, path, ABI, etc. (this is
important)
*allow for common setup (all C compilers, etc.) as now
*tools should be versioned so user can check if up to date, etc.
*Tool chains:
*
*either-or
*and
*collections
*Platform
*
*How much do we need to know about the platform, for tools to initialize
themselves?
*Cross-compilation comes into this, but may be too much to include as a
general part of this project.
*It may be useful to define toolchains and enable/disable them by platform
*Of course the default toolchains need to be different by platform
*It may be possible for a default toolchain to just search for all tools
in a particular order and pick the first, as long as the tool-dependency system
is robust enough.
*Usability
*
*$CC etc. must never be left blank (without a prior tool-missing error
message at least) - this is a common problem
*Must be backward compatible, at least for all common cases.
*Must not require any new user files (e.g. something in site_scons) for
normal operation
*Need a clear guide on requirements for new tools
*
*how to make a a tool
*how to include tests
*Considerations
*
*batteries included?
*
*Each tool should do its best to set itself up, find executables, etc.
*What about
Re: [Scons-dev] API for warnings and debug messages
On Wed, May 28, 2014 at 8:59 PM, Kenny, Jason L jason.l.ke...@intel.comwrote: I agree it look complex, but it is not that bad… Here is my first look. == Problem outline is good, needs more time to become obvious == I like the idea to make 100% cover of all possible inputs/outputs (text data that comes out and into SCons). The diagram is not enough to illustrate the concept and requires reading the doc thoroughly. I started to draw it again: https://docs.google.com/drawings/d/13Ibva0qyYtEJq9enYkA3kj93ANAONduLxeE7I2H9l4k/edit There are many parts that seems evident, but for most people they are new, so they need to be explained, and better to have one picture per one concept. Just more time to polish. == output.py == It is just a bunch of functions. I see there are some magic string like sep, but no explanation, and for the first look I can not really investigate deeper. My use case - I have an external debugger and I want to see what SCons does _without modifying_ SCons. I want to filter messages, get into more deep level of detail (1-9), or to turn off processing at all to speed up the process. How can I turn off processing for all these functions? How can I replace the function? How can I filter error or warning, how can I get only specific errors and warnings for my output? So, I need to filter both error_ and warning_ functions, and all others as well. I see that we will just reimplement logging the Java way coming through the same trapdoor. == Easy logging for the start, overengineering later == Python is good for prototyping. Why not to start from scratch and make the most simple system possible. Just one function that produces strings. Let's call it probe(). How would you produce strings is another question. If we need a component name, we produce it like: probe(component: created object) What probe does is up to you. It is overridable function - by SConstruct, by external tool, by anything. The only restriction is that any modification to probe() is itself logged. Who when where changed probe. This can be just a memory log. I want logging to be completely decoupled from the program (or as decoupled as possible). Like in real world you can tap to wires to eavesdrop without performance penalty - the same interface should be possible in open source world, and it will be good if SCons can provide one. Not sure about implementation, but in the end I see it as an external monitor that connects to SCons and reacts to some events that appear through the probe() function. If probe() is implemented blocking - this can be used to control speed of execution. == On big chunk problem == I need more time to review the document. It is much easier to review stuff if you have one real world user story to tackle at a time. Story first - theory second. Right now I'll just wait for the next chunk of free time to come. It mostly about taking output from different sources and putting it together allowing for coloring, no mangled text ( as we get with raw scons and a –j based build). Each part is simple and does a simple thing. What I was suggesting here was more of a view to add a SCons.api.output module that has some sort of error_msg(), warning_msg() etc… This allow an easy way to standardize output formats. For verbose and or debugging messages it can be very useful to allow for a filter logic. This is just a formatting API. http://techtonik.rainforce.org/2013/02/formatting-api-anti-pattern.html What is important is to understand where messages are coming from, where they end, what is the processing logic, how do you define a message, why do you define a message, to analyze the data structure first. Who said that we only have debugs and warnings? What if we need 10 different circles of messages on one axis and different components on another? How to query about available components that can be turned on and off to produce messages? What I have is simple in this. You can say –verbose=type,type1 on the commandline and only verbose_msg(“type”,”message”) will print. This makes for a simple API to output stuff. And control what gets outputted vs the dump everything and grep logic that is common. With this, what it does under the covers can change to do different things as found to be useful. Having an API that is clear I feel is more useful. Problem is that, as a user, I don't know which type of message I need to debug some issue. I error message (traceback) contains the command that I can type to get more info - that will help. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Tool initialization is called twice
Gary, I agree ( I think you mentioned in another thread) that lazy tool initialization would remove a lot of wasted and duplicated tool initialization. -Bill On Wed, May 28, 2014 at 11:50 PM, anatoly techtonik techto...@gmail.com wrote: On Thu, May 29, 2014 at 12:43 AM, Bill Deegan b...@baddogconsulting.com wrote: Gary, I was under the impression (need to check the code), that you could do: DefaultEnvironment(tools=[]) in SConstruct and it would prevent any tools from being initialized and speed SCons startup. Did I misunderstand? No, that's correct. https://stackoverflow.com/questions/15205210/prevent-scons-from-looking-for-standard-tools#comment21499852_15206709 -Bill On Wed, May 28, 2014 at 5:40 AM, Gary Oberbrunner ga...@oberbrunner.com wrote: On Wed, May 28, 2014 at 3:39 AM, anatoly techtonik techto...@gmail.com wrote: Hi, I found that SCons calls tool initialization twice for this simple SConstruct: env = Environment() env.Program(target='flite', source=[]) Right, this is one of the things I want to address with the toolchain revamp. DefaultEnvironment is constructed before the SConstruct is read, and all the default tools are detected there. Environment() is distinct from DefaultEnvironment(); DefaultEnvironment is _only_ used for plain Object(), Program() etc. calls. This is by design. Environment() does not inherit changes in DefaultEnvironment(). -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons and Java
Team, I have a couple points I'd like to discuss, but for the sake of organization, I intend to split them into separate emails. Java Part #2 There has been some discussion on making Scons more ANT-like, http://www.scons.org/wiki/JavaSupport, and that might solve some issues like identifying Java targets. There is merit here assuming that Java support stays SCons-like at the same time; otherwise, why not just use ant or maven? Problem(s) this will solve: 1. Determining the source - target(s) from *.java - *.class is painful, possibly java version dependent, and performance intensive (possibly); thus, we should optimize this out. 2. Java developers are plagued requiring manual cleaning of a jar far too often with ant-like system(s) and the current scon(s) methodology. 1. Clean could be required when deleting or moving a file since the last build output isn't removed otherwise; avoidable in current SCons depending on how sources are specified (not obvious and/or intuitive coming from ant-like system(s)). 3. Simplify the SCons Java API to allow building jar file(s) without having to interact with *.class file(s) at all. 4. Simplify the SCons Java support for resource file(s); currently, you have to do copying by hand (if it's supported I know not where or how). 5. Reduce the amount of code required to build a jar (by a lot). How I imagine this should work: 1. Treat class file(s) like side-effect(s) and do not include them (at all) in the dependency tree. 1. This means we don't have to scan the Java file(s) and determine *.class targets. 2. Remove the Java(...) function from SCons; thus, do not interact with *.class file(s) directly ever. There isn't a *good* reason to do this. 3. Update the Jar function (or new name) to something like: 1. Jar( sources = Glob(*.java) compiled sources and ! *.java resources to be copied, target = output jar name/destination, outputDir = the tmp working directory, isOutputDirRemoved = true/false ) *Example setup*: 1. Source tree: src/org/example/a.java /b.java src/org/resources/x.png /y.png src/org/config/n.xml 2. SConscript Jar( sources = 'src', target = '#jar/example.jar' ) // assume some smart auto-generated unique outputDir name 3. Dependency tree generation expectations for the example #jar/example.jar | - all *.java in 'src' | | | - javac | - all item(s) in JAVACLASSPATH: should probably restrict to only *.class and *.jar (anything else?) | | - all ! *.java in 'src' which implies resources or other *Use Case(s)* *Note: each case starts in the final state of the previous use case.* 1. Build from scratch 1. Compile *.java in sources into outputDir: 'javac -classpath *.java' 2. Copy ! *.java in sources into outputDir: use SCons copy factory 3. Create the jar of all contents in outputDir as target 4. *Maybe* delete outputDir since this is a side-effect directory only 2. Build with no changes 1. SCons builds no targets. 3. Build with changes to a file in src, new file in src, removed file in src 1. Delete outputDir (IE. if not removed in case 1.4). 2. Perform steps 1.X. 4. Change JAVACLASSPATH, target is removed. 1. Perform steps 3.X. 5. Delete the outputDir (if not removed). 1. SCons builds no targets. 6. Clean 1. Perform steps 3.1. 2. Remove the target. *This is an example set; feedback wanted.* V/R, William On Mon, May 26, 2014 at 1:11 PM, anatoly techtonik techto...@gmail.com wrote: I'd say that as long as everything discussed here ends up in pull requests, all such discussions are welcome. As particular Java problem, I miss an overview of the problems that need to be solved for this domain. On Sat, May 24, 2014 at 6:27 AM, William Blevins wblevins...@gmail.com wrote: SCons Team, I've been using SCons for a couple years now and I have some thoughts about the Java toolkit. I know that there has been consideration for revamping the Java toolkit and I was wondering if anyone would be open to constructive feedback. Firstly, I'd like to say that SCons is a fantastic toolkit, and I would like to help bring the Java portion up to par with the support for c-like languages. I have always been underwhelmed by ANT-like builders and I think SCons has a great start to be something better. Would this be an appropriate place for discussion or should I use the scons-user list (or other) instead? V/R, William ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
