Re: [Scons-dev] Schedule for redesign?
Dirk, If you have changes which substantially improve SCons's memory footprint and/or runtime. I would ask that you break those up into smaller submissions and send pull requests. While we want to be aware of Jason's Part's work. We don't necessarily want to be held up by it. Once you submit the pull requests, then it would be great if Jason could comment on any changes which might impact his Parts work. That said. Let's not make better the enemy of good. If we can release your improvements in the near term that would be wonderful. Assuming your changes as they are today don't break any tests, then we should get them in as quickly as possible. If the changes are just organizational, then it's worth taking more time to consider impact on other projects. Thanks, Bill On Oct 28, 2013, at 11:44 AM, Dirk Bächle wrote: > Hi Jason, > > and thanks for chiming in. > > On 28.10.2013 17:07, Kenny, Jason L wrote: >> I hope I would not be a wrench in engine :-) >> >> Honestly The issue that this is work and I have to do this at home is the >> main reason I have not pushed anything at this point. ( And having twin 3 >> years-old means I don't have the go power at the end of the day...) >> >> I would agree with what is being stated. I would go a bit farther however. > > I understand that you have done some work on redesigning/refactoring parts of > SCons, and you certainly don't want to lose it for nothing. So, I'd like to > make the following suggestion: I'll factor out some attributes of the Node > class into their own module and commit these changes (together with the other > rewritings I mentioned) to a named branch in my personal "experimental" SCons > fork. > Then you can have a look at it and check whether you'd be able to base your > further work on that. If not, you can tell me what should be changed such > that you can continue properly with your work. If I get your "okay", the > changes find their way into the SCons core and you can take it from there > whenever you find the time. How does that sound? > >> I would redesign the Node API. But also redo the executioner logic. Ideally >> we want I have learned is that the task logic needs to be updated. Greg Noel >> was on the correct path with TNG in that we want to execute "builders" not >> nodes. We also need the API ( internal) to decouple the relationships with >> the tasks and the nodes. >> >> I would also look at fixing up the subst engine. I found that this is >> another area of memory waste at the moment ( and not in the good way). This >> might be more of an issue with Parts as I currently use the subst engine to >> share data between different "parts"/ components. Until I do a new part file >> format and or a "continuous loader" I have no way to pass data correctly. > > During my investigations I tried to speedup subst() by caching some > pre-expanded strings, but it didn't work out. My current impression is that > you either have to compile it (including optimizations by hand, simply > running Cython over it is not enough!) or reduce functionality (like cutting > down the support for callables) to get it faster. > > But don't let me stop you from trying... ;) > > Best regards, > > Dirk > > ___ > Scons-dev mailing list > Scons-dev@scons.org > http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Schedule for redesign?
Sounds good... Technically I can open up what I have for the Scons nodes
rewrite as well.. I just trying to get the interfaces fixed and clear and clean.
I have with Parts two node trees. One for the SCons nodes, and one for the
component nodes. Ideally I would want to merge these together as one, with one
node manager, not many ( like we have in Scons). We need to get Symlinks in the
node tree for file based nodes. I have it working in Parts (yaaa...) It works
well, and allows us to define a gnu style symlink setup in a cross platform
way. I believe one thing I found with this is that we only need symlink for
file. Directory links are a can of worms.
For subst() stuff it is the memory size. There is a set of special objects that
are buggy and wasteful in memory. There is very little caching going on here.
This can be improved.
I should add from what I learned in what I have for Scons speed up with Parts
there are few things that can be done in SCons. So some thoughts here to think
about.
I store in my parts cache for a given cache key the same info SCons stores,
plus a lot more. For some reason the size I was seeing stored is less that the
master scons db file. I looked at it a little and honestly got lost in some
code that tries to mess with the pickled data.
>From that one area of speed in SCons is that all scons files are executed when
>they are read. From a reclusive makefile point of view this might be needed,
>but when you have a more component load model ( as in Parts) this is a BIG
>waste if we don't need to build that component. Currently Parts tries to
>figure out what it needs to load, and loads only that. This gives a huge speed
>up. When mixed in to some no depends/unsafe loading logic it is really fast (
>and useful is you are only trying to rebuild you top-level component... no
>need to load all the dependent components as these are already built and we
>know we did not modify them, or have the system to check to see if they are)
>This is the main reason why I would like to have the ability have a new format
>for Parts as this would allow for delayed execution.. ie something like ( very
>simple example):
PartName("foo")
PartVerson("1.2"
@config
def config(env):
... setup.. test stuff..
@emit:
Def emit(env)
Files=env.Glob(*.c)
Env.SharedLibrary("$PARTNAME_$PARTVERSION",Files)
In this case the need to configure a component or call emit ( lack of better
name) can be delayed unit we need to do it. For example here the Glob() call is
slow, as well as the SharedLibrary() call. The setting a name or adding a
CPPFLAG is fast.
I also believe that we can do a lot more with the DB/cache files. Given you
skip the python only builder issues. Most actions are command line ones. These
actions can be stored in the DB. Doing a set of Checks like Parts.. Scons can
easily detected ( in less than a second) if any files that define context of
what we would build. Given that is not changed, we load information about what
we last built instead of load it all from files and executing code to make a
tree of what to do again and instead just start building the nodes from the
stored action string without loading any build files. This is ideally the most
common case for developers as they are modifying code, not adding/removing file
or modifying build files. ( even in the case of adding removing files, we could
do a lot to reduce what is loaded)
I should say that Given some tweaks, I do believe SCons can do a correct safe
build much faster, vs doing trick to get speed up but with risk of an broken
build.
I should add that I have seen a lot of SCons files are more scripts. They call
execute() to build stuff vs letting SCons do what it does best. These seems to
be happen because the developer does not understand what SCons does. They still
used SCons because the Environment object provides and easy to under model to
control different tools chains and cross build in a single pass. This notion
seems to be much hard to deal with in other many other systems.
I am sure I have more stuff/experience to add.. but I need to get some work
done :-)
Jason
-Original Message-
From: scons-dev-boun...@scons.org [mailto:scons-dev-boun...@scons.org] On
Behalf Of Dirk Bächle
Sent: Monday, October 28, 2013 1:45 PM
To: SCons developer list
Subject: Re: [Scons-dev] Schedule for redesign?
Hi Jason,
and thanks for chiming in.
On 28.10.2013 17:07, Kenny, Jason L wrote:
> I hope I would not be a wrench in engine :-)
>
> Honestly The issue that this is work and I have to do this at home is
> the main reason I have not pushed anything at this point. ( And having
> twin 3 years-old means I don't have the go power at the end of the
> day...)
>
> I would agree with what is being stated. I would go a bit farther however.
I understand that you have done some
Re: [Scons-dev] Schedule for redesign?
Hi Jason, and thanks for chiming in. On 28.10.2013 17:07, Kenny, Jason L wrote: I hope I would not be a wrench in engine :-) Honestly The issue that this is work and I have to do this at home is the main reason I have not pushed anything at this point. ( And having twin 3 years-old means I don't have the go power at the end of the day...) I would agree with what is being stated. I would go a bit farther however. I understand that you have done some work on redesigning/refactoring parts of SCons, and you certainly don't want to lose it for nothing. So, I'd like to make the following suggestion: I'll factor out some attributes of the Node class into their own module and commit these changes (together with the other rewritings I mentioned) to a named branch in my personal "experimental" SCons fork. Then you can have a look at it and check whether you'd be able to base your further work on that. If not, you can tell me what should be changed such that you can continue properly with your work. If I get your "okay", the changes find their way into the SCons core and you can take it from there whenever you find the time. How does that sound? I would redesign the Node API. But also redo the executioner logic. Ideally we want I have learned is that the task logic needs to be updated. Greg Noel was on the correct path with TNG in that we want to execute "builders" not nodes. We also need the API ( internal) to decouple the relationships with the tasks and the nodes. I would also look at fixing up the subst engine. I found that this is another area of memory waste at the moment ( and not in the good way). This might be more of an issue with Parts as I currently use the subst engine to share data between different "parts"/ components. Until I do a new part file format and or a "continuous loader" I have no way to pass data correctly. During my investigations I tried to speedup subst() by caching some pre-expanded strings, but it didn't work out. My current impression is that you either have to compile it (including optimizations by hand, simply running Cython over it is not enough!) or reduce functionality (like cutting down the support for callables) to get it faster. But don't let me stop you from trying... ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Schedule for redesign?
I hope I would not be a wrench in engine :-) Honestly The issue that this is work and I have to do this at home is the main reason I have not pushed anything at this point. ( And having twin 3 years-old means I don't have the go power at the end of the day...) I would agree with what is being stated. I would go a bit farther however. I would redesign the Node API. But also redo the executioner logic. Ideally we want I have learned is that the task logic needs to be updated. Greg Noel was on the correct path with TNG in that we want to execute "builders" not nodes. We also need the API ( internal) to decouple the relationships with the tasks and the nodes. I would also look at fixing up the subst engine. I found that this is another area of memory waste at the moment ( and not in the good way). This might be more of an issue with Parts as I currently use the subst engine to share data between different "parts"/ components. Until I do a new part file format and or a "continuous loader" I have no way to pass data correctly. The another area that could use a bit of work to is the environment objects. Ideally I would move for a more aggressive copy on write logic as the default environment object the user would use. I this would save memory as well, and help with speed. Creating a new Environment is very slow, Cloning is faster, but still causes a lot of memory to be used. Given that Parts for example makes a "Clone" of a Default Environment to make sure we have a clean environment for the component being defined, this would be a big deal. I have noticed that most environments are different because one or two values. There is little value in copying ( which clone does) most of the data. We only need to have a copy on write for what needs to be different. Last area would be to Setting object I have not finished in Parts. This was again based on Greg idea for IAPAT. This is more functional to help with managing large projects, cross build, configuration information. I personal have started to rewrite Scons ( merge of some Parts stuff) myself.. but like I said above... I have not made any real progress to talk about. Jason ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Schedule for redesign?
Dirk, On Oct 27, 2013, at 3:02 AM, Dirk Bächle wrote: > Hi, > > over the last few days I had another look at SCons' speed and memory > problems. As posted in an earlier email, I am able to reduce the maximum > amount of memory used during runtime (both, clean and update builds) by up to > 50% in large C/CPP projects. > This is reached by freeing infos in the Node class itself, when they aren't > needed anymore after a target is built/visited. > > Having identified a set of Node attributes like this, somewhat lends itself > to refactoring them out into its own class (TargetInfo?). There has always > been some rumor about a redesign of the Node classes and the Taskmaster, > which is what this is pretty much about...although not in a very broad range. > > What I'd like to do (and actually already started in a private branch) is to > refactor a few classes and imports, like follows: > > - move the GetOption/SetOption/FakeOptionParser stuff out of > SCons.Script.Main into SCons.Options (have this ready) > - move the Task classes (Clean/AlwaysBuild/...) out of the SCons.Taskmaster > into their own module SCons.Tasks (ready, too) > - move the Node attributes mentioned above into their own class > SCons.TargetInfo (tbd) > > In effect, funtionality gets shifted to where it belongs conceptually, and > not where it is "needed". This might make access to certain parts a little > more complicated, in the sense that one more often needs to delegate things > or distribute information like option flags to several places. But it also > makes the design overall easier to grasp and explain...at least in my opinion. > > Now my question (see the title): When would this fit into our current > development schedule? Should I just go ahead and propose one (or several) > pull requests, or wait until the Python 2to3 conversion is over and things > have settled. Do you think we need a special deprecation cycle for this, or > maybe even should switch the version number to 3.x then? > > Your comments are welcome… My two cents.. Smaller pull requests sound good. I don't think we need any deprecation if it's an internal API. Those have never had any promise of being stable. Though you might want to point out the pull requests specifically to Jason Kenny to see if it's going to throw a wrench into his Parts extension. I don't see any reason to wait until 2to3 is completed. Though it might be work 2to3'ing your new/changed code to see if there is any simple changes which might resolve some issues. So in other words, try not to add additional difficulties in getting code ready for python3. Thanks for your efforts! -Bill ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] Schedule for redesign?
Hi, over the last few days I had another look at SCons' speed and memory problems. As posted in an earlier email, I am able to reduce the maximum amount of memory used during runtime (both, clean and update builds) by up to 50% in large C/CPP projects. This is reached by freeing infos in the Node class itself, when they aren't needed anymore after a target is built/visited. Having identified a set of Node attributes like this, somewhat lends itself to refactoring them out into its own class (TargetInfo?). There has always been some rumor about a redesign of the Node classes and the Taskmaster, which is what this is pretty much about...although not in a very broad range. What I'd like to do (and actually already started in a private branch) is to refactor a few classes and imports, like follows: - move the GetOption/SetOption/FakeOptionParser stuff out of SCons.Script.Main into SCons.Options (have this ready) - move the Task classes (Clean/AlwaysBuild/...) out of the SCons.Taskmaster into their own module SCons.Tasks (ready, too) - move the Node attributes mentioned above into their own class SCons.TargetInfo (tbd) In effect, funtionality gets shifted to where it belongs conceptually, and not where it is "needed". This might make access to certain parts a little more complicated, in the sense that one more often needs to delegate things or distribute information like option flags to several places. But it also makes the design overall easier to grasp and explain...at least in my opinion. Now my question (see the title): When would this fit into our current development schedule? Should I just go ahead and propose one (or several) pull requests, or wait until the Python 2to3 conversion is over and things have settled. Do you think we need a special deprecation cycle for this, or maybe even should switch the version number to 3.x then? Your comments are welcome... Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
