Re: [Scons-dev] Time for a release?
William, Of course your vote counts! It's open source. -Bill p.s. Plus you have a good first name so I give it a little more weight :) On Wed, May 20, 2015 at 4:45 PM, William Blevins wrote: > > On Wed, May 20, 2015 at 3:14 PM, Dirk Bächle wrote: > >> On 20.05.2015 20:35, Bill Deegan wrote: >> >>> Could we push default as is as 2.3.5, and then merge slots and push that >>> in a couple weeks? >>> Then we could branch default as 2.3 prior to merge of slots? >>> >> >> Sure, I have no objections to that. After 2.3.5 we could also branch out >> from default to a "2.4" branch, and then merge the "slots" stuff only on >> that branch as a first step. So we would never risk to "taint" our trunk >> with merges that would be kind of hard to undo. >> If no problems show up with 2.4 we could then still merge it again to >> default, and continue from there towards 2.5 and 3.x. Just as an idea... > > > I think this is a good idea. Risk mitigation plus consistency with the > SCons version numbers~! > > Not that my vote counts ;) > > V/R, > William > > ___ > Scons-dev mailing list > Scons-dev@scons.org > https://pairlist2.pair.net/mailman/listinfo/scons-dev > > ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
On Wed, May 20, 2015 at 3:14 PM, Dirk Bächle wrote: > On 20.05.2015 20:35, Bill Deegan wrote: > >> Could we push default as is as 2.3.5, and then merge slots and push that >> in a couple weeks? >> Then we could branch default as 2.3 prior to merge of slots? >> > > Sure, I have no objections to that. After 2.3.5 we could also branch out > from default to a "2.4" branch, and then merge the "slots" stuff only on > that branch as a first step. So we would never risk to "taint" our trunk > with merges that would be kind of hard to undo. > If no problems show up with 2.4 we could then still merge it again to > default, and continue from there towards 2.5 and 3.x. Just as an idea... I think this is a good idea. Risk mitigation plus consistency with the SCons version numbers~! Not that my vote counts ;) V/R, William ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
On 20.05.2015 21:20, Bill Deegan wrote: Dirk, Sounds great! Is that something you execute? :) Well, I'd be willing to try. ;) However, this definitely requires to have the Wiki available again, so I can lookup the single steps for the release procedure...and I'm not sure that I have all the logins/passwords, e.g. for uploading the final packages to SourceForge. So you might hear from me again soon... :) Other thing is, that I'll be away over the weekend...so I'd start today doing a few first checks, like ensuring that the generated docs are up-to-date. Release would then happen during the next week, maybe Thursday or so. Sounds good to everybody? Dirk ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
Dirk, Sounds great! Is that something you execute? :) -Bill On Wed, May 20, 2015 at 12:14 PM, Dirk Bächle wrote: > On 20.05.2015 20:35, Bill Deegan wrote: > >> Could we push default as is as 2.3.5, and then merge slots and push that >> in a couple weeks? >> Then we could branch default as 2.3 prior to merge of slots? >> > > Sure, I have no objections to that. After 2.3.5 we could also branch out > from default to a "2.4" branch, and then merge the "slots" stuff only on > that branch as a first step. So we would never risk to "taint" our trunk > with merges that would be kind of hard to undo. > If no problems show up with 2.4 we could then still merge it again to > default, and continue from there towards 2.5 and 3.x. Just as an idea... > > > Dirk > > ___ > Scons-dev mailing list > Scons-dev@scons.org > https://pairlist2.pair.net/mailman/listinfo/scons-dev > ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
On 20.05.2015 20:35, Bill Deegan wrote: Could we push default as is as 2.3.5, and then merge slots and push that in a couple weeks? Then we could branch default as 2.3 prior to merge of slots? Sure, I have no objections to that. After 2.3.5 we could also branch out from default to a "2.4" branch, and then merge the "slots" stuff only on that branch as a first step. So we would never risk to "taint" our trunk with merges that would be kind of hard to undo. If no problems show up with 2.4 we could then still merge it again to default, and continue from there towards 2.5 and 3.x. Just as an idea... Dirk ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
I would have to look. But I think if you do the "build [targets]" command it will have values until it is finished building those targets. Jason -Original Message- From: Scons-dev [mailto:scons-dev-boun...@scons.org] On Behalf Of anatoly techtonik Sent: Wednesday, May 20, 2015 1:31 PM To: SCons developer list Subject: Re: [Scons-dev] How to traverse the graph after files are read On Wed, May 20, 2015 at 6:51 PM, Kenny, Jason L wrote: > [...] Thanks for the explanation. I wrote it down to ask questions later when I get some output with the tree. > To get the tree you want all you have to do is grab the target list ie... > SCons.Script.BUILD_TARGETS and for each item in the list, turn it into a node > ( as it might be a string still) and then you start to iterate it children. > As pointed out the main code for this in SCons.Script.Main.py. you can look > at TreePrinter as a starting point and search for the code that uses it. I inserted "print SCons.Script.BUILD_TARGETS" right before checking condition for interactive mode after SConscript files are read), and it is empty. But SCons is still trying to build something. How it determines those targets? -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
Could we push default as is as 2.3.5, and then merge slots and push that in a couple weeks? Then we could branch default as 2.3 prior to merge of slots? On Wed, May 20, 2015 at 11:12 AM, Dirk Bächle wrote: > Bill, > > On 20.05.2015 19:37, Bill Deegan wrote: > >> Dirk, >> >> Are you suggesting we should merge slots into default and push 2.4? >> Or push what we have as 2.4 and then merge slots? >> >> > if I got Jason right we don't have to wait for Parts to synchronize our > development, so we're free to latch on. I suggest that we now merge the > slots branch into default and then release this as 2.4 (as was our original > plan). As a next step we will then integrate the stubprocess.py wrapper > (and not much more) and then quickly let another release 2.5 follow. > > Like this, both minor version numbers signal that there is a (somewhat) > bigger change, but without changing APIs or breaking existing scripts. > > After that we're free to work on the 2.7/3.x version which we'll finally > release as 3.0. This is what makes the most sense to me right now. > > I'm thinking since slots will (could) break compatibility for some that >> we should have a major version change? (3.0?) >> >> If I remember correctly, that was our criteria for major version number >> changes. >> Unfortunately that means 3.0 may confuse some with thinking python 3.0 is >> supported by the new (slots) version. >> (No need to discuss python 3.0 here, that's an entirely other ball of wax) >> >> *There are 3 broken tests on our win32 buildbot slave. We should resolve >> those prior to a release.* >> >> > I agree that we should ensure that there are no broken tests left. However > I've seen those three tests to be a little flaky under Windows over time > anyway. So maybe we should wait what the buildbots say after the "slots" > merge? > We can then try to fix the tests or get them more robust. > > In general, it would be good to account for another week or so of general > testing, after the "slots" merge (and before releasing v2.4). It would be > good if as many developers as possible could then try out the > "default/trunk" version again on their projects or in general. > > > Regards, > > Dirk > > ___ > Scons-dev mailing list > Scons-dev@scons.org > https://pairlist2.pair.net/mailman/listinfo/scons-dev > ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Wed, May 20, 2015 at 6:51 PM, Kenny, Jason L wrote: > [...] Thanks for the explanation. I wrote it down to ask questions later when I get some output with the tree. > To get the tree you want all you have to do is grab the target list ie... > SCons.Script.BUILD_TARGETS and for each item in the list, turn it into a node > ( as it might be a string still) and then you start to iterate it children. > As pointed out the main code for this in SCons.Script.Main.py. you can look > at TreePrinter as a starting point and search for the code that uses it. I inserted "print SCons.Script.BUILD_TARGETS" right before checking condition for interactive mode after SConscript files are read), and it is empty. But SCons is still trying to build something. How it determines those targets? -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
I agree.. sorry that my last e-mail was a little hard to read. I have to remember to proof read when I am running on a few hour of sleep. I always seem to forget when I am half asleep :-) Jason -Original Message- From: Scons-dev [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Wednesday, May 20, 2015 1:12 PM To: SCons developer list Subject: Re: [Scons-dev] Time for a release? Bill, On 20.05.2015 19:37, Bill Deegan wrote: > Dirk, > > Are you suggesting we should merge slots into default and push 2.4? > Or push what we have as 2.4 and then merge slots? > if I got Jason right we don't have to wait for Parts to synchronize our development, so we're free to latch on. I suggest that we now merge the slots branch into default and then release this as 2.4 (as was our original plan). As a next step we will then integrate the stubprocess.py wrapper (and not much more) and then quickly let another release 2.5 follow. Like this, both minor version numbers signal that there is a (somewhat) bigger change, but without changing APIs or breaking existing scripts. After that we're free to work on the 2.7/3.x version which we'll finally release as 3.0. This is what makes the most sense to me right now. > I'm thinking since slots will (could) break compatibility for some > that we should have a major version change? (3.0?) > > If I remember correctly, that was our criteria for major version number > changes. > Unfortunately that means 3.0 may confuse some with thinking python 3.0 is > supported by the new (slots) version. > (No need to discuss python 3.0 here, that's an entirely other ball of > wax) > > *There are 3 broken tests on our win32 buildbot slave. We should > resolve those prior to a release.* > I agree that we should ensure that there are no broken tests left. However I've seen those three tests to be a little flaky under Windows over time anyway. So maybe we should wait what the buildbots say after the "slots" merge? We can then try to fix the tests or get them more robust. In general, it would be good to account for another week or so of general testing, after the "slots" merge (and before releasing v2.4). It would be good if as many developers as possible could then try out the "default/trunk" version again on their projects or in general. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
Bill, On 20.05.2015 19:37, Bill Deegan wrote: Dirk, Are you suggesting we should merge slots into default and push 2.4? Or push what we have as 2.4 and then merge slots? if I got Jason right we don't have to wait for Parts to synchronize our development, so we're free to latch on. I suggest that we now merge the slots branch into default and then release this as 2.4 (as was our original plan). As a next step we will then integrate the stubprocess.py wrapper (and not much more) and then quickly let another release 2.5 follow. Like this, both minor version numbers signal that there is a (somewhat) bigger change, but without changing APIs or breaking existing scripts. After that we're free to work on the 2.7/3.x version which we'll finally release as 3.0. This is what makes the most sense to me right now. I'm thinking since slots will (could) break compatibility for some that we should have a major version change? (3.0?) If I remember correctly, that was our criteria for major version number changes. Unfortunately that means 3.0 may confuse some with thinking python 3.0 is supported by the new (slots) version. (No need to discuss python 3.0 here, that's an entirely other ball of wax) *There are 3 broken tests on our win32 buildbot slave. We should resolve those prior to a release.* I agree that we should ensure that there are no broken tests left. However I've seen those three tests to be a little flaky under Windows over time anyway. So maybe we should wait what the buildbots say after the "slots" merge? We can then try to fix the tests or get them more robust. In general, it would be good to account for another week or so of general testing, after the "slots" merge (and before releasing v2.4). It would be good if as many developers as possible could then try out the "default/trunk" version again on their projects or in general. Regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Wiki is down Re: How to traverse the graph after files are read
Anatoly, I did note on this issue in another email. Whenever someone trys to DOS the wiki the hosting provider takes the wiki offline by chmod'ing the wiki cgi script to not be executable. I'll take a look. -Bill On Wed, May 20, 2015 at 8:54 AM, anatoly techtonik wrote: > On Wed, May 20, 2015 at 6:50 PM, Alexandre Feblot > wrote: > > I did such kind of traversal once: http://pastebin.com/KyEg5ngS > > Maybe that was even based on something found in the wiki. > > BTW, what happened to wiki ? > > > http://www.scons.org/wiki/DeveloperGuide/TestingMethodology#Working_with_fixtures > > CGIWrap Error: Execution of this script not permitted > > > > Execution of (moin.cgi) is not permitted for the following reason: > > Script is not executable. Issue 'chmod 755 filename' > > > > Local Information and Documentation: > > Contact EMail: support@pair.comContact Web Site: http://support.pair.com/ > > Server Data: > > Server Administrator/Contact: scons@pair.comServer Name: > www.scons.orgServer Port: 80Server Protocol: HTTP/1.1Virtual Host: > www.scons.org > > Request Data: > > User Agent/Browser: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 > (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36Request Method: > GETRemote Host: 86.57.247.127Remote Address: 86.57.247.127Remote Port: > 26154Extra Path Info: /DeveloperGuide/TestingMethodology > ___ > Scons-dev mailing list > Scons-dev@scons.org > https://pairlist2.pair.net/mailman/listinfo/scons-dev > ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
Dirk, Are you suggesting we should merge slots into default and push 2.4? Or push what we have as 2.4 and then merge slots? I'm thinking since slots will (could) break compatibility for some that we should have a major version change? (3.0?) If I remember correctly, that was our criteria for major version number changes. Unfortunately that means 3.0 may confuse some with thinking python 3.0 is supported by the new (slots) version. (No need to discuss python 3.0 here, that's an entirely other ball of wax) *There are 3 broken tests on our win32 buildbot slave. We should resolve those prior to a release.* -Bill On Wed, May 20, 2015 at 9:46 AM, Dirk Bächle wrote: > Hi Bill, > > > On 20.05.2015 16:36, Bill Deegan wrote: > >> All, >> >> It's been a while since the last release and a number of fixes are in the >> repo, is it a good time for a release? >> >> I forget where we are with slots? Is that still on a branch or in >> default now? >> >> > the slots branch hasn't been merged yet. From my perspective the current > situation is as follows: > > - I agree that the basic fixes we have would justify a release v2.4 right > now. Current trunk seems to be reasonably > stable > - The slots changes seem to work basically, at least nobody else > complained after we added the getattr > wrapper for backward compatibility. > - Jason Kenny is still examining the impact on Parts. He refactored some > source code, but is still working > on some minor warts here and there...but over all it looked promising. > > So, I'd like to pass the token to Jason to hear what he has to say. The > questions we all should probably discuss together are: > > - Is there more work needed to get Parts going with this new "slotted" > version of SCons? > - If yes, what are the single steps...and also: do we have to block the > release for them? > - What is our basic plan for getting Parts synchronized with SCons again? > In some way, a new Parts release will > depend on SCons v2.4 and won't work with any version lower than that. > How do we tell the user, and how do > we plan the next release of Parts in relation to SCons v2.4? > > There's probably more, but I forgot. ;) > > Best regards, > > Dirk > > ___ > Scons-dev mailing list > Scons-dev@scons.org > https://pairlist2.pair.net/mailman/listinfo/scons-dev > ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
I the question to me. - Is there more work needed to get Parts going with this new "slotted" version of SCons? I turned off the cache logic to help with speed up. This mean parts currently is running as if --load-logic=all was set. Ie load everything. There are two reasons for this.. one is the some code in Parts was saving state via a means that the new __slots__ usage prevents. I have backup for this. So no big deal. Another is that I am trying to look at how to make the data cache logic faster and to use less memory. - If yes, what are the single steps...and also: do we have to block the release for them? I need to sync current work in Parts with main git repro I need to update some tests and samples I think I have a test failure. I need to see what is wrong. I need to setup the main bitbucket "team" so there is a clean url to get at this. Ie the like we have a https://bitbucket.org/scons/scons vs the https://bitbucket.org/%username%/scons. I think I going to use SConsParts as the ideal Parts/part/prt/prts names is used as a private repro. Add a basic MD file for stuff like bootstrapping based on using the newer pip based setup.py, etc.. Need to see ( you have thoughts) if I can get parts to depend on SCons in some way with the pip script.. also need to see how to register Parts on pypy ( should be easy.. just need to take time) Nothing here I believe should block a Scons drop, as I have to support older drops at the moment. Until I make a new drop of Parts people will have to use 2.1 to 2.3 versions - What is our basic plan for getting Parts synchronized with SCons again? In some way, a new Parts release will depend on SCons v2.4 and won't work with any version lower than that. How do we tell the user, and how do we plan the next release of Parts in relation to SCons v2.4? So technically I can support scons 2.1 and above. I plan to drop support for anything but 2.4 going forward after the next drop. I have to support 2.1 at the moment internally because of an issue with builder copy logic on clone that started in 2.2 with an internal build at work. ( I am pushing them to address there custom builder so this issues does not happen anymore.. but it has been slow) I personally want to try to sync on issues I pointed out in my last e-mail to you about the directory copy issue I saw ( I did not see any response yet on that e-mail to you yet.. I can resend if needed).. and the node API as it is as I would like to see if we can clean up some stuff to get a better "dev" level API for stuff like getting various state that we can document to help make it easier for general user that are making builder, etc . Some basic stuff also like getting in a symlink node and some patches in some place in SCons that I have in Parts to get hardlink and symlink support on windows would also be really nice to get in as it general useful. Given the improvements in have seen in the __slots__ version of SCons from Dirk.. I would really push to see if we can use in 2.4. This is good step forward in memory and speed improvements Jason -Original Message- From: Scons-dev [mailto:scons-dev-boun...@scons.org] On Behalf Of Dirk Bächle Sent: Wednesday, May 20, 2015 11:46 AM To: scons-dev@scons.org Subject: Re: [Scons-dev] Time for a release? Hi Bill, On 20.05.2015 16:36, Bill Deegan wrote: > All, > > It's been a while since the last release and a number of fixes are in the > repo, is it a good time for a release? > > I forget where we are with slots? Is that still on a branch or in default > now? > the slots branch hasn't been merged yet. From my perspective the current situation is as follows: - I agree that the basic fixes we have would justify a release v2.4 right now. Current trunk seems to be reasonably stable - The slots changes seem to work basically, at least nobody else complained after we added the getattr wrapper for backward compatibility. - Jason Kenny is still examining the impact on Parts. He refactored some source code, but is still working on some minor warts here and there...but over all it looked promising. So, I'd like to pass the token to Jason to hear what he has to say. The questions we all should probably discuss together are: - Is there more work needed to get Parts going with this new "slotted" version of SCons? - If yes, what are the single steps...and also: do we have to block the release for them? - What is our basic plan for getting Parts synchronized with SCons again? In some way, a new Parts release will depend on SCons v2.4 and won't work with any version lower than that. How do we tell the user, and how do we plan the next release of Parts in relation to SCons v2.4? There's probably more, but I forgot. ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Time for a release?
Hi Bill, On 20.05.2015 16:36, Bill Deegan wrote: All, It's been a while since the last release and a number of fixes are in the repo, is it a good time for a release? I forget where we are with slots? Is that still on a branch or in default now? the slots branch hasn't been merged yet. From my perspective the current situation is as follows: - I agree that the basic fixes we have would justify a release v2.4 right now. Current trunk seems to be reasonably stable - The slots changes seem to work basically, at least nobody else complained after we added the getattr wrapper for backward compatibility. - Jason Kenny is still examining the impact on Parts. He refactored some source code, but is still working on some minor warts here and there...but over all it looked promising. So, I'd like to pass the token to Jason to hear what he has to say. The questions we all should probably discuss together are: - Is there more work needed to get Parts going with this new "slotted" version of SCons? - If yes, what are the single steps...and also: do we have to block the release for them? - What is our basic plan for getting Parts synchronized with SCons again? In some way, a new Parts release will depend on SCons v2.4 and won't work with any version lower than that. How do we tell the user, and how do we plan the next release of Parts in relation to SCons v2.4? There's probably more, but I forgot. ;) Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
Let me give you an understanding of this as I understand it based on my work and talks I had with Steve Knight... Starting with the basics.. some of this you already know.. Scons has nodes. These node are used to represent items. We have *Value nodes that can holds some value like a string, list, some python object, *Alias nodes that in general are used to represent a set of other nodes ( but you can give an alias an action to run ). This does not have a state that is stored in SCons and since different target can cause alias to have a different set of children, it being out of date is based on the current read state of the build files. *file nodes which are nodes that are based on what is on disk. In this set we have currently: ** Dir node which represent some directory on disk ** File node which is some file ** Entry node which is either a file or directory, but is assumed to be unknown at the current moment. It is morphed into a Dir or File node later. In general if it is not a directory Scons assumes it is a file. Sometimes this is wrong and will lead to an error about looking up node X as a file but it is really a directory error message. In this cases you normally work around it by forcing it to Dir node when you provide the item to the builder. ** in Parts we have added a FileSymbolicLink node object to help deal with symlinks on disk. Given this any node can have a builder associated to it. Given the node does not exist SCons will call the builder to make it ( directory can be funny here.. as I noted a bug I have found to Dirk with directory nodes and copy logic). Any node that does not have a builder and does not exist leads to a "Don't know how to build XYZ" message from SCons. Normally a node without a builder is source/dependent node that is added as an explicit dependency via a builder call, Depends() call or via a scanner. The FS object only holds file based nodes ( Dir, File, Entry and FileSymbolicLink [given Parts]). When you iterate the FS object you can generally view it as a virtual file system. When I had talked to Steve knight years ago, he told me that he had hoped to make the nodes a portable way to process files independent of the Python file API. Currently SCons is not there, but much of the plumbing for this exists and is used internally. Anyways you can iterate the FS object as a virtual file system and see that directory X contain a set of files and or directories or entry objects. These items may or may not exists on disk yet. This is allows Glob to return object that don't exist on disk yet as before the glob call happened something defined a build action that was to output some file based node. Call to add a node after a Glob call means Globs does not see it as a dependent, or least not until it exists on disk from a previous build run. Whenever you make a node such as File(readme.txt) scons will try to lookup this code in the FS object and return the existing node that represents this file if it already exists, if it does not exist it make a new node adds it to the FS object in the correct place and returns the reference to that node. ( given a variant directory, etc are being used this may be many nodes for the different locations) To get the tree you want all you have to do is grab the target list ie... SCons.Script.BUILD_TARGETS and for each item in the list, turn it into a node ( as it might be a string still) and then you start to iterate it children. As pointed out the main code for this in SCons.Script.Main.py. you can look at TreePrinter as a starting point and search for the code that uses it. Hope this helps Jason As a side note.. the main point of the Gregs taskmaster TNG was to change the logic in SCons to make a builder tree based on the node tree and use the builder tree to build node as this tree was much smaller and removes lots of wasted iteration we have when iterating node directly. This is really seen in cases such as package builders that depend on children as SCons has to iterate the children node, vs just understand that all the builders for the children are up to date ( which is a difference of a few builder objects vs ideally 1000s... 100,000 of node objects. -Original Message- From: Scons-dev [mailto:scons-dev-boun...@scons.org] On Behalf Of anatoly techtonik Sent: Wednesday, May 20, 2015 9:25 AM To: SCons developer list Subject: Re: [Scons-dev] How to traverse the graph after files are read On Wed, May 20, 2015 at 4:59 PM, Kenny, Jason L wrote: >>>Are there two separate FS objects for that? > > No there is one global fs that is shared with all environment objects. > > After the Sconstruct are read in this will hold all nodes that are known to > the SCons and their relationships. The scanner have not been run yet, so > there might be some node that are still to be added as dependent children. I > think a call to get_children() will call the scanners however and return all >
[Scons-dev] Wiki is down Re: How to traverse the graph after files are read
On Wed, May 20, 2015 at 6:50 PM, Alexandre Feblot wrote: > I did such kind of traversal once: http://pastebin.com/KyEg5ngS > Maybe that was even based on something found in the wiki. BTW, what happened to wiki ? http://www.scons.org/wiki/DeveloperGuide/TestingMethodology#Working_with_fixtures CGIWrap Error: Execution of this script not permitted Execution of (moin.cgi) is not permitted for the following reason: Script is not executable. Issue 'chmod 755 filename' Local Information and Documentation: Contact EMail: support@pair.comContact Web Site: http://support.pair.com/ Server Data: Server Administrator/Contact: scons@pair.comServer Name: www.scons.orgServer Port: 80Server Protocol: HTTP/1.1Virtual Host: www.scons.org Request Data: User Agent/Browser: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36Request Method: GETRemote Host: 86.57.247.127Remote Address: 86.57.247.127Remote Port: 26154Extra Path Info: /DeveloperGuide/TestingMethodology ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Wed, May 20, 2015 at 11:50 AM, Alexandre Feblot wrote: > I did such kind of traversal once: http://pastebin.com/KyEg5ngS > Maybe that was even based on something found in the wiki. I'm not 100% certain, but I believe calling node.children() can invoke scanners and other things that should normally be done in particular orders by the build process rather than during the SConstruct-reading phase. Of course it is likely to be more complete than node.sources. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
I did such kind of traversal once: http://pastebin.com/KyEg5ngS Maybe that was even based on something found in the wiki. 2015-05-20 17:43 GMT+02:00 Gary Oberbrunner : > > On Wed, May 20, 2015 at 11:25 AM, anatoly techtonik > wrote: > >> I want to get target Node based on name passed from command line. >> How to do that? >> > > node = File(name) > > How to get list of all targets to be built? User guide: > http://www.scons.org/doc/HTML/scons-user/ch10s03.html#idp3074280 > > the BUILD_TARGETS variable contains a list of the command-line targets, > if any were specified, and if no command-line targets were specified, it > contains a list of the targets specified via the Default method or > function > > > Note that it contains a list of Nodes, so you don't need to do anything, > just iterate over it. > > -- > Gary > > ___ > Scons-dev mailing list > Scons-dev@scons.org > https://pairlist2.pair.net/mailman/listinfo/scons-dev > > ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Wed, May 20, 2015 at 11:25 AM, anatoly techtonik wrote: > I want to get target Node based on name passed from command line. > How to do that? > node = File(name) How to get list of all targets to be built? User guide: http://www.scons.org/doc/HTML/scons-user/ch10s03.html#idp3074280 the BUILD_TARGETS variable contains a list of the command-line targets, if any were specified, and if no command-line targets were specified, it contains a list of the targets specified via the Default method or function Note that it contains a list of Nodes, so you don't need to do anything, just iterate over it. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Wed, May 20, 2015 at 5:46 PM, Gary Oberbrunner wrote:
> On Wed, May 20, 2015 at 9:47 AM, anatoly techtonik
> wrote:
>>
>> If so, then that means that every target should
>> be a filesystem object?
>>
>> What is target then? If it is a name, how a lookup if made to locate it in
>> FS tree?
>
>
> Every target (and source, and intermediate) is a Node. The Node object
> knows everything about that node, including name, path, parent, sources,
> builder, build state, and so on. There is no "lookup" in the sense you're
> talking about.
>
> If you want to get the Node object corresponding to a filename, just call
> f=File("myfile") or f=File("/a/b/c/myfile") etc.
I want to get target Node based on name passed from command line.
How to do that?
If there is no name, how do I find default node to start from? From what I
see in Main.py, I have only fs object at my disposal.
--
anatoly t.
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Wed, May 20, 2015 at 9:47 AM, anatoly techtonik
wrote:
> If so, then that means that every target should
> be a filesystem object?
>
> What is target then? If it is a name, how a lookup if made to locate it in
> FS tree?
>
Every target (and source, and intermediate) is a Node. The Node object
knows everything about that node, including name, path, parent, sources,
builder, build state, and so on. There is no "lookup" in the sense you're
talking about.
If you want to get the Node object corresponding to a filename, just call
f=File("myfile") or f=File("/a/b/c/myfile") etc.
--
Gary
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev
[Scons-dev] Time for a release?
All, It's been a while since the last release and a number of fixes are in the repo, is it a good time for a release? I forget where we are with slots? Is that still on a branch or in default now? -Bill ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Tutorial for Linux versioned libraries
Anatoly, The docs in hg have some updated text on this. They'll be on the site when the next release is published. Feel free to take a look and send some pull requests if you have suggestions to improve them. -Bill On Wed, May 20, 2015 at 12:07 AM, anatoly techtonik wrote: > On Tue, May 19, 2015 at 10:59 PM, Kenny, Jason L > wrote: > > I think a little clarification is needed. > > > > > > > > SCons install() api makes an install sandbox. There is a list that holds > all > > the install files. When you call Install() those files get added to the > > internal list. This information is used by the Package() and > > GetInstallFiles() api in SCons to allow a package to be created. One can > get > > this to work. > > This definitely needs some better docs and maybe even rename it. I thought > that Install actually copies files. > > Another mystery is when I specify different prefix for install target. > SCons rebuils > the libraries. Why? > > > Anyways ( minus the Parts extension) I hope the understand that Install() > > setups an install sandbox that is used by the package() API to create > some > > install package based on what was installed. > > "base on what was installed"? Do you mean that if the current system does > have some files already installed, the created package will miss those > files? > > -- > anatoly t. > ___ > Scons-dev mailing list > Scons-dev@scons.org > https://pairlist2.pair.net/mailman/listinfo/scons-dev > ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Wed, May 20, 2015 at 4:59 PM, Kenny, Jason L wrote: >>>Are there two separate FS objects for that? > > No there is one global fs that is shared with all environment objects. > > After the Sconstruct are read in this will hold all nodes that are known to > the SCons and their relationships. The scanner have not been run yet, so > there might be some node that are still to be added as dependent children. I > think a call to get_children() will call the scanners however and return all > explicit and implicit node dependents for a given node. So, I thought that FS object is just used to access filesystem. Not it looks like an FS is not only a namespace and cache for filesystem access, but also a data/state storage for the build graph. Perhaps that makes SCons more memory efficient, but it is hard to wrap my head around it. Now the parsing code from Main.py start to make sense: SCons.Script._SConscript._SConscript(fs, script) So, this one parses the script and updates fs tree, right? ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
>>Are there two separate FS objects for that? No there is one global fs that is shared with all environment objects. After the Sconstruct are read in this will hold all nodes that are known to the SCons and their relationships. The scanner have not been run yet, so there might be some node that are still to be added as dependent children. I think a call to get_children() will call the scanners however and return all explicit and implicit node dependents for a given node. Jason -Original Message- From: Scons-dev [mailto:scons-dev-boun...@scons.org] On Behalf Of anatoly techtonik Sent: Wednesday, May 20, 2015 8:47 AM To: SCons developer list Subject: Re: [Scons-dev] How to traverse the graph after files are read On Wed, May 20, 2015 at 3:36 PM, Gary Oberbrunner wrote: > On Wed, May 20, 2015 at 6:43 AM, anatoly techtonik > > wrote: >> >> The confusing part is that >> node graph is derived from filesystem. > > > FS represents the filesystem, both on disk and what will exist once > all targets are built. Are there two separate FS objects for that? > I didn't say start from the filesystem, I said start from your target(s). > > myprog=Program(...) > ... > # at end of SConstruct: > visit_node_sources_recursively(myprog) > > you just have to write visit_node_sources_recursively, following each > node's .sources list. myprog is a File node (actually a list of > those). Nowhere should you need FS. First, I don't need to do this from SConstruct. I am looking how to add new debug option to scons itself. So, there is no graph I can dump? Just a list of targets that are being traversed using sources? If so, then that means that every target should be a filesystem object? What is target then? If it is a name, how a lookup if made to locate it in FS tree? -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Wed, May 20, 2015 at 3:36 PM, Gary Oberbrunner wrote: > On Wed, May 20, 2015 at 6:43 AM, anatoly techtonik > wrote: >> >> The confusing part is that >> node graph is derived from filesystem. > > > FS represents the filesystem, both on disk and what will exist once all > targets are built. Are there two separate FS objects for that? > I didn't say start from the filesystem, I said start from your target(s). > > myprog=Program(...) > ... > # at end of SConstruct: > visit_node_sources_recursively(myprog) > > you just have to write visit_node_sources_recursively, following each node's > .sources list. myprog is a File node (actually a list of those). Nowhere > should you need FS. First, I don't need to do this from SConstruct. I am looking how to add new debug option to scons itself. So, there is no graph I can dump? Just a list of targets that are being traversed using sources? If so, then that means that every target should be a filesystem object? What is target then? If it is a name, how a lookup if made to locate it in FS tree? -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Wed, May 20, 2015 at 6:43 AM, anatoly techtonik wrote: > The confusing part is that > node graph is derived from filesystem. > FS represents the filesystem, both on disk and what will exist once all targets are built. I didn't say start from the filesystem, I said start from your target(s). myprog=Program(...) ... # at end of SConstruct: visit_node_sources_recursively(myprog) you just have to write visit_node_sources_recursively, following each node's .sources list. myprog is a File node (actually a list of those). Nowhere should you need FS. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Wed, May 20, 2015 at 10:26 AM, Dirk Bächle wrote: > On 20.05.2015 09:01, anatoly techtonik wrote: >> On Tue, May 19, 2015 at 3:19 PM, Gary Oberbrunner >> wrote: >>> >>> At the end of your SConstruct, start with the target nodes and recurse >>> into >>> each node.sources . > > > the "recurse into" is what's important here, Anatoly. Yes. The problem is to identify the start node. The confusing part is that node graph is derived from filesystem. >>> [...] >> >> >> But the logic what happens here and how FS object is transformed >> into DAG, escapes me. > > > There is no real DAG stored anywhere within the code, like an object that > you can grab and visit its attributes or traverses its nodes. > With the call > > else: > > # Build the targets > nodes = _build_targets(fs, options, targets, target_top) > if not nodes: > > SCons starts to dynamically expand the list of children for each given > target (which includes the so-far detected children of other targets as > well). So I don't think that you will get at what you actually want...but > you can try to do a "recursion" light as Gary suggested. So, if I understood this correctly, the filesystem (fs) tree is the draft of final graph. E.g. fs object is not just a mirror of filesystem tree - it is a selective set of files and dirs referenced in SConstruct, right? I am interested in getting the preliminary graph that is available after reading SConstruct files. The graph that doesn't add any additional processing. Is it possible to get it without triggering expansion logic? Then I may be interested to track graph transformation during the build phase. But that's secondary. ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
Hi all, On 20.05.2015 09:01, anatoly techtonik wrote: On Tue, May 19, 2015 at 3:19 PM, Gary Oberbrunner wrote: At the end of your SConstruct, start with the target nodes and recurse into each node.sources . the "recurse into" is what's important here, Anatoly. [...] But the logic what happens here and how FS object is transformed into DAG, escapes me. There is no real DAG stored anywhere within the code, like an object that you can grab and visit its attributes or traverses its nodes. With the call else: # Build the targets nodes = _build_targets(fs, options, targets, target_top) if not nodes: SCons starts to dynamically expand the list of children for each given target (which includes the so-far detected children of other targets as well). So I don't think that you will get at what you actually want...but you can try to do a "recursion" light as Gary suggested. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] Tutorial for Linux versioned libraries
On Tue, May 19, 2015 at 10:59 PM, Kenny, Jason L wrote: > I think a little clarification is needed. > > > > SCons install() api makes an install sandbox. There is a list that holds all > the install files. When you call Install() those files get added to the > internal list. This information is used by the Package() and > GetInstallFiles() api in SCons to allow a package to be created. One can get > this to work. This definitely needs some better docs and maybe even rename it. I thought that Install actually copies files. Another mystery is when I specify different prefix for install target. SCons rebuils the libraries. Why? > Anyways ( minus the Parts extension) I hope the understand that Install() > setups an install sandbox that is used by the package() API to create some > install package based on what was installed. "base on what was installed"? Do you mean that if the current system does have some files already installed, the created package will miss those files? -- anatoly t. ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
Re: [Scons-dev] How to traverse the graph after files are read
On Tue, May 19, 2015 at 3:19 PM, Gary Oberbrunner wrote: > At the end of your SConstruct, start with the target nodes and recurse into > each node.sources . This is incomplete of course if you have any source > generation or emitters etc. But it may be useful depending on what you're > doing. I am in the middle of Main.py - at this point: >> >>> https://bitbucket.org/scons/scons/src/95b566e4baf0253e1b36b8a9e00a97cd84d22566/src/engine/SCons/Script/Main.py?at=default#cl-1091 There is FS object named fs, but I don't see why is it is a build graph. Later, the node list is derived from it at line 1204: nodes = [_f for _f in map(Entry, targets) if _f] Entry is: def Entry(x, ltop=lookup_top, ttop=target_top, fs=fs): if isinstance(x, SCons.Node.Node): node = x else: node = None # Why would ltop be None? Unfortunately this happens. if ltop is None: ltop = '' # Curdir becomes important when SCons is called with -u, -C, # or similar option that changes directory, and so the paths # of targets given on the command line need to be adjusted. curdir = os.path.join(os.getcwd(), str(ltop)) for lookup in SCons.Node.arg2nodes_lookups: node = lookup(x, curdir=curdir) if node is not None: break if node is None: node = fs.Entry(x, directory=ltop, create=1) if ttop and not node.is_under(ttop): if isinstance(node, SCons.Node.FS.Dir) and ttop.is_under(node): node = ttop else: node = None return node But the logic what happens here and how FS object is transformed into DAG, escapes me. ___ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev
