I'm using: PYTHONPATH=$PWD/QMTest:$PWD/src/engine:$PWD/testing/framework python src/engine/SCons/EnvironmentValuesSubstTests.py
In my current sandbox. Most unit tests (under src/engine) are runnable that way. The tests under test/... may not be as easy to run without runtest.py. Still depending on some SCons testing code. -Bill On Wed, Dec 20, 2017 at 12:13 PM, Andrew Featherstone < andrew.featherst...@cantab.net> wrote: > Are you able to provide a one liner that helps clarify this? > > python -m unittest discover -s QMTest --pattern "Test*.py" produces a very > quick failure. > > Andrew > > On 20 December 2017 at 16:39, Bill Deegan <b...@baddogconsulting.com> > wrote: > >> 4. We're already using unittest. QMtest is just the name of the directory >> at this point. (You used to have to install qmtest to run the test suite) >> >> On Wed, Dec 20, 2017 at 11:28 AM, Daniel Moody <dmoody...@gmail.com> >> wrote: >> >>> 1. are you referring to the use of runtest.py command, or how the >>> various builders and environments are started? >>> 2. agreed >>> 3. With the free CI services you are limited to whatever they offer, and >>> at this point splitting up the tests is a workaround, instead of rewriting >>> the entire test suite. Even then, scaling it up, it may still be necessary >>> to split up tests across jobs. >>> 4. agreed, a well known standard python testing framework would be >>> great. I like pytest. >>> >>> My goal currently is to at least get basic CI on github working with as >>> little rework as possible. >>> >>> On Wed, Dec 20, 2017 at 11:20 AM, Bill Deegan <b...@baddogconsulting.com >>> > wrote: >>> >>>> >>>> >>>> On Wed, Dec 20, 2017 at 11:05 AM, Andrew Featherstone < >>>> andrew.featherst...@cantab.net> wrote: >>>> >>>>> My preferred way of working on this problem: >>>>> >>>>> >>>>> 1. Get parity with what happens at http://buildbot.scons.org/#/. I >>>>> can't work out from there how to read the actual commands being run in >>>>> the >>>>> CI job, so I'm mostly replicating what happens in the Travis-based >>>>> builds. >>>>> >>>>> >>>> Basically: >>>> 1) git clone >>>> 2) python2.7 runtest.py -a >>>> >>>> Current docs and packages aren't being built (but should be) >>>> >>>>> >>>>> 1. Get all of scons' build processes onto cloud-based CI. >>>>> Everything that is at http://scons.org/pages/download.html should >>>>> come out of a CI job. >>>>> >>>>> Sounds good. >>>> We used to build dev packages an make available a while back. >>>> setup.py needs a rewrite and a few other pieces of logic can be removed. >>>> >>>> >>>>> 1. >>>>> 2. Get the tests running faster. This needs some profiling, but >>>>> from the logs it reads like every test is run in a separate process, >>>>> which >>>>> feels pretty expensive to me. There's also references to 2008-era blog >>>>> posts which I take with a large grain of salt. We need the tests >>>>> running as >>>>> a pre-requisite of course. Devising way to slice up long running tests >>>>> feels like solving the wrong problem. >>>>> >>>>> My guess is that most of the tests could have >>>> DefaultEnvironment(tools=[]), and all Environment()'s could have a more >>>> limited list of tools. That would make the test faster to much faster >>>> depending on the test. Likely more speedup on win32 than on linux due to >>>> msvc detection logic. >>>> >>>> >>>>> >>>>> 1. Move away from a QMTest to something more commonplace. The >>>>> canonical home of QMTest seems to be https://github.com/MentorEmbed >>>>> ded/qmtest, last updated in 2011. I think it's time to look at >>>>> something else, e.g. plain old unittest or pytest. >>>>> >>>>> QMtest is basically gone. Tests are basically unittests now. >>>> >>>> >>>>> >>>>> 1. >>>>> >>>>> Perhaps it's worth splitting off another thread for roadmapping? >>>>> >>>> >>>> Sure. >>>> >>>> >>>> >>>>> >>>>> Andrew >>>>> >>>>> On 20 December 2017 at 14:00, Daniel Moody <dmoody...@gmail.com> >>>>> wrote: >>>>> >>>>>> What about this: >>>>>> >>>>>> --interval-start FLOAT Percentile as float (0.0 - 1.0) of what >>>>>> test to start on from the >>>>>> determined list of all >>>>>> tests >>>>>> --interval-end FLOAT Percentile as float (0.0 - 1.0) of what >>>>>> test to start on from the >>>>>> determined list of all >>>>>> tests >>>>>> >>>>>> interval-start default is 0 >>>>>> interval-end default is 1 >>>>>> >>>>>> A little more flexible I think. >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Dec 20, 2017 at 8:41 AM, Bill Deegan < >>>>>> b...@baddogconsulting.com> wrote: >>>>>> >>>>>>> Maybe add a flag directly to runtest.py to split up the tests. >>>>>>> --test_mod 4 --test_index 0..3 ? >>>>>>> >>>>>>> Or something to that affect. >>>>>>> >>>>>>> On Wed, Dec 20, 2017 at 8:27 AM, Daniel Moody <dmoody...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hey Andrew, >>>>>>>> >>>>>>>> I also was working on this. I think in your case the builds timed >>>>>>>> out due to issues with vswhere: >>>>>>>> https://github.com/Microsoft/vswhere/issues/87 >>>>>>>> https://github.com/Microsoft/vswhere/issues/91 >>>>>>>> >>>>>>>> In this case we will need to use the Visual Studio 2017 image. >>>>>>>> >>>>>>>> Also just to note; the issue mentioned above is also an issue with >>>>>>>> SCons MSVS detection in general now for SCons 3 and newer since that is >>>>>>>> when it was switched to use vswhere. >>>>>>>> >>>>>>>> I liked some of the things you had in your script so I took those >>>>>>>> and merge them into my script. I also implemented a script that will >>>>>>>> split >>>>>>>> the build up into multiple jobs like the travis script does, however I >>>>>>>> have >>>>>>>> not been able to get appveyor to do multi-line scripts correctly so it >>>>>>>> is >>>>>>>> all in a one liner at the moment: >>>>>>>> https://github.com/dmoody256/scons/blob/AppveyorCI/.appveyor.yml >>>>>>>> https://ci.appveyor.com/project/dmoody256/scons/build/1.0.53 >>>>>>>> >>>>>>>> Currently python 3 will fail from several tests so I have them >>>>>>>> commented out. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Dec 20, 2017 at 8:00 AM, Bill Deegan < >>>>>>>> b...@baddogconsulting.com> wrote: >>>>>>>> >>>>>>>>> Parallel should help. >>>>>>>>> >>>>>>>>> On my buildbot worker (with 2 other builds running single threaded >>>>>>>>> tests) it takes 2:05. >>>>>>>>> So on a reasonably modern machine, -j2 should finish in under an >>>>>>>>> hour if not, try -j3? >>>>>>>>> >>>>>>>>> Or we can split up runs as we've done with the travis run.. >>>>>>>>> >>>>>>>>> -Bill >>>>>>>>> >>>>>>>>> On Wed, Dec 20, 2017 at 5:54 AM, Andrew Featherstone < >>>>>>>>> andrew.featherst...@cantab.net> wrote: >>>>>>>>> >>>>>>>>>> I've been trying to get AppVeyor working for Windows-based CI, >>>>>>>>>> but we're hitting their 1 hour time limit (see >>>>>>>>>> https://github.com/ajf58/scons/blob/appveyor/.appveyor.yml and >>>>>>>>>> https://ci.appveyor.com/project/ajf58/scons. >>>>>>>>>> >>>>>>>>>> My next pass at this will be trying to run the unit tests in >>>>>>>>>> parallel (as the Windows VM has two cores available). Either that, >>>>>>>>>> or we >>>>>>>>>> split the job matrix a different way, e.g. run tests grouped by >>>>>>>>>> something >>>>>>>>>> else other than Python version. >>>>>>>>>> >>>>>>>>>> Thoughts? >>>>>>>>>> >>>>>>>>>> Andrew >>>>>>>>>> >>>>>>>>>> On 18 December 2017 at 22:51, Bill Deegan < >>>>>>>>>> b...@baddogconsulting.com> wrote: >>>>>>>>>> >>>>>>>>>>> Daniel, >>>>>>>>>>> >>>>>>>>>>> Can we get travis to test with py2.7, 3.5, and 3.6 ? >>>>>>>>>>> >>>>>>>>>>> -Bill >>>>>>>>>>> >>>>>>>>>>> On Wed, Dec 6, 2017 at 12:03 AM, Bill Deegan < >>>>>>>>>>> b...@baddogconsulting.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> That's pretty cool. >>>>>>>>>>>> I'll try to get the coverage hooked up soon. >>>>>>>>>>>> That'll also be very useful.. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Dec 5, 2017 at 8:27 PM, Jonathon Reinhart < >>>>>>>>>>>> jonathon.reinh...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yes, it should automatically do that. >>>>>>>>>>>>> >>>>>>>>>>>>> See this (merged) PR from one of my projects: >>>>>>>>>>>>> https://github.com/JonathonReinhart/scuba/pull/98 >>>>>>>>>>>>> >>>>>>>>>>>>> Towards the bottom you'll see a "View Details" button. >>>>>>>>>>>>> Clicking that will expand a box showing the results of all the >>>>>>>>>>>>> "checks" that ran. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Dec 5, 2017 at 11:13 PM, Bill Deegan < >>>>>>>>>>>>> b...@baddogconsulting.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Is there a way to get travis to post the results back into >>>>>>>>>>>>>> the pull request? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Scons-dev mailing list >>>>>>>>>>>>> Scons-dev@scons.org >>>>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Scons-dev mailing list >>>>>>>>>>> Scons-dev@scons.org >>>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Scons-dev mailing list >>>>>>>>>> Scons-dev@scons.org >>>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Scons-dev mailing list >>>>>>>>> Scons-dev@scons.org >>>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Scons-dev mailing list >>>>>>>> Scons-dev@scons.org >>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Scons-dev mailing list >>>>>>> Scons-dev@scons.org >>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Scons-dev mailing list >>>>>> Scons-dev@scons.org >>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Scons-dev mailing list >>>>> Scons-dev@scons.org >>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Scons-dev mailing list >>>> Scons-dev@scons.org >>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>> >>>> >>> >>> _______________________________________________ >>> Scons-dev mailing list >>> Scons-dev@scons.org >>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>> >>> >> >> _______________________________________________ >> Scons-dev mailing list >> Scons-dev@scons.org >> https://pairlist2.pair.net/mailman/listinfo/scons-dev >> >> > > _______________________________________________ > Scons-dev mailing list > Scons-dev@scons.org > https://pairlist2.pair.net/mailman/listinfo/scons-dev > >
_______________________________________________ Scons-dev mailing list Scons-dev@scons.org https://pairlist2.pair.net/mailman/listinfo/scons-dev