Re: [testing][cruise-control] how testing infrastructure should be improved

[Geir Magnusson Jr.]

On Jan 30, 2007, at 5:28 AM, Vladimir Ivanov wrote:
On 1/29/07, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:


> OK. One minor note, some time ago on the dev@ list somebody ask  
to add
> 'server jit' mode. Is it OK to add it to the current CC ('short'
> cycle)?

Sure.  Why not.

After staring at my two systems for a couple of days now, I've come
to the conclusion that our short isn't short enough.  I think we need
a "shorter" that simply builds stuff.  No tests.

That would be quick "canary in the coalmine" test to help catch
broken builds, before such breakage affects the longer running
systems, because right now, the cycle is fairly long just for unit
tests for drlvm and classlib

geir


My timings are:
WinXP: classlib ~17 min, drlvm ~10, cl_tests ~34, drlvm_tests ~55
Linux ia32: ~15, ~11, ~40, ~50min
Linux em64t: ~11, ~11, ~120, ~50min

So, the cycle duration is about 2hours (I hope we can switch Linux  
em64t
from 'perTest' to 'once' mode soon :)).

I'm running x86_64 (remember, we decided to use that token rather  
than em64t :) in 'once' mode - I didn't ahve to switch to perTest.

If we define the 'short' cycle as 'build the classlib and the  
drlvm' the
minimal cycle will be reduced up to ~30 min. In this case we will have
'short' cycle ~30 min, user defined 'medium' cycle and all-included  
'long'.
Is it OK?

I think so.  I think we need better names that are descriptive rather  
than relative, so we can add more w/o resorting to things like "ultra  
medium short long" or whatever...

so maybe :

"build-profile" : just build classlib, drlvm and tools
"unit_profile" : runs the drlvm tests and classlib tests, can be used  
after "build-profile"
"interative_profile" : can run after either the above, does iterative  
testing
"functional1_profile" : ...
etc

?

I don't know CC well enough - can I have these pre-set things, and  
tell CC to do "build-profile" only, or do build-profile, and then if  
that passes, do the unit-profile?

geir


thanks, Vladimir



>
>>
>> >
>> > It requires some 'standard' interface for all integrated  
scripts. I
>> > like
>> > classlib interface so how about:
>> >  - call of "ant setup;ant" will run all available scripts;
>> >  - call of "ant -Dmodule=hit setup;ant" will run current  
version of
>> > CC –
>> > Harmony integration tests;
>> >  - call of "ant -Dmodule=eut setup;ant" will run Eclipse   
unit test
>> > etc.
>> >
>>
>> > Note, in this case each module should implement proper 'setup'
>> > target and
>> > has configuration for CC. The root-script will iterate over all
>> > modules to
>> > call their 'setup' and this setup should include whole test  
setup
>> > (downloading software, adding modules cc-configuration to  
working
>> > configuration etc).
>> >
>> > Is it OK?
>>
>> I dunno - this sounds like disjoint and separate CC runs,  
rather than
>> a CC run with multiple projects.
>>
>> For example, I'd like to have a set of "modules", which would be
>> incomplete cc config files, that somehow get glommed into a  
bigger cc
>> file - maybe the config.xml would have some kind of include
>> mechanism.
>>
>> Suppose that we have :
>>
>> trunk/cc/
>>     config.xml
>>     modules/
>>        default_module.xml
>>        hut_module.xml
>>        eut_module.xml
>>        iterative_module.xml
>>        dacapo_module.xml
>>        specjbb_module.xml
>>        short_module.xml
>>        medium_module.xml
>>        long_modules.xml
>>
>> so then I could do :
>>
>>   $ ant
>>
>> to get what we have now - runs the default module - or
>>
>>   $ ant -Dmodules=hut,eut,dacapo
>>
>> to run those...
>>
>> Something like "medium_module.xml" would look like  (the  
following is
>> 'psuedo-code') :
>>
>>     <includeconfig name="default_module.xml"/>
>>     <includeconfig name="dacapo_module.xml"/>
>>
>>
>> so that you can nest this as you want.
>>
>> >
>> > If nobody objects I'll start restructuring of buildtest  
module and
>> > will try
>> > to integrate one from extensions.
>>
>> Please describe how you want to do it.
>
>
> I think about following option: in the root file we have predefined
> string.
> Something like 'modules=hut'. In the 'long' mode CC will iterate
> over all
> entries. The medium cycle depend on users wishes and defined
> through the
> command line like: 'ant -Dmodules=hut,iterative'. Each module has
> predefined
> target (for example, 'setup'). In this target script should
> download all
> software, apply all patches, add module configuration to the
> current CC
> configuration (just copy content of the module configuration file
> to the end
> of current configuration) etc. After 'ant <...> setup' we will have
> ready-to-run system with user-defined configuration.
>
>
>> >
>> >
>> > Thanks, Vladimir
>> >
>> >
>> >
>> > PS I think the resulting structure should be easy to extend  
and may
>> > looks
>> > like this:
>> >
>> > buildtest
>> >
>> > |--config  (default CC configuration to build classlib and  
DRLVM)
>> >
>> > |--hit (CC configuration to run Harmony classlib&DRLVM tests)
>> >
>> > |--eclipse
>> >
>> >     |-- eut (setup and CC configuration to run eclipse non-
>> interactive
>> > tests)
>> >
>> >     |-- eclipse3.1.1
>> >
>> >         |-- some scenario
>> >
>> > |-- build.xml (common setup + call of module's 'setup')
>>
>>
>> Interesting.  I think one issue is that it seems like heavy  
lifting
>> to add a new module - each module becomes a "peer".   What do you
>> think of the other approach above?
>>
>> Either way, we don't want hit, eclipse, etc as peers.  If  
anything,
>> they should go in a modules/ directory...
>
>
>
> For each module we have at least 2 files: cc-config and build file.
> But in
> some cases we will have some additional files (patches etc). For
> example, script for testing of JEdit application (issue 3012) has
> about 15
> files. For me is better to store all staff in one place instead of
> having
> parallel structure.
>
> thanks, Vladimir
>
> geir
>>
>>