Re: [testing][cruise-control] how testing infrastructure should be improved

[Geir Magnusson Jr.]

On Jan 30, 2007, at 12:17 PM, Vladimir Ivanov wrote:
On 1/30/07, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:


I think so.  I think we need better names that are descriptive rather
than relative, so we can add more w/o resorting to things like "ultra
medium short long" or whatever...

so maybe :

"build-profile" : just build classlib, drlvm and tools
"unit_profile" : runs the drlvm tests and classlib tests, can be used
after "build-profile"
"interative_profile" : can run after either the above, does iterative
testing
"functional1_profile" : ...
etc

?

I don't know CC well enough - can I have these pre-set things, and
tell CC to do "build-profile" only, or do build-profile, and then if
that passes, do the unit-profile?


Seems, that I also don't know CC as well as I want :(  As I  
understand it,
CC operates with projects (in CC terms). Each project may depend on  
other
project status. For example, in the current buildtest configuration  
drlvm
build depends on classlib build status, run of drlvm tests depends  
on drlvm
build status and run of classlib tests depends on classlib and  
drlvm builds
status. So this dependency defined on the project level. In the  
case of
different independent profiles seems them it will depends on  
classlib and
drlvm builds only.

One solution is to have a two step process - have modular "project- 
lets"  for which there's a deployment step to produce a CC project file.

So on my machine, I specify what I want, "generate", and then run...

geir



geir

>
> thanks, Vladimir
>
>
>
>> >
>> >>
>> >> >
>> >> > It requires some 'standard' interface for all integrated
>> scripts. I
>> >> > like
>> >> > classlib interface so how about:
>> >> >  - call of "ant setup;ant" will run all available scripts;
>> >> >  - call of "ant -Dmodule=hit setup;ant" will run current
>> version of
>> >> > CC –
>> >> > Harmony integration tests;
>> >> >  - call of "ant -Dmodule=eut setup;ant" will run Eclipse
>> unit test
>> >> > etc.
>> >> >
>> >>
>> >> > Note, in this case each module should implement proper  
'setup'
>> >> > target and
>> >> > has configuration for CC. The root-script will iterate  
over all
>> >> > modules to
>> >> > call their 'setup' and this setup should include whole test
>> setup
>> >> > (downloading software, adding modules cc-configuration to
>> working
>> >> > configuration etc).
>> >> >
>> >> > Is it OK?
>> >>
>> >> I dunno - this sounds like disjoint and separate CC runs,
>> rather than
>> >> a CC run with multiple projects.
>> >>
>> >> For example, I'd like to have a set of "modules", which  
would be
>> >> incomplete cc config files, that somehow get glommed into a
>> bigger cc
>> >> file - maybe the config.xml would have some kind of include
>> >> mechanism.
>> >>
>> >> Suppose that we have :
>> >>
>> >> trunk/cc/
>> >>     config.xml
>> >>     modules/
>> >>        default_module.xml
>> >>        hut_module.xml
>> >>        eut_module.xml
>> >>        iterative_module.xml
>> >>        dacapo_module.xml
>> >>        specjbb_module.xml
>> >>        short_module.xml
>> >>        medium_module.xml
>> >>        long_modules.xml
>> >>
>> >> so then I could do :
>> >>
>> >>   $ ant
>> >>
>> >> to get what we have now - runs the default module - or
>> >>
>> >>   $ ant -Dmodules=hut,eut,dacapo
>> >>
>> >> to run those...
>> >>
>> >> Something like "medium_module.xml" would look like  (the
>> following is
>> >> 'psuedo-code') :
>> >>
>> >>     <includeconfig name="default_module.xml"/>
>> >>     <includeconfig name="dacapo_module.xml"/>
>> >>
>> >>
>> >> so that you can nest this as you want.
>> >>
>> >> >
>> >> > If nobody objects I'll start restructuring of buildtest
>> module and
>> >> > will try
>> >> > to integrate one from extensions.
>> >>
>> >> Please describe how you want to do it.
>> >
>> >
>> > I think about following option: in the root file we have  
predefined
>> > string.
>> > Something like 'modules=hut'. In the 'long' mode CC will iterate
>> > over all
>> > entries. The medium cycle depend on users wishes and defined
>> > through the
>> > command line like: 'ant -Dmodules=hut,iterative'. Each module  
has
>> > predefined
>> > target (for example, 'setup'). In this target script should
>> > download all
>> > software, apply all patches, add module configuration to the
>> > current CC
>> > configuration (just copy content of the module configuration  
file
>> > to the end
>> > of current configuration) etc. After 'ant <...> setup' we  
will have
>> > ready-to-run system with user-defined configuration.
>> >
>> >
>> >> >
>> >> >
>> >> > Thanks, Vladimir
>> >> >
>> >> >
>> >> >
>> >> > PS I think the resulting structure should be easy to extend
>> and may
>> >> > looks
>> >> > like this:
>> >> >
>> >> > buildtest
>> >> >
>> >> > |--config  (default CC configuration to build classlib and
>> DRLVM)
>> >> >
>> >> > |--hit (CC configuration to run Harmony classlib&DRLVM tests)
>> >> >
>> >> > |--eclipse
>> >> >
>> >> >     |-- eut (setup and CC configuration to run eclipse non-
>> >> interactive
>> >> > tests)
>> >> >
>> >> >     |-- eclipse3.1.1
>> >> >
>> >> >         |-- some scenario
>> >> >
>> >> > |-- build.xml (common setup + call of module's 'setup')
>> >>
>> >>
>> >> Interesting.  I think one issue is that it seems like heavy
>> lifting
>> >> to add a new module - each module becomes a "peer".   What  
do you
>> >> think of the other approach above?
>> >>
>> >> Either way, we don't want hit, eclipse, etc as peers.  If
>> anything,
>> >> they should go in a modules/ directory...
>> >
>> >
>> >
>> > For each module we have at least 2 files: cc-config and build  
file.
>> > But in
>> > some cases we will have some additional files (patches etc). For
>> > example, script for testing of JEdit application (issue 3012)  
has
>> > about 15
>> > files. For me is better to store all staff in one place  
instead of
>> > having
>> > parallel structure.
>> >
>> > thanks, Vladimir
>> >
>> > geir
>> >>
>> >>
>>
>>