Hi,
On 05/02/2012 01:25 PM, Chris wrote:
> On 02/05/2012 04:23, Roman Grigoryev wrote:
>> Hi Cris,
>>
>> On 05/01/2012 08:17 PM, Chris wrote:
>>> The metadata can be used in a multitude of ways, for example we can
>>> create dynamic test sets based on
>>> the changes made or target area of testing. What we are doing here is
>>> creating an understanding of the
>>> tests that we have so that we can improve our processes and testing
>>> capabilities in the future.
>> I think that when are are defining tool we should say about purpose.
>> F.e. good description and summary is not needed for creating dynamic
>> test sets. I think, it very important to say how will we use it.
>> Continue of this idea please read below.
> The purpose is to enable use to develop and store knowledge/information
> about the tests, the information should be in a conical form, objective
> and correct. If we do this then the whole community can make use of it
> as they see fit. I want to ensure that the initial set of stored
> variables describes the tests as completely as reasonably possible. The
> conical description of each test is not effected by the usage to which
> the data is put.
>
>>> The metadata does not go to the results. The metadata is a database in
>>> it's own right and should metadata
>>> about a test be required it would be accessed from the source (database)
>>> itself.
>> I think fields like title, summary, and, possible. description should be
>> present in results too. It can be very helpful for quickly understanding
>> test results.
> They can be presented as part of results but I would not store with the
> results, if for example Maloo presents the description it will fetch it
> from the correct version of the source, we should not be making copies
> of data.
ok, good.
>
> I cannot suppose if you should store this information with your results
> because I have no insight into your private testing practices.
I just want to have info not only in maloo or other big systems but in
default test harness. Developers can run results by hand, tester also
should have possibility to execute in specific environment. If we can
provides some helpful info - i think it is good. few kilobytes is not so
match as logs, but can help in some cases.
>>
>>>> On 04/30/2012 08:50 PM, Chris wrote:
>>> ... snip ...
>>>
>>>
>>> As I said we can mine this data any-time and anyway that we want, and
>>> the purpose of this
>>> discussion is the data not how we use it. But as an example something
>>> that dynamically built
>>> test sets would need to know prerequisites.
>>>
>>> The suffix of a,b,c could be used to generate prerequisite information
>>> but it is firstly inflexible, for example
>>> I bet 'b','c' and 'd' are often dependent on 'a' but not each other,
>>> secondly and more importantly we want a
>>> standard form for storing metadata because we want to introduce order
>>> and knowledge into the test
>>> scripts that we have today.
>> Why I asked about way of usage: if we want to use this information in
>> scripts and in other automated way we must strictly specify logic on
>> items and provides tool for check it.
>>
>> F.e. we will use it when built test execution queue. We have chain like
>> this: test C prerequisite B, test B prerequisite A. Test A doesn't have
>> prerequisite. In one good day test A became excluded. Is it possible to
>> execute test C?
>> But if we will not use it in scripting there is no big logical problem.
>>
>> (My opinion: I don't like this situation and think that test
>> dependencies should be used only in very specific and rare case.)
> I don't think people should introduce dependencies either, but they have
> and we have to deal with that fact. In your example if C is dependent on
> A and A is removed then C cannot be run.
Maybe I'm incorrect, but fight with dependencies looks like more
important then adding descriptions.
>>
>>>> I suggest add keywords(Components could be translated as keywords too)
>>>> and test type (stress, benchmark, load, functional, negative, etc) for
>>>> quick filtering. For example, SLOW could transform to keyword.
>>> This seems like a reasonable idea although we need a name that describes
>>> what it is,
>>> we will need to define that set of possible words as we need to with the
>>> Components elements.
>> I mean that 'keywords' should be separated from components but could be
>> logically included. I think, 'Components' is special type of keywords.
> I don't think of Components as a keyword, I think of it as a factual
> piece of data and if we want to add the test purpose then we should call
> it that. The use of keywords in data is generally a typeless catch-all.
> All of this metadata should be clear and well defined which does not in
> my opinion allow scope for a keywords element.
I agreed that Components aren't keywords.
>
> I would suggest that we add a variable called Purposes which is an array
> containing a set of predefined elements like stress, benchmark, load and
> functional etc.
>
> For example
>
> Purposes:
> - stress
> - load
What about SLOW(which should be named as be smoke or sanity) , negative
keywords? It is not about purposes but mostly about test type.
>
>>> It would be easier to store the data separately and we could use Maloo
>>> but it's very important
>>> that this data becomes part of the Lustre 'source' so that everybody can
>>> benefit from it. Adding
>>> tickets is not a problem as part of the resolution issue is to ensure
>>> that at least one test exercises
>>> the problem and proves it has been fixed, the fact that this assurance
>>> process requires active
>>> interaction by an engineer with the scripts is a positive.
>>>
>>> As for pass rate, execution time and gold configurations this
>>> information is just not 1 dimensional
>>> enough to store in the source.
>>>
>> I'm not accidentally in previous letter said about group of fields. All
>> meta data may be separated by rare and often changed fields. F.e.
>> Summary will change not so often. But test timeout in golden
>> configuration (I mean that this timeout will be set as default based on
>> 'gold' configuration and can be overloaded in specific configuration)
>> could be more variable(and possible more important for testing).
> What exactly is a gold configuration? Lustre has such breadth of
> possibilities that gold configurations would be a matrix of
> distro/architecture/distro version/interconnect/cpu
> speed/memory/storage/oss count/client count/... . To try and summarise
> this into some useful single value does not make any sense to me.
I incorrectly used phrase 'gold configurations', correctly says
'development configuration' or maybe 'default configuration'.
I absolutely agree that some test characteristic are relative to
configuration.
But, for many tests it is possible (and for many people is could be very
helpful) to have suggested f.e. timeout which indicates assumed upper
limit of execution time on often used configuration. In this case,
'default configuration' should be, I think, 4 nodes VM or 4 nodes real
cluster in one subnet.
For covering other configuration, we can use option 'timeout
multiplexor' in test framework and this option allows to cover 100%
configurations, I think.
Currently I use 300 sec per test in my scripts, it is overkill for the
most of tests and only for few tests I set longer time. (Also I think it
should be true for the most of configuration exclude f.e. configuration
with complex lnet routing or systems under high load)
>> Using separated files provides more flexibility and nobody stop us to
>> commit it to lustre repo and it became " Lustre 'source'". In separated
>> files we can use format which we want and all information will be
>> available without parsing shell script or without running it. More over,
>> in great future, it give us very simple migration from shell to other
>> language.
> This data is valuable and needs to be treated with the same respect and
> discipline as we treat the source, to imagine we can have a 'free for
> all' where people just update it at will does not work. The controls on
> what goes into the Lustre tree are there for very good reason and we are
> not going to circumvent those controls. We have to invest in this as we
> do with all the test infrastructure, it cannot be done on the cheap.
I don't really understand why we cannot cover by discipline separated
yaml sources too as shell code. More over, the most of yaml testing can
be done by automated tools. I don't say that this data should be 'free
for all'. But, I think, it is good idea(mostly for test developers):
providing way for user to override main metadata by his own metadata
with simple switch.
>
> Parsing the scripts for the data is easy because computers are really
> good at it. I would expect someone will write a library to access and
> modify the data as required, I'd also expect them to publish that library.
It looks not so simple to have one more library for cutting from shell
yaml data which have to parse by yaml reader. Question is not in cpu
time, but in full complexity of bash, external utilities and libraries
and test for them.
>
> If test's were re-written then this data will probably change, and the
> cost of migrating unchanged data will be insignificant compared to the
> cost of re-writing the test itself.
I'm not sure what do you mean there, could you please explain.
Thanks,
Roman
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
