Re: [Lustre-discuss] Metadata storage in test script files
On Mon, May 7, 2012 at 7:33 PM, Nathan Rutman nrut...@gmail.com wrote: On May 4, 2012, at 7:46 AM, Chris Gearing wrote: Hi Roman, I think we may have rat-holed here and perhaps it's worth just re-stating what I'm trying to achieve here. We have a need to be able to test in a more directed and targeted manner, to be able to focus on a unit of code like lnet or an attribute of capability like performance. However since starting work on the Lustre test infrastructure it has become clear to me that knowledge about the capability, functionality and purpose of individual tests is very general and held in the heads of Lustre engineers. Because we are talking about targeting tests we require knowledge about the capability, functionality and purpose of the tests not the outcome of running the tests, or to put it another way what the tests can do not what they have done. One key fact about cataloguing the the capabilities of the tests is that for almost every imaginable case the capability of the test only changes if the test itself changes and so the rate of change of the data in the catalogue is the same and actually much less than the rate of change test code itself. The only exception to this could be that a test suddenly discovers a new bug which has to have a new ticket attached to it, although this should be a very very rare if we manage our development process properly. This requirement leads to the conclusion that we need to catalogue all of the tests within the current test-framework and a catalogue equates to a database, hence we need a database of the capability, functionality and purpose of the individual tests. With this requirement in mind it would be easy to create a database using something like mysql that could be used by applications like the Lustre test system, but using an approach like that would make the database very difficult to share and will be even harder to attach the knowledge to the Lustre tree which is were it belongs. So the question I want to solve is how to catalogue the capabilities of the individual tests in a database, store that data as part of the Lustre source and as a bonus make the data readable and even carefully editable by people as well as machines. Now to focus on the last point I do not think we should constrain ourselves to something that can be read by machine using just bash, we do have access to structure languages and should make use of that fact. I think we all agree 100% on the above... The solution to all of this seemed to be to store the catalogue about the tests as part of the tests themselves ... but not necessarily that conclusion. , this provides for human and machine accessibility, implicit version control and certainty the what ever happens to Lustre source the data goes with it. It is also the case that by keeping the catalogue with the subject the maintenance of the catalogue is more likely to occur than if the two are separate. I agree with all those. But there are some difficulties with this as well: 1. bash isn't a great language to encapsulate this metadata The thing to focus on I think is the data captured not the format. The parser for yaml encapsulated in the source or anywhere else is a small amount of effort compared to capturing the data in the first place. If we capture the data and it's machine readable then changing the format is easy. There are many advantages today to keeping the source and the metadata in the same place, one being that when reviewing new or updated tests the reviewers can and will be encouraged to by the locality to ensure the metadata matches the new or revised test. If the two are not together then they have very little chance of being kept in sync. 2. this further locks us in to current test implementation - there's not much possibility to start writing tests in another language if we're parsing through looking for bash-formatted metadata. Sure, multiple parsers could be written... I don't think it is a lock in at all, the data is machine readable and moving to a new format when and should we need it will be easy. Let's focus on capturing the data so we increase our knowledge, once we have the data we can manipulate it however we want. The data and the metadata together in my opinion increases the chance of capturing and updating the data given todays methods and tools. 3. difficulty changing md of groups of tests en-mass - eg. add slow keyword to a set of tests The data can read and written by machine and the libraries/application to do this would be written. Referring back to the description of the metadata we would not be making sweeping changes to test metadata because the metadata should only change when the test changes [exceptions will always apply but we should not optimize for exceptions]. Also I don't think 'slow' would not be part of the metadata because it is not an attribute
Re: [Lustre-discuss] Metadata storage in test script files
Hi Roman,
I think we may have rat-holed here and perhaps it's worth just
re-stating what I'm trying to achieve here.
We have a need to be able to test in a more directed and targeted
manner, to be able to focus on a unit of code like lnet or an attribute
of capability like performance. However since starting work on the
Lustre test infrastructure it has become clear to me that knowledge
about the capability, functionality and purpose of individual tests is
very general and held in the heads of Lustre engineers. Because we are
talking about targeting tests we require knowledge about the capability,
functionality and purpose of the tests not the outcome of running the
tests, or to put it another way what the tests can do not what they have
done.
One key fact about cataloguing the the capabilities of the tests is that
for almost every imaginable case the capability of the test only changes
if the test itself changes and so the rate of change of the data in the
catalogue is the same and actually much less than the rate of change
test code itself. The only exception to this could be that a test
suddenly discovers a new bug which has to have a new ticket attached to
it, although this should be a very very rare if we manage our
development process properly.
This requirement leads to the conclusion that we need to catalogue all
of the tests within the current test-framework and a catalogue equates
to a database, hence we need a database of the capability, functionality
and purpose of the individual tests. With this requirement in mind it
would be easy to create a database using something like mysql that could
be used by applications like the Lustre test system, but using an
approach like that would make the database very difficult to share and
will be even harder to attach the knowledge to the Lustre tree which is
were it belongs.
So the question I want to solve is how to catalogue the capabilities of
the individual tests in a database, store that data as part of the
Lustre source and as a bonus make the data readable and even carefully
editable by people as well as machines. Now to focus on the last point I
do not think we should constrain ourselves to something that can be read
by machine using just bash, we do have access to structure languages and
should make use of that fact.
The solution to all of this seemed to be to store the catalogue about
the tests as part of the tests themselves, this provides for human and
machine accessibility, implicit version control and certainty the what
ever happens to Lustre source the data goes with it. It is also the case
that by keeping the catalogue with the subject the maintenance of the
catalogue is more likely to occur than if the two are separate.
My original use of the term test metadata is intended as a more modern
term for catalogue or the [test] library.
So to refresh everybody's mind, I'd like to suggest that we place test
metadata in the source code itself using the following format, where the
here doc is inserted into the copy about the test function itself.
===
TEST_METADATA
Name:
before_upgrade_create_data
Summary:
Copies lustre source into a node specific directory and then creates
a tarball using that directory
Description:
This should be called prior to upgrading Lustre and creates a set of
data on the Lustre partition
which be accessed and checked after the upgrade has taken place.
Several methods are using
including tar'ing directories so the can later be untar'ed and
compared, along with create sha1's
of stored data.
Component:
- lnet
- recovery
Prerequisites:
- before_upgrade_clear_filesystem
TicketIDs:
- LU-123
- LU-432
Purposes:
- upgrade
TEST_METADATA
test_before_upgrade_create_data() {
...
}
run_test before_upgrade_create_data Copying lustre source into a
directory $IOP_DIR1, creating and then using source to create a tarball
===
Again thoughts and input very much appreciated
Chris
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Test restructuring
Hi Nathan, This is definitely something we have been thinking about but also something that's not got to the top of our to-do list yet. I think we can safely say that we share a desire to make them more reliable, repeatable, and easier to run and automate, and also a view that to do this is going to need restructuring. Perhaps we should use this mail to start a discussion on lustre-discuss and then agree a plan/approach that someone can execute. Chris should talk about and also something that -Original Message- From: Nathan Rutman Sent: Saturday, October 08, 2011 12:30 AM To: ch...@whamcloud.com Cc: Nathan Rutman ; Roman Grigoryev ; wc-discuss ; lustre-discuss Subject: Test restructuring Hi Chris - talking to Eric after EOFS he mentioned that you were not just interested in automating the existing Lustre tests but also in restructuring how they were designed / set up to make them more reliable, repeatable, and easier to run and automate. Or, at least that is my desire and I was hoping you had similar thoughts / plans. In particular, I'm interested in a few things: 1. Separating out the setup of the filesystem from the execution of the tests. 2. Reducing the ordering dependencies on the tests and clarifying their individual prerequisites and postconditions. 3. Reorganizing the tests into more logical groupings. 4. Cutting out redundant / ineffective tests. Are you / WhamCloud pursuing any of this at the moment? __ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. __ ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Landing and tracking tools improvements
We now have a whole kit of tools [Jira, Gerrit, Jenkins and Maloo] used for tracking, reviewing and testing of code that are being used for the development of Lustre. A lot of time has been spent integrating and connecting them appropriately but as with anything the key is to continuously look for ways to improve what we have and how it works. So my question is what do people think of the tools as they stand today and how can we improve them moving forwards. if people can respond to lustre-discuss then I'll correlate the outcome of any discussions and then create a Wiki page that can form some plan for improvement. Please be as descriptive as possible in your replies and take into account that I and others have no experience of Lustre past so if you liked something prior to the current tools you'll need to help me and them understand the details. Thanks Chris --- Chris Gearing Snr Engineer Whamcloud. Inc. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] lustre-discuss@lists.lustre.org
Hi Aurelien, That's useful thanks. Our plan is to create a community database that enables every valuable facet of a lustre test to be stored. This means all the results, all the logs, stack traces etc etc. Sometimes we will only store failing logs, but for acceptance and release testing we will store passing and failing(!) logs. The aim is that when someone is looking into a test failure, nothing they require will have been left behind. A further use of this approach is that we hope to be able to deliver landing collateral to reviewers as a standard part of our process. For example a patch could have a link to an example of the failing test that isolated the bug or missing feature, the code that fixes the failure or adds the feature and an example of the passing test showing the bug fix or feature addition working. This additional usage is not entirely near term, but is a direction that would add great value. As for yml vs. xml, the reality is that we use the yml as a way of transporting the results from the test machine, which could be a test-cluster or an individuals laptop running VMs, to the database. If xml made this data more widely usable and the requirements of junit do not require us to leave information out I can't really see a problem in switching to it. Chris On 20/12/2010 14:50, DEGREMONT Aurelien wrote: Hi Chris, Here is a rough example of junit report we produce. This was a quick and dirty implementation of Junit that could be improved. Some part of junit report content was limited due to lack of information acceptance small upcalls have. ?xml version=1.0 encoding=UTF-8 ? testsuite name=Lustre.acceptance-small testcase classname=Lustre.sanity name=Test #0: touch .../ ; rm .../ time=1 / testcase classname=Lustre.sanity name=Test #0b: chmod 0755 /mnt/lustre time=1 / testcase classname=Lustre.sanity name=Test #0c: check import proc time=1 / testcase classname=Lustre.sanity name=Test #1a: mkdir .../d1; mkdir .../d1/d2 time=1 / testcase classname=Lustre.sanity name=Test #1b: rmdir .../d1/d2; rmdir .../d1 time=1 / testcase classname=Lustre.sanity name=Test #2a: mkdir .../d2; touch .../d2/f time=1 / testcase classname=Lustre.sanity name=Test #2b: rm -r .../d2; checkstat .../d2/f time=1 / testcase classname=Lustre.sanity name=Test #3a: mkdir .../d3 time=1 / testcase classname=Lustre.sanity name=Test #3b: touch .../d3/f time=1 / testcase classname=Lustre.sanity name=Test #3c: rm -r .../d3 time=1 / testcase classname=Lustre.sanity name=Test #4a: mkdir .../d4 time=1 / testcase classname=Lustre.sanity name=Test #4b: mkdir .../d4/d2 time=1 / ... testcase classname=Lustre.sanity name=Test #180a: test obdecho on osc time=80 failure type=FAIL![CDATA[rc=1 test_180a failed with 1]]/failure /testcase ... testcase classname=Lustre.sanity-benchmark name=Test #dbench: test_dbench time=0 skipped/ /testcase testcase classname=Lustre.sanity-benchmark name=Test #dbench: dbench time=2 / ... /testsuite This is a imple junit report. This could be improved. Partly in improving acceptance-small.sh and test-framework.sh. I'm open to switch to any other standard format, supported by Hudson, if this can helps. What's your needs/plan? Aurélien Chris Gearing a écrit : Hi Aurélien, Do you have a specification for the junit test results you produce, or an example of one of your test results sets. We would be more than willing to pick up and go with something that can be used with a wider set of tools, with the obvious caveat that it provides everything needed to completely capture the test results for Lustre today and in the future. If you have some example results set's that you can forward please mail them to chris whamcloud.com Thanks Chris I see that PerfPublisher uses xml, although this seems to be the only specification. On 17/12/2010 20:11, Aurélien wrote: Robert Read a écrit : We don't plan to use Hudson to manage our testing results as I don't think it would scale very well for all the testing we might do for each build. We're currently building a more custom results server that's similar (in spirit at least) to the kinds of tools we had at Oracle. We'll make it available once it's in presentable form. Actually, our first step was to replace the acceptance-small.sh driver script with one that has a more sensible user interface for running the standard tests. Since the test-framework.sh on master has already been changed to produce test results in yaml format, the new script collects these with the logs, and is capable of submitting them to the test results server. Currently this is being run manually, though. Automating the test execution and connecting all the pieces will be next step. Ok. I will be very interested in seeing the final result. But I think it is a good idea to stick to standard format and tools as much as possible. This could be a pity
Re: [Lustre-discuss] lustre-discuss@lists.lustre.org
Hi Aurélien, Do you have a specification for the junit test results you produce, or an example of one of your test results sets. We would be more than willing to pick up and go with something that can be used with a wider set of tools, with the obvious caveat that it provides everything needed to completely capture the test results for Lustre today and in the future. If you have some example results set's that you can forward please mail them to chris whamcloud.com Thanks Chris I see that PerfPublisher uses xml, although this seems to be the only specification. On 17/12/2010 20:11, Aurélien wrote: Robert Read a écrit : We don't plan to use Hudson to manage our testing results as I don't think it would scale very well for all the testing we might do for each build. We're currently building a more custom results server that's similar (in spirit at least) to the kinds of tools we had at Oracle. We'll make it available once it's in presentable form. Actually, our first step was to replace the acceptance-small.sh driver script with one that has a more sensible user interface for running the standard tests. Since the test-framework.sh on master has already been changed to produce test results in yaml format, the new script collects these with the logs, and is capable of submitting them to the test results server. Currently this is being run manually, though. Automating the test execution and connecting all the pieces will be next step. Ok. I will be very interested in seeing the final result. But I think it is a good idea to stick to standard format and tools as much as possible. This could be a pity if all your new work will be only usable by your tool. Junit is quite standard. PerfPublisher has its own format due to junit limitations. There is other ones. It could be really good if you do not create a new one. And indeed, acc-sm is a bit limited and improve it could be really interesting. Aurélien ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
