inline
On 8/13/07, P. van der Velde <[EMAIL PROTECTED]> wrote:
>
>
> Leonid Lastovkin wrote:
> >
> > Patrick,
> >
> > I think your situation is very specific, and cannot be implemented
> > with just 20 lines of code. But it should not be hard to do by writing
> > some of your own classes.
> >
> >
> > The following stuff:
> >
> > [Source(Reader = typeof(MatrixEquationReader), File=" SomeSillyFile.mtx
> ")]
> > [Compare(typeof(MatrixEquationComparer))]
> > [Target(Writer=typeof(MatrixEquationWriter),File="
> SomeSillyResultFile.xml")]
> >
> > Is a syntactic sugar for a few dozen lines of C# code. I think you are
> > trying to do something specific, and so you have to decide the format
> > for " SomeSillyFile.mtx" and "SomeSillyResultFile.xml". By XML file
> > you probably do not mean the XML Test report. I do not think you want
> > that anyway. I think you will want to write a Reader class and a
> > Writer class yourself. My advice would be: once you know what you
> > need, pick file formats that are as simple as possible. You can end up
> > spending a large chunk of your time writing a file parser. Plain ASCII
> > comma-separated values (CSV) is a good format for storing tabular
> > data, and there are some libraries for it.
> I probably didn't explain myself properly but the main reason for the
> attributes is that I'm not only going to be testing matrix (solvers)
> with these tests. They will be used for all the verification tests I
> need, ranging from matrix solvers (probably the first application) to my
> fluid flow solver etc. etc. So there will be many different types of
> solvers tested and for each solvers there will be many different tests.
> What I'm trying to do is reduce overhead inside the tests, not because
> of the typing but because I want to prevent unnecessary copy / paste and
> unnecessary coupling between the data and the test. So I figured that a
> test basically reads some test data, runs the solver on that test data,
> then compares the output with the baseline and then writes the results
> back to a file. That's how the attributes came about.
Sounds like writing tests is a big project in itself. I have not written
tests that would require that much code reuse.
Hence, I am not the best person to ask about the advanced
features/customization of MbUnit.
Other people on this mailing list can probably give you a much better
answer.
I did, however, have to implement a similar test.
> I do not know if the algorithm that you are using for solving changes
> > over time, but you want to make sure that it is consistent. If it
> > does, then you probably want to store the results in time-stamped
> > files, and so the logic of the scanner class would become more complex.
> Good idea. I'll add that to the list of things to implement :-)
Something to keep in mind: you want to make sure that these timestamped
files
do not fill up your drive. You can make up a rule that you keep at least X
days
worth, or at least Y sets of most recent files, whichever gives you the
largest set of files,
but not to exceed Z megabytes. If you have to start deleting files, then
start
deleting oldest first.
You can pick X, Y, Z and implement this rule in code.
>
> > Another thing I am not sure about is how do you compare the results. I
> > am guessing that the results are different each time because it is not
> > possible to come up with a closed-form solution, and the algorithm
> > uses some sort of numerical approximation. You'd have to write the
> > comparer class yourself and think about how it should work. It would
> > have to make sure that one vector is similar to another. Well' it
> > sounds to me that you want to create a baseline (several results)
> > using your algorithm, when you know that your algorithm works
> > correctly, and save those somewhere. Then, during testing you would
> > solve the same equation several times, and then make sure that the N
> > results in the baseline and the N results that you just got look
> > reasonably close to each other. If the solver utilizes random number
> > generators, then you may be able to find a stored result A and a new
> > result B where A and B are not terribly similar (because A may be in
> > the left tail of a distribution, and B may be in the right tail). But
> > , when extracting the statistics about N stored results and N new
> > results, the numbers should be consistent. You have to know how the
> > algorithm works before you can decide what deviation is acceptable.
> > Or, you have to know what sort of deviation between consecutive
> > results the client is willing to tolerate. ... You probably know more
> > than I do about how to compare the results.
> This is pretty much how I was planning to implement it. Not entirely
> sure about the exact implementation but I'll work that out when I need
> to. I'd probably track the results between different tests so that I can
> see if an implementation is getting better (more accurate) or worse.
> Also it be nice to be able to link changes to bug fixes etc.
>
> Also it sounds like you have a lot more knowledge about numerics than
> you claim credit for, and you probably know more than I do ;-) Would you
> mind telling me what your work is in?
I have a bachelors in math. I just happened to implement a test case that
was similar to what you want to do. It was a numerical
simulation written by someone else. Random numbers were utilized.
If the seed was kept constant, then the results of a simulation had
to be the same every time you run it. If not, then they should not be the
same but similar.
Something else you should keep in mind is that the precision of a double is
limited
If you are working with numbers between 0 and 10^9, then anything after the
12th digit
after decimal point (you really do have to look this up as I do not know
this precisely)
can be useless. This really depends on the range of the values that you work
with.
Unless mathematics involved is trivial,
If you want to check that double a is exactly equal to double b,
then you will have to settle for something less perfect:
One way is to use an additive threshold:
static bool AreEqual(double a, double b, double addThresh) {
return Math.Abs(a - b) < addThresh;
}
Or a multiplicative threshold, where a == b iff |a - b| < (multThresh) *
|b|
>
> > However, I do not think you really need a decorator like this one:
> > [Compare(typeof(MatrixEquationComparer))]
> >
> > It looks like the Comparer class won't be simple, so what's a big deal
> > if you add a few more lines of code to your test suite?
> > Just initialize the comparer in the constructor or init, and use it
> > inside of a test. The decorator is not really shortening your test.
> The reason for the comparer is that I want to be able to write
> verification tests for many different algorithms, but I only want to
> write the comparer once and if possible I want to keep the tests really
> simple. I've noticed in my current tests that much of a test is in both
> setting up the test conditions and then verifying them, so I'm trying to
> find a way to provide a verification API (through the comparer etc.) so
> that the tests are shorter, simpler and more clear on what they actually
> test.
>
> Anyway, more clarification about my (possibly silly) ideas. Any more
> comments? I'd be grateful for them
There is more than one way to implement this. No one way is perfect.
I do not know what's the best way to deal with values that can form a
distribution.
I ended up checking rather obvious conditions. I also made the tests fail
if the difference between two values was too far from the average.
It is better to have a false negative, then a false positive.
The point is not so much to have all green on the test report as to stop
bugs from creeping in.
To do that, I think it is better to write test defensively.
When the failures occur, you take a look at the report and find out why.
If you are failing the test, it does not hurt to be verbose. For example:
Debug.Writeline(String.Concat("The value at row ", row.ToString(), ", column
", column.ToString(),
" was not the same between iteration ", i.ToString(), " and iteration ",
j.ToString()));
Debug.Writeline(String.Concat("The values were ", a.ToString(), " and ",
b.ToString()));
Debug.Writeline(String.Concat("Other useful information is: ", c.ToString
()));
Assert.Fail();
This is specially true for the kinds of tests failures that you cannot
reproduce exactly,
like waiting for values that are very far from the average to appear.
You'd have to re-run the test many times before you hit that condition.
Finally, it is better to have some simple tests going before implementing
elegant but time-consuming solutions.
It does not hurt to have testing logic overlapping (unless there is so much
redundancy that you can hardly manage the code)
For example, you want to test that "A is always less than 100, no matter
what the values of B are, and B is always greater than 50, no matter what
the values of A are".
You can start simple with test1 that generates a bunch of A's and makes sure
that A is always less than 100
and test2 that generates a bunch of B's and makes sure that B is always
greater than 50.
When both of these test start passing, you will become more confident that
the original condition is satisfied.
When/if you have extra time, then you can implement test3, which finally
checks for what you want.
Theoretically, test1 and test2 become redundant when you do that, because
test3 is a superset of test1 and test2.
But you still may want to keep test1 and test2 around. There is not too much
harm. and, if test3 starts failing,
then you may want to check whether test1 or test2 failed as well. They will
probably be easier to debug and understand.
This particular example is not very useful, but you know what I mean.
Good luck.
Regards
>
> Petrik
>
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"MbUnit.User" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/MbUnitUser?hl=en
-~----------~----~----~----~------~----~------~--~---
