2. If a contract verifier consists of 20 individual test cases, I cannot run a single test case, in order to debug just the failing code. I have to run all the test cases. This is all fine, if the test cases are light, but what if they are not? Why a contract verifier is treated differently then a dynamic test fixture? I really want to be able to refer to individual contract verifier test cases. To summarize, it is my personal opinion that contract verifier must be expandable in Icarus, so that individual test cases could be run and debugged (of course, that means these individual test cases should be runnable from Echo and Gallio task). I think we can do fine with only the native Gallio tools supporting it. In fact, I stopped using TD.NET and work exclusively with Icarus + VS, because Icarus lets me debug test selection, regardless whether they are statically compiled or dynamic. 3. http://code.google.com/p/mb-unit/issues/detail?id=521 Nested loops is not a very scalable solution. In my scenario dynamic tests wrap existing statically compiled ones, binding the first argument to some known value. If the prototype test is combinatorial, the respective dynamic test should be combinatorial as well. How can I do it with nested loops? Don't think I can.
Jeff, I agree with completely on striving to simplicity. However, as Einstein once said "make things as simple as possible, but not simpler". My scenarios seem complex, but I have complex testing needs: * I have to unit test (actually, they are integration tests, but never mind) the base entity infrastructure. * Actual business entities should be tested to make sure they do not break the base infrastructure. These tests come in addition to the specific business tests for the respective entity, which do not concern me. * The business entity developer cannot be expected to copy/paste the base infrastructure tests and manually eliminate those, which test infrastructure features not supported by the respective business entity. This is because, the base infra unit tests are added all the time and the developer will be constantly out of sync, let alone other problems always associated with copy/paste. * A business entity may not support some features, provided by the infrastructure. So, naturally, not all the base infra unit tests are relevant for it, but most are indeed. These testing needs define my solution, I did not come with it out of fancy. It was an evolution, from simple mstest to mstest with in- memory data provider and custom combinatorial test logic to simulate combinatorial tests and finally to MbUnit, because I had no choice. Making mstest behave like MbUnit is just too costly. BTW, [Disable] attribute may be a good idea in general, but it will do no good to me, because I place all the logic of creating unit tests to make sure business entities do not break the infrastructure in a single generic base class, which serves more than one business entity type. So it must suppress unwanted tests dynamically based on the features supported by the entity under test and the features expected by the particular unit test. The attribute is of no help here. On 04/09/2009, at 10:32, Jeff Brown wrote: > > 1. Debugging contract verifiers is the same as anything else. Stick a > breakpoint within the code of the contract verifier that you want to > hit. That said, they are intended to be black boxes for the most part > in the same way that assertions are. Users of contract verifiers > aren't really expected to set breakpoints in them (but they can). > > One implication is that you can't set a breakpoint on the field that > integrates the contract verifier into the fixture. You have to set a > breakpoint within the code of the verifier itself. There isn't > anything that we can do about this. > > 2. This is not supported. Think of contract verifiers as super-fancy > asserts. I know it is inconvenient but given the dynamic nature of > the contract verifier it's the best we can do. In general test > runners are not able to obtain information about the individual tests > within the contract verifier until they run. Consequently there's no > way to pick them or filter them. > > This is an area that may change later on. Gallio has a little bit of > code here and there that could ultimately be used to enable a test > runner to inspect and manipulate dynamic test cases. Truth be told > though, I don't expect much to happen here mainly for usability > reasons. We could perhaps support some of this functionality in > Icarus, Echo, Sail and other native Gallio runners but TDNet, > ReSharper and VSTT are unlikely to be able to support these features > without some significant UI work. > > 3. Yes that's true. Please feel free to open up a feature request for > MbUnit extensions to support data-driven custom test suites. Sounds > like something Yann might enjoy designing. I'll be honest though, > you're probably much better off with some nested for-loops than any > additional API affordances we may provide for this case. > > 4. It is not possible to override a single contract verifier test > case. However contract verifiers support extensive customization via > properties. You can also subclass them if you like to add new > features (although the built-in ones aren't really designed with > subclassing in mind yet.) > > > General comment: > > I have noticed that several of the things you are trying to do with > MbUnit are very complex. For example, it sounds like you use abstract > test fixtures very heavily and you are using many other dynamic > features. While MbUnit v3 is one of the most powerful .Net test > frameworks currently available, I do recommend opting for simplicity > when possible. > > I have personally written and experienced a lot of pain from writing > very clever test fixtures with fancy setup/teardown methods, template > methods, and fancy extensions. In the beginning it's pretty cool to > see just how much test code can be condensed by factoring out common > bits in novel ways but later on it can become problematic. Often the > complex abstractions are almost but not quite perfect which results in > the need to add shims here and there during reason to disable some > tests and tweak other behaviors. > > (But just for you I'm planning to add a [Disable] attribute to > indicate that a class / method is not a test. Be careful with it > though.) > > I think that Brad and Jim are on the right track with xUnit.net. > Granted, there are some things that you cannot do as easily with > xUnit.net but I believe the built-in constraints do help to stay > focused on what matters: developing robust and cohesive test suites. > > That said, it is a lot of fun for me as a framework author to have > someone like you pushing the edges of the framework and finding > opportunities to make it better! > > Cheers, > Jeff. > > On Aug 27, 12:49 am, Mark Kharitonov <[email protected]> > wrote: >> Another question - is it possible to override a single contract test >> case? >> Thanks. >> >> On Aug 27, 9:42 am, Mark Kharitonov <[email protected]> >> wrote: >> >>> I have played a bit with it. >>> A few things I do not understand about contract verifiers: >>> 1. How to debug them? >>> 2. How to run just one contract test case? Icarus shows the contract >>> verifier tests as one atomic unit. >>> 3. If a contract verifier should include a combinatorial test, then >>> one has to implement it all by oneself, because dynamic test >>> generation does not provide this service, unlike the statically >>> compiled unit test cases, am I correct? >> >>> Thanks. >> >>> On Aug 24, 10:04 pm, "Jeff Brown" <[email protected]> wrote: >> >>>> That's pretty much it but you could instead create a TestSuite >>>> that contains >>>> multiple TestCases. >> >>>> There is a bunch of framework code for data-driven testing in >>>> Gallio.Framework.Data. Some of this may be useful to use >>>> however I expect >>>> you will find it easier to keep it simple and generate >>>> combinatorial tests >>>> using nested loops. >> >>>> Jeff. >> >>>> _____ >> >>>> From: [email protected] >>>> [mailto:[email protected]] On >>>> Behalf Of mark Kharitonov >>>> Sent: Monday, August 24, 2009 9:05 AM >>>> To: [email protected] >>>> Subject: MbUnit Re: How to extract the test Metadata from within >>>> the SetUp >>>> method? >> >>>> On the subject of manually writing combinatorial tests. >>>> I have a question here. >>>> A combinatorial test appears just once in Gallio.Icarus. It >>>> serves as a well >>>> defined report unit, under which the actual test steps appear. >>>> How can I achieve the same effect with manually created >>>> combinatorial tests? >>>> Looks like I need to create a test and then the test steps as >>>> children of >>>> that test, is that right? >> >>>> BTW, Is it possible to reuse Gallio code when manually creating >>>> combinatorial test steps? >> >>>> Thanks, >> >>>> On Sat, Aug 22, 2009 at 10:37 PM, Jeff Brown >>>> <[email protected]> wrote: >> >>>> At runtime you can obtain information about the current test using >>>> TestContext.Current.Test. >> >>>> What you cannot do at the moment is to easily obtain information >>>> about other >>>> tests. Well, in principle the whole test model is fully >>>> accessible but in >>>> practice you'll find it to be inconvenient to work with. >> >>>> I have been thinking about what you said with respect to >>>> regenerating >>>> dynamic test cases from static ones for testing entities of >>>> different types. >>>> I think I would recommend that you instead implement a new contract >>>> verifier. >> >>>> Take a look at IContract, VerifyContractAttribute and some of >>>> the existing >>>> contract verifiers like EqualityContract. >> >>>> You could pretty easily implement an EntityContract of your >>>> own. Basically >>>> a contract generates test cases in the same manner as a >>>> [StaticTestFactory] >>>> but they are packaged in a more convenient form for reuse. >> >>>> However, contracts then have the same disadvantage as the static >>>> & dynamic >>>> test factories in that they do not (yet) provide convenient >>>> abstractions for >>>> representing data-driven test cases. I expect that this can be >>>> remedied >>>> with a little work but it's out of scope for v3.1. As a result >>>> you will >>>> have to write your own combinatorial test generators and such >>>> but it's >>>> pretty trivial code. >> >>>> Anyways here's what an entity contract might look like to the user: >> >>>> [VerifyContract] >>>> public readonly EntityContract MyEntityContract = new >>>> EntityContract<MyEntity>(... options...); >> >>>> Jeff. >> >>>> -----Original Message----- >>>> From: [email protected] >>>> [mailto:[email protected]] On >> >>>> Behalf Of Mark Kharitonov >>>> Sent: Saturday, August 22, 2009 11:52 AM >>>> To: MbUnit.User >>>> Subject: MbUnit How to extract the test Metadata from within the >>>> SetUp >>>> method? >> >>>> I mean, besides using the reflection on the test method by its >>>> name. I >>>> expected to find something like GetMetadata method on the Test >>>> class, but >>>> there is none. >> >>>> -- >>>> Be well and prosper. >>>> ============================== >>>> "There are two kinds of people.Those whose guns are loaded and >>>> those who >>>> dig." >>>> ("The good, the bad and the ugly") >>>> So let us drink for our guns always be loaded. >> >> > > ======================================================================== == There are two kinds of people. Those whose guns are loaded and those who dig. (The good, the bad and the ugly). So let us raise our cups for our guns always be loaded. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "MbUnit.User" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/mbunituser?hl=en -~----------~----~----~----~------~----~------~--~---
