Re: [Boston.pm] Test code generation (was C++ books)
On Mon, Apr 15, 2013 at 11:35:22AM -0700, Ben Tilly wrote: On Mon, Apr 15, 2013 at 11:09 AM, Greg London em...@greglondon.com wrote: For unit testing I've been emitting TAP protocol and testing it with prove, but are there better approaches? I get a test file with a lot of code that looks like this: printf( %s %d: Some useful description and maybe a number %d\n, (expected_value == test_value) ? ok : not ok, ++tests, some_useful_debugging_info ); This method is usually tolerable for a while but gets old as soon as I realize I'm doing the computer's job much slower and less accurately. Instead, I generate the test file from a simpler data file that only contains the relevant test data. I've successfully used three different approaches for this: 1. Extract the relevant data into a text file and use it as input to a program that produces the test file. This works well when the input has a lot of variety or could change in unexpected ways, but then you also have a code generator to maintain. 2. Extract the relevant data into XML and use gsl (https://github.com/imatix/gsl) to generate the test file. Works well when the test data varies in ways that can be relatively easily expressed by xml. While this approach works well, I don't particularly like editing xml so I actually write my data in a hierarchical text format and use, shameless plug, txt2xml (https://github.com/gyepisam/txt2xml) to convert it to xml. 3. I have some similar things with autogen (apt-get install autogen) and it works pretty well too. The various conversions from txt to xml to c and h, do produce long pipelines, but I use make so I just define implicit rules and let make resolve the dependency chain for me. Then I can kick it off with. make test It works quite well and the pieces are modular enough that the next guy is likely to understand it. However, understanding is not required for effective use, which is usually the primary goal. -Gyepi ___ Boston-pm mailing list Boston-pm@mail.pm.org http://mail.pm.org/mailman/listinfo/boston-pm
Re: [Boston.pm] Test code generation (was C++ books)
Instead, I generate the test file from a simpler data file that only contains the relevant test data. When I was verifying a hardware-based image converter, I used that approach. You gave it an image file, a small file that contained some register settings that indicated what we wanted the hardware to do. This went into a golden model and the dut, and a simple diff at the end gave a pass/fail. But the environment I'm working on now has many hundreds of registers. And it's not simply a matter of selecting the mode for the dut. There are a lot of sequential operations that need to be done that isn't simply a matter of data-based testing. If a fault-bit is set, I need to change the condition so that the cause of the fault goes away. Then I check the bit to make sure it's still set because its sticky. Then I clear the fault, and read it to make sure that it finally got clear. I was working on one project where the project manager suggested data-based testing. We looked at the DUT, and I said we need an if statement. I looked some more, and it was clear we needed a looping statement. And the looping statement needed variables. And rather than re-inventing a computer language, we dropped the data approach, and wrote the tests with an existing language. Greg ___ Boston-pm mailing list Boston-pm@mail.pm.org http://mail.pm.org/mailman/listinfo/boston-pm
Re: [Boston.pm] Test code generation (was C++ books)
On Tue, Apr 16, 2013 at 6:51 AM, Gyepi SAM gy...@praxis-sw.com wrote: On Mon, Apr 15, 2013 at 11:35:22AM -0700, Ben Tilly wrote: On Mon, Apr 15, 2013 at 11:09 AM, Greg London em...@greglondon.com wrote: For unit testing I've been emitting TAP protocol and testing it with prove, but are there better approaches? I get a test file with a lot of code that looks like this: printf( %s %d: Some useful description and maybe a number %d\n, (expected_value == test_value) ? ok : not ok, ++tests, some_useful_debugging_info ); This method is usually tolerable for a while but gets old as soon as I realize I'm doing the computer's job much slower and less accurately. Instead, I generate the test file from a simpler data file that only contains the relevant test data. I've successfully used three different approaches for this: 1. Extract the relevant data into a text file and use it as input to a program that produces the test file. This works well when the input has a lot of variety or could change in unexpected ways, but then you also have a code generator to maintain. 2. Extract the relevant data into XML and use gsl (https://github.com/imatix/gsl) to generate the test file. Works well when the test data varies in ways that can be relatively easily expressed by xml. While this approach works well, I don't particularly like editing xml so I actually write my data in a hierarchical text format and use, shameless plug, txt2xml (https://github.com/gyepisam/txt2xml) to convert it to xml. 3. I have some similar things with autogen (apt-get install autogen) and it works pretty well too. The various conversions from txt to xml to c and h, do produce long pipelines, but I use make so I just define implicit rules and let make resolve the dependency chain for me. Then I can kick it off with. make test It works quite well and the pieces are modular enough that the next guy is likely to understand it. However, understanding is not required for effective use, which is usually the primary goal. I think you're visualizing something different than my actual problem. There are indeed a lot of lines in my test file resembling the pattern that I showed. But I do not primarily have values in/out. Instead I set up method calls, make them, then go poking around the innards of data structures and verify that they have what I expect them to have. This requires intimate knowledge of the class that I'm testing, which is not so automatable. But when I have a different type of problem, a data-driven approach is very good. So is throwing random data at stuff and seeing what breaks. That just doesn't happen to be the type of problem that I have right now. ___ Boston-pm mailing list Boston-pm@mail.pm.org http://mail.pm.org/mailman/listinfo/boston-pm
Re: [Boston.pm] Test code generation
On Tue, Apr 16, 2013 at 11:35:55AM -0700, Ben Tilly wrote: On Tue, Apr 16, 2013 at 6:51 AM, Gyepi SAM gy...@praxis-sw.com wrote: On Mon, Apr 15, 2013 at 11:35:22AM -0700, Ben Tilly wrote: On Mon, Apr 15, 2013 at 11:09 AM, Greg London em...@greglondon.com wrote: For unit testing I've been emitting TAP protocol and testing it with prove, but are there better approaches? I get a test file with a lot of code that looks like this: printf( %s %d: Some useful description and maybe a number %d\n, (expected_value == test_value) ? ok : not ok, ++tests, some_useful_debugging_info ); Instead, I generate the test file from a simpler data file that only contains the relevant test data. I think you're visualizing something different than my actual problem. There are indeed a lot of lines in my test file resembling the pattern that I showed. But I do not primarily have values in/out. Instead I set up method calls, make them, then go poking around the innards of data structures and verify that they have what I expect them to have. This requires intimate knowledge of the class that I'm testing, which is not so automatable. Yeah, I'd have been quite surprised if my response solved your problem directly. In general, I use code generation as a mechanism to give myself a little more room between the various stages of program construction (edit, compile, run) so I can insert a translation step which allows me to introduce new modes of expression that more closely model the particular sub-problem I'm trying to solve. Also, I'd rather write a program that writes programs. Particularly when the resulting program is tedious. I was not suggesting that data files should only contain in/out values. Sometimes that is the case, but sometimes you need something more complicated. In most cases, my data files actually specify some kind of a little language close to the problem domain and my translator converts it back into the host language. Actually, that reminds me you could also follow the old embedded SQL path and use a tool like cog to simplify some of your tests. Obviously, if the validity of the test depends on eyeballing the results there's not much to say in the way of automation. Until you get tired of eyeballing ;) -Gyepi ___ Boston-pm mailing list Boston-pm@mail.pm.org http://mail.pm.org/mailman/listinfo/boston-pm
