I concur with Kay, particularly with point d) and its consequences. Sometimes it is obvious which result is better, often it is not. For example, one of the hkl users was testing new options and found that all the statistics (including refinement r-free) were worse, but the experimental and refinement maps were much better.
Myself I am testing crystallographic programs written by others. For easy cases they typically produce equivalent results, and for borderline cases, they tend to be sensitive to the input parameters, and sometimes one program works better, and on other data, another works better. This even applies to programs written by the same person (e.g. DM and Parrot), particularly when adjusting input parameters. The problems in real life are so diverse, that it is not clear what would be a representative set to test programs and make general conclusions. Zbyszek Otwinowski > Am 20:59, schrieb Van Den Berg, Bert: >> I have heard this before. I’m wondering though, does anybody know of a >> systematic study where different data processing programs are compared >> with real-life, non-lysozyme data? >> >> Bert > > Bert, > > some time ago I tried to start something to this effect - take a look at > the "Quality Control" article in XDSwiki. > (<http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Quality_Control>). > But it hasn't worked out, i.e. nobody participated (so far). > Possible reasons include: > a) it is considered politically incorrect (many years ago I wrote about > a comparison that I did ... the reactions from a few people were rather > harsh) > b) for reasons un-intelligible to me, people do not like to make their > raw data public (even if I ask directly) > c) it does take time to do and document > d) it's difficult to agree on the right methodology > e) it's a question that seems to interest only specialists > f) there's probably not a single answer > g) the programs are being constantly improved > > Concerning the last point, a wiki seems to be a good place to collect > the results (a table can be used to follow progress in a program, but > also to see the differences between programs). But that brings me to my > last point - a wiki article does not count as a paper. > > best, > > Kay > Zbyszek Otwinowski UT Southwestern Medical Center at Dallas 5323 Harry Hines Blvd. Dallas, TX 75390-8816 Tel. 214-645-6385 Fax. 214-645-6353
