> -----Original Message----- > From: Dale Tronrud [mailto:det...@uoxray.uoregon.edu] > Sent: 24 September 2009 17:21 > To: Ian Tickle > Cc: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] Rfree in similar data set
> While I agree with Ian on the theoretical level, in practice > people use free R's to make decisions before the ultimate model > is finished, and our refinement programs are still limited in > their abilities to find even a local minimum. I wasn't saying that Rfree is only useful for the ultimate finished model. My argument also applies to all intermediate models; the criterion is that the refinement has converged against the current working set, even if it is only an incomplete model, or if it is only to a local optimum. So it's perfectly possible to use Rfree for overfitting & other tests on intermediate models. The point is that it doesn't matter how you arrived at that optimum (whether local or global), Rfree is a function only of the parameters at that point, not of any previous history. If you arrived at that same local or global optimum via a path which didn't involve switching datasets midway, you must get the same answer for Rfree, so I just don't see how it can be biased one way and not biased the other. Note that this is meant as a 'thought experiment', I'm not saying necessarily that it's possible to perform this experiment in practice! > On the automated level the test set is used, sometimes, to > determine bulk solvent parameter, and more importantly to calibrate > the likelihood calculations in refinement. If the test set is > not "free" the likelihood calculation will overestimate the reliability > of the model and I'm not confident that error will not become > a self-fulfilling prophecy. It is not useful to divine meaning > from the free R until convergence is achieved, but the test > set is used from the first cycle. That is indeed a fair point, but I would maintain that the test set becomes 'free', i.e. free of the memory of all previous models, the first time you reach convergence, so the question of the effect on sigmaA calculations, which use the test set, is only relevant to the first refinement after switching test sets, thereafter it should be irrelevant. Converging to a local or global optimum wipes out all memory of previous models because the parameter values at that optimum are independent of any previous history, and so Rfree must be the same for that optimum no matter what path you took to get there. > Perhaps I'm in one of my more persnickety moods, but every > paper I've read about optimization algorithms say that the method > requires a number of iteration many times the number of parameters > in the model. The methods used in refinement programs are pretty > amazing in their ability to drop the residuals with a small number > of cycles, but we are violating the mathematical warranty on > each and every one of them. A refinement program will produce > a model that is close to optimal, but cannot be expected to be > optimal. Since we haven't seen an optimal model yet it's hard > to say how far we are off. I thought that for a quadratic approximation CG requires a number of iterations that is not more than the number of parameters (not that we ever use even that many iterations!)? Anyway that's a problem in theory, but it's possible to refine until nothing more 'interesting' happens, i.e. further changes appear to be purely random and at the level of rounding errors. Plotting the maximum shift of the atom positions or B factors from one iteration to the next is a very sensitive way of detecting whether convergence has been achieved; looking at changes in R factors or in RMSDs of bonds etc is a bad way, since R factors are not sensitive to small changes and atoms can move in concert without affecting bond lengths etc. (or it may just be the waters that are moving!). As a final point I would note that cell parameters frequently vary by several % between crystals even from the same batch due to unavoidable variations in rates of freezing etc, so what you think are independent test set reflections may in reality overlap significantly in reciprocal space with working set reflections from another dataset anyway! -- Ian Disclaimer This communication is confidential and may contain privileged information intended solely for the named addressee(s). It may not be used or disclosed except for the purpose for which it has been sent. If you are not the intended recipient you must not review, use, disclose, copy, distribute or take any action in reliance upon it. If you have received this communication in error, please notify Astex Therapeutics Ltd by emailing i.tic...@astex-therapeutics.com and destroy all copies of the message and any attached documents. Astex Therapeutics Ltd monitors, controls and protects all its messaging traffic in compliance with its corporate email policy. The Company accepts no liability or responsibility for any onward transmission or use of emails and attachments having left the Astex Therapeutics domain. Unless expressly stated, opinions in this message are those of the individual sender and not of Astex Therapeutics Ltd. The recipient should check this email and any attachments for the presence of computer viruses. Astex Therapeutics Ltd accepts no liability for damage caused by any virus transmitted by this email. E-mail is susceptible to data corruption, interception, unauthorized amendment, and tampering, Astex Therapeutics Ltd only send and receive e-mails on the basis that the Company is not liable for any such alteration or any consequences thereof. Astex Therapeutics Ltd., Registered in England at 436 Cambridge Science Park, Cambridge CB4 0QA under number 3751674
