Sticking to the same test set is a great and practical idea! It
lowers the chance to get biased from "self-validation". However,
logically,
0) based on Bragg's law and HKL<->XYZ theory, no reflections are truly
free from the others from the same crystal. But when the reflection
number is larger than the number of parameters minus other restriction
conditions, you have more degree of freedom, statistically.
Uncertainties contributed from many aspects increase the freedom.
Even though, free test set is not purely free.
1) if the refined model is optimal/quasi-optimal, the model is then
supposed to be consistent to the data used for test set too; or the
model is far from optimal. In this regard, switching test sets/using
different test sets will not be a problem for this kind of "ideal"
case---enough refinement cycles should be able to bring consistent
models.
2) keeping the same reflections for test set will leave these
reflections lost any chance to contribute the minimization process in
a direct fashion, which itself causes a kind of bias. From the point
of data, if any data are excluded, no matter randomly or not, from
calculation, artificial bias would be resulted! If the initial model
is biased, it will be biased forever if it caused due to the exclusion
of test set (this sounds more true when with low resolution data).
So, a judgement may need be based on your data! At the "end" (I mean
you are going to stop) of a smooth refinement, switching test sets
should not be a "huge" problem, or the model is too wrong!
Back to Mike's question: I suggest you keep the same test set, since
your data were from the exactly same crystal. At least it saves your
convergence time.
Lijun Liu
On Sep 24, 2009, at 10:24 AM, Ian Tickle wrote:
-----Original Message-----
From: Dale Tronrud [mailto:det...@uoxray.uoregon.edu]
Sent: 24 September 2009 17:21
To: Ian Tickle
Cc: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Rfree in similar data set
While I agree with Ian on the theoretical level, in practice
people use free R's to make decisions before the ultimate model
is finished, and our refinement programs are still limited in
their abilities to find even a local minimum.
I wasn't saying that Rfree is only useful for the ultimate finished
model. My argument also applies to all intermediate models; the
criterion is that the refinement has converged against the current
working set, even if it is only an incomplete model, or if it is
only to
a local optimum. So it's perfectly possible to use Rfree for
overfitting & other tests on intermediate models. The point is that
it
doesn't matter how you arrived at that optimum (whether local or
global), Rfree is a function only of the parameters at that point, not
of any previous history. If you arrived at that same local or global
optimum via a path which didn't involve switching datasets midway, you
must get the same answer for Rfree, so I just don't see how it can be
biased one way and not biased the other. Note that this is meant as a
'thought experiment', I'm not saying necessarily that it's possible to
perform this experiment in practice!
On the automated level the test set is used, sometimes, to
determine bulk solvent parameter, and more importantly to calibrate
the likelihood calculations in refinement. If the test set is
not "free" the likelihood calculation will overestimate the
reliability
of the model and I'm not confident that error will not become
a self-fulfilling prophecy. It is not useful to divine meaning
from the free R until convergence is achieved, but the test
set is used from the first cycle.
That is indeed a fair point, but I would maintain that the test set
becomes 'free', i.e. free of the memory of all previous models, the
first time you reach convergence, so the question of the effect on
sigmaA calculations, which use the test set, is only relevant to the
first refinement after switching test sets, thereafter it should be
irrelevant. Converging to a local or global optimum wipes out all
memory of previous models because the parameter values at that optimum
are independent of any previous history, and so Rfree must be the same
for that optimum no matter what path you took to get there.
Perhaps I'm in one of my more persnickety moods, but every
paper I've read about optimization algorithms say that the method
requires a number of iteration many times the number of parameters
in the model. The methods used in refinement programs are pretty
amazing in their ability to drop the residuals with a small number
of cycles, but we are violating the mathematical warranty on
each and every one of them. A refinement program will produce
a model that is close to optimal, but cannot be expected to be
optimal. Since we haven't seen an optimal model yet it's hard
to say how far we are off.
I thought that for a quadratic approximation CG requires a number of
iterations that is not more than the number of parameters (not that we
ever use even that many iterations!)? Anyway that's a problem in
theory, but it's possible to refine until nothing more 'interesting'
happens, i.e. further changes appear to be purely random and at the
level of rounding errors. Plotting the maximum shift of the atom
positions or B factors from one iteration to the next is a very
sensitive way of detecting whether convergence has been achieved;
looking at changes in R factors or in RMSDs of bonds etc is a bad way,
since R factors are not sensitive to small changes and atoms can
move in
concert without affecting bond lengths etc. (or it may just be the
waters that are moving!).
As a final point I would note that cell parameters frequently vary by
several % between crystals even from the same batch due to unavoidable
variations in rates of freezing etc, so what you think are independent
test set reflections may in reality overlap significantly in
reciprocal
space with working set reflections from another dataset anyway!
-- Ian
Disclaimer
This communication is confidential and may contain privileged
information intended solely for the named addressee(s). It may not
be used or disclosed except for the purpose for which it has been
sent. If you are not the intended recipient you must not review,
use, disclose, copy, distribute or take any action in reliance upon
it. If you have received this communication in error, please notify
Astex Therapeutics Ltd by emailing i.tic...@astex-therapeutics.com
and destroy all copies of the message and any attached documents.
Astex Therapeutics Ltd monitors, controls and protects all its
messaging traffic in compliance with its corporate email policy. The
Company accepts no liability or responsibility for any onward
transmission or use of emails and attachments having left the Astex
Therapeutics domain. Unless expressly stated, opinions in this
message are those of the individual sender and not of Astex
Therapeutics Ltd. The recipient should check this email and any
attachments for the presence of computer viruses. Astex Therapeutics
Ltd accepts no liability for damage caused by any virus transmitted
by this email. E-mail is susceptible to data corruption,
interception, unauthorized amendment, and tampering, Astex
Therapeutics Ltd only send and receive e-mails on the basis that the
Company is not liable for any such alteration or any consequences
thereof.
Astex Therapeutics Ltd., Registered in England at 436 Cambridge
Science Park, Cambridge CB4 0QA under number 3751674
Cardiovascular Research Institute
University of California, San Francisco
1700 4th Street, Box 2532
San Francisco, CA 94158
Phone & Fax: (415)514-2836
