Just to add that the R-free and R-sleep are nothing else than what is
'officially' known as 'test' set and 'validation' set.
In statistical pattern recognition for example, one would use a
'training' set to establish the 'model'; the power of the model
(in this case the 'model' can be for example a neural network scheme)
is judged against a 'test' set that is kept aside from training.
However, the model is chosen and improved against this 'test' set!
Thus, at the end the 'performance' of the model has to be documented
against the 'validation' test, that has not been used for anything
else !
Since R-free set is often - fortunately - referred to as 'test set',
R-sleep should maybe referred to as 'validation set'.
Although in principle very appropriate and quite standard in other
fields, I agree with Ian, that its practical use
can be questioned and might need some investigation. I suspect that
if one would engage in some kind of automation
project such a whole-PDB re-refinement (brrrrr ....) an R-sleep might
be necessary, since refinement protocols will have
to be chosen against the R-free to achieve automation.
Tassos
On Oct 1, 2007, at 16:21, Ian Tickle wrote:
The question is how significant is this bias, and is the cure (i.e.
leaving out more reflections from the working set) worse than the
disease? For refinements at 'medium' typical resolution (around
2.5 to 2 Ang) we are working with an observation/parameter count
ratio of say < 3 (naturally I'm counting the geometric restraints
with the X-ray observations). The amount of bias in Rwork and
other statistics derived from the working set depends critically on
how close the obs/param ratio is to 1. The Rfree optimisation is
used only to determine weighting parameters (including sigma-A) and
it's unlikely there will be more than say 20 of these. Typically
there are at least 1000 refls in the test set, so for the Rfree
optimisation the obs/param ratio will be around 50. This is much
larger than the obs/param ratio for Rwork and may well mean that
the biasing effect on Rfree is negligible. It should be easy to do
some tests comparing Rfree with Rsleep to check the bias (taking
into account errors to limited sample sizes of course), and also to
see what are the effects of leaving out the sleeping set on the
refinement and the maps. I don't think it would be wise to rush
into this until we have done proper evaluations.
-- Ian
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Mark J. van Raaij
Sent: 01 October 2007 14:58
To: CCP4BB@JISCMAIL.AC.UK
Subject: R-sleep
Dear All,
the short paper by Gerard Kleywegt (ActaD 63, 939-940) treats
an interesting subject (at least I think so...). I agree that
what we are now doing in many cases is effectively refining
against Rfree. For example, the standard CNS torsion angle
refinement does n refinement trials with randomised starting
points. If you then take the one with lowest Rfree (or let a
script do this for you), you are biasing Rfree!
Therefore, his proposal to put an extra set of reflections in
a dormant "vault" (R-sleep) sounds like a good idea to me.
However, how would the "vault" be implemented to be
effective? If left to the experimenter, it would be very
tempting to check R-sleep once in a while (or often) during
refinement, rendering it useless as an unbiased validator.
or am I being paranoid and too pessimistic?
Mark J. van Raaij
Unidad de Bioquímica Estructural
Dpto de Bioquímica, Facultad de Farmacia
and
Unidad de Rayos X, Edificio CACTUS
Universidad de Santiago
15782 Santiago de Compostela
Spain
http://web.usc.es/~vanraaij/
Disclaimer
This communication is confidential and may contain privileged
information intended solely for the named addressee(s). It may not
be used or disclosed except for the purpose for which it has been
sent. If you are not the intended recipient you must not review,
use, disclose, copy, distribute or take any action in reliance upon
it. If you have received this communication in error, please notify
Astex Therapeutics Ltd by emailing [EMAIL PROTECTED]
and destroy all copies of the message and any attached documents.
Astex Therapeutics Ltd monitors, controls and protects all its
messaging traffic in compliance with its corporate email policy.
The Company accepts no liability or responsibility for any onward
transmission or use of emails and attachments having left the Astex
Therapeutics domain. Unless expressly stated, opinions in this
message are those of the individual sender and not of Astex
Therapeutics Ltd. The recipient should check this email and any
attachments for the presence of computer viruses. Astex
Therapeutics Ltd accepts no liability for damage caused by any
virus transmitted by this email. E-mail is susceptible to data
corruption, interception, unauthorized amendment, and tampering,
Astex Therapeutics Ltd only send and receive e-mails on the basis
that the Company is not liable for any such alteration or any
consequences thereof.
Astex Therapeutics Ltd., Registered in England at 436 Cambridge
Science Park, Cambridge CB4 0QA under number 3751674
